Chapter 2: Engineering Reproducibility and Computational Provenance

Published

June 5, 2026

Modified

August 13, 2026

An Unrecorded Step is Scientific Fiction

In bioinformatics, your code and software environment form your experimental record. If you run a script, generate a figure, and cannot trace the exact code revision, software dependencies, and reference parameters that produced it, your results are not science—they are an unverified assertion. Computational history is part of the experiment.

Core Concepts

Imagine a scenario that unfolds in computational research every day: You spend four months analyzing an RNA-seq dataset of 120 samples. You identify a novel gene expression signature, write a clean script, generate Figure 3 for a manuscript draft, and submit the paper to a journal. Six months later, during peer review, Reviewer 2 asks a straightforward scientific question: Can you re-plot Figure 3 after excluding 4 samples that had borderline RNA integrity (RIN) scores, and apply a slightly more stringent false discovery rate (FDR < 0.01) threshold?

You open your project folder on your laptop. You locate a script named normalize_counts_v2_final.R. You load the original raw data, modify the sample filter line, adjust the threshold parameter, and rerun the code.

To your horror, the output plot shifts dramatically. Genes that were previously highlighted as top biological targets vanish, while new, uncharacterized transcripts appear at the top of the table. You check your script edits—you only modified the sample exclusion list and the FDR threshold.

I ask you to consider the fundamental question: Why did the exact same analysis script produce an entirely different computational result?

The answer exposes a fundamental truth: Same code \(\neq\) same computation.

A computational result in bioinformatics is never the output of a single script file in isolation. A result is the output of a multi-layered historical state—a system equation:

\[\text{Result} = f(\text{Code}, \text{Software Dependencies}, \text{Execution Environment}, \text{Parameters}, \text{Reference Data})\]

If any layer of that historical state changes—if an underlying R library updated automatically, if default parameters in a Bioconductor package shifted between minor versions, or if a reference GTF annotation file was updated on your machine—the computation changes.

To make a computational experiment scientific, you must capture its provenance: the complete, verifiable record of every code transformation, software dependency, and environment setting that produced the result.

Layer 1: Structural Isolation of Data and Code

Before tracking code history or software packages, an analyst must establish a clear physical boundary between immutable inputs, transformation logic, and derived outputs.

A common failure mode in bioinformatics is treating a project folder as a sandbox—saving raw FASTQ files, intermediate script edits, temporary text files, and final PDF plots in a single unorganized directory. When scripts output files into their working directory, raw sequencing data risks being accidentally overwritten, truncated, or modified.

I suggest you enforce a fundamental rule: Raw biological data must be treated as immutable physical inputs. Once raw sequencing files (.fastq.gz, .bam) are written to storage, no script should ever modify or overwrite them.

A robust, explicit project directory architecture separates these concerns:

my_project/
├── .gitignore         # Declares files excluded from version control (e.g., raw data)
├── environment.yml    # Declares exact software dependencies and versions
├── README.md          # Documents project overview, execution order, and biological goals
├── data/
│   ├── raw/           # Immutable raw sequencing data (strictly read-only)
│   └── processed/     # Derived, reproducible data products (filtered matrices, BAMs)
├── scripts/           # Version-controlled code executing the analysis pipeline
└── results/           # Derived outputs (figures, summary tables, published statistics)

The deeper computational principle is not about memorizing a rigid folder naming convention. It is about establishing structural isolation: an explicit, predictable organization where raw inputs, transformation logic, intermediate state, and published results are clearly segregated.

Layer 2: Temporal Provenance via Version Control (Git)

Once your project directory is structured, you need a mechanism to record how your code evolves over time.

Standard file backup strategies (saving copies named script_v1.R, script_v2_final.R, script_v2_edit_final.R) fail because they provide no cryptographic record of what lines of code changed, why they changed, or which exact version generated a specific figure.

Git is a version control system that models code history as a Directed Acyclic Graph (DAG) of cryptographic snapshots (commits). Each commit records the exact author, timestamp, commit message, and SHA-1/SHA-256 hash representing the complete state of your codebase at a specific point in time.

The essential Git workflow for scientific tracking:

Initialize: Create a Git repository inside your project directory:
```
git init my_project
```
Inspect: Check which files have been modified or staged:
```
git status
```
Stage: Mark specific code modifications for snapshotting:
```
git add scripts/normalize_counts.R
```
Commit: Save the staged snapshot into permanent cryptographic history with an explanatory message:
```
git commit -m "Filter low-count genes using DESeq2 independent filtering"
```
Review: Inspect line-by-line changes between your active file and the last committed version:
```
git diff scripts/normalize_counts.R
```

First Principles of `.gitignore`: Why Git Fails on Large Binary Files

Young researchers often make the mistake of running git add . on a folder containing 50-gigabyte .fastq.gz or .bam files. Within minutes, Git stalls, memory usage spikes, and the repository becomes corrupt or unpushable.

I ask you to understand why this fails from first principles:

Git is designed around line-by-line text delta algorithms. For text files (like .R, .py, or .sh scripts), Git stores compact diffs—tracking only the specific lines added or deleted (\(O(\Delta N)\) storage complexity).
Sequencing files (.fastq.gz, .bam, .vcf.gz) are massive, compressed binary blobs. A single byte change in a compressed block invalidates the entire file hash. Git cannot compute line deltas; it is forced to store full, uncompressed binary copies of the entire file in its internal object database for every commit (\(O(N)\) space expansion).

To prevent repository bloat, you must create a .gitignore file at the root of your repository to explicitly exclude data directories and large binary formats:

# Exclude data and results directories from Git tracking
data/
results/

# Exclude raw and binary genomic file formats
*.fastq
*.fastq.gz
*.bam
*.bai
*.cram
*.vcf
*.vcf.gz
*.h5ad

# Exclude temporary operating system and R files
.DS_Store
.Rhistory
.RData

Layer 3: Environmental Provenance via Software Isolation (Conda and Mamba)

Git records the code you write, but it does not record the software environment that executes your code.

Suppose you write a Python script using scanpy version 1.8 to cluster single-cell RNA-seq data. A year later, a colleague runs your exact Git commit on a fresh machine where the system administrator installed scanpy version 1.9. Between these minor releases, the default community detection algorithm in scanpy.tl.leiden changed its default resolution parameter.

The Git commit is identical. The raw FASTQ data is identical. Yet the resulting cell cluster counts and UMAP plot are different.

Conda (and its accelerated solver, Mamba) solves the package dependency crisis by creating isolated software environments. Mamba downloads, compiles, and links binaries, R libraries, and Python packages into a dedicated, self-contained directory tree—completely isolated from the operating system’s global libraries.

Declarative Environments with `environment.yml`

To achieve environmental provenance, you must document every software tool, language runtime, and library version in a declarative configuration file named environment.yml:

name: rnaseq_provenance
channels:
  - bioconda
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - r-base=4.2.3
  - bioconductor-deseq2=1.38.0
  - salmon=1.10.0
  - samtools=1.17
  - subread=2.0.3

Any researcher can recreate your isolated execution environment on a new machine using a single declarative command:

mamba env create -f environment.yml

The Chain of Controlled Dependencies: Why Reproducibility is Not Binary

I recommend you avoid treating reproducibility as a binary state (“100% exact” vs. “unreproducible”). In physical computation, reproducibility is a chain of controlled dependencies.

Pinning an environment.yml constrains software package versions, but subtle variations can still arise across different hardware platforms (x86_64 vs. ARM64), operating system C-libraries (glibc version shifts), or underlying linear algebra libraries (BLAS/LAPACK floating-point rounding differences).

Understanding reproducibility as a continuous spectrum allows you to choose the appropriate level of control:

Directory Architecture: Segregates data inputs from transformation logic.
Git Commit History: Controls code logic state over time.
Conda/Mamba Environments: Controls software library dependencies across machines.
OCI Containers (Docker/Apptainer): Controls operating system user-space libraries.
Workflow Orchestration (Nextflow/Snakemake): Controls graph execution, caching, and state transitions.

Biological Interpretation

For a biological researcher, computational provenance is not an administrative burden; it is your scientific defense during peer review and post-publication discovery.

The Forensic Reviewer Mindset: Causal Attribution of Shifting Signals

When a reviewer asks you to re-evaluate an analysis, or when a follow-up experiment in your lab fails to replicate a published result, the fundamental question is never merely: Can I re-execute this script?

The forensic question is: Can I determine exactly WHY the result changed?

Suppose a differential expression analysis produces 150 significant genes in Draft 1, but only 45 significant genes in Draft 2. Without explicit provenance, you cannot determine the cause of the discrepancy. Was the shift caused by:

A change in your filtering logic in the R script?
A minor version update in DESeq2 shifting dispersion estimation trends?
A patch in the reference GTF gene annotation file changing exon lengths?
A change in the FDR threshold parameter?
System-level floating-point differences across compute nodes?

When your project has complete provenance—a recorded Git commit hash, a pinned environment.yml, and an isolated directory structure—you can perform forensic attribution. You can check out the exact Git commit that produced Draft 1, compare it line-by-line (git diff) against Draft 2, and isolate the exact parameter or package version responsible for the biological shift.

Current Landscape

Modern bioinformatics infrastructure has expanded the chain of provenance to support large-scale distributed pipelines:

Environment Lockfiles (Conda-lock and Pixi): While environment.yml declares target package versions, exact resolution can vary slightly if channel index metadata updates over time. Tools like conda-lock and pixi generate fully resolved lockfiles containing exact SHA-256 hashes and download URLs for every transitive binary dependency, guaranteeing identical environment resolution across time.
Containerized Execution (Docker and Apptainer): High-Performance Computing (HPC) centers increasingly package Conda environments inside lightweight Open Container Initiative (OCI) images using Apptainer (formerly Singularity). Containers encapsulate both the software packages and the underlying OS distribution, eliminating system library incompatibilities between Ubuntu desktop workstations and RHEL cluster nodes.
Declarative Provenance Graphs (RO-Crate and W3C PROV): Production pipelines and clinical genomics hubs auto-generate machine-readable provenance metadata (such as Research Object Crates [RO-Crate]). These frameworks log the exact input URIs, workflow DAG step executions, tool container digests, and output checksums, producing immutable cryptographic records of clinical genomic assertions.

Summary and Required Reading

Same code does not equal same computation: A result is a function of code logic, software dependencies, execution environment, parameters, and reference data.
Raw data is immutable: Treat raw sequencing reads (data/raw/) as read-only physical inputs; never overwrite raw inputs with intermediate script outputs.
Git tracks code deltas, not binary data: Use Git to record text code transformations over time, and exclude large sequencing files (.fastq, .bam) using .gitignore.
Mamba isolates software environments: Document execution dependencies in environment.yml to prevent package version drift and dynamic linking conflicts across machines.
Provenance enables forensic attribution: Complete provenance allows you to isolate the exact code edit, package update, or parameter shift responsible for a biological change.

Required Reading

Chacon & Straub: Pro Git (2nd Edition), Chapter 1 (Getting Started) and Chapter 2 (Git Basics).
Blischak, Davenport, & Reshef: “A Quick Guide to Organizing Computational Biology Projects” (PLOS Computational Biology, 2016).

Johnson’s Rule: A computational result without explicit provenance is a hypothesis without evidence. Document your software environment, commit your code history, and render your execution path traceable.