Chapter 2: Engineering Reproducibility and Workflow Orchestration

Johnson’s First Principle: A Non-Reproducible Pipeline is Scientific Fiction

In a 2016 Nature survey, more than 70% of researchers reported failing to reproduce another scientist’s computational experiments, and over 50% failed to reproduce their own experiments from just a year prior. This is not a failure of diligence — it is a failure of infrastructure. A pipeline that works today on your laptop but cannot execute on a collaborator’s machine six months later is not a pipeline; it is a draft. Code must be isolated from the host environment to guarantee identical mathematical outputs across time and space.

Reproducibility requires three distinct layers of infrastructure, each solving a failure mode the previous one cannot: version control tracks what changed, environment isolation ensures it can still run, and workflow orchestration makes it survive resumption, parallelization, and scaling. A professional bioinformatician uses all three — and knows which layer prevents which failure.

---

Core Concepts

Layer 1: Version Control — Tracking What Changed

The first reproducibility question is: “what code produced these results, and how does it differ from last week’s version?” Git answers this by storing the entire project history as an immutable Directed Acyclic Graph (DAG) of four object types:

1. Blob — file contents, named by SHA-1 hash of contents
2. Tree — directory listing mapping filenames to blobs
3. Commit — a snapshot: root tree pointer, parent commit(s), message, author, timestamp
4. Tag — human-readable name for a specific commit

Every object is identified by its SHA-1 hash. Changing a single byte in a file changes the blob hash, which changes the tree hash that contains it, which changes the commit hash, and so on up the chain. This cryptographic linking makes Git history immutable — it is computationally infeasible to alter any past commit without detection. If the code that produced Figure 3B is committed at hash a1b2c3d, that hash uniquely identifies the exact state of every file at that moment.

Concrete trace. Consider a project with one file analysis.R containing library(DESeq2). Git computes SHA-1("blob library(DESeq2)\n") as the blob hash, then hashes the tree object mapping analysis.R to that blob, then hashes the commit containing that tree. Adding a single byte to analysis.R produces a completely different blob hash, which propagates to a new tree hash and a new commit hash. The old commit still exists in history, unchanged, pointing to the original file state. This is how Git guarantees that no past state is ever lost or overwritten.

The three trees in Git’s architecture:

Working Directory → git add → Staging Area (Index) → git commit → HEAD (History)

# Basic workflow
git init my_project
git add README.md
git commit -m "Initial commit"
git diff                  # Review changes before staging
git diff --cached         # Review staged changes

Branching and code review. The main branch represents verified truth; all new work occurs on feature branches and enters main through pull requests with code review:

git checkout -b feature/add-analysis
# ... work ...
git commit -am "Add DESeq2 analysis script"
git checkout main
git merge feature/add-analysis

What Git cannot do. Git tracks file changes, but it does not track the software environment. A commit hash tells you which version of analysis.R produced a result, but it does not tell you which version of DESeq2, ggplot2, or R itself was used. If the package API changed between commits, the same code may produce different results or fail entirely.

Essential bioinformatics .gitignore: Exclude *.fastq, *.fastq.gz, *.bam, *.cram, *.sam, *.vcf, results/, __pycache__/, .env. Commit code, not data. Raw data belongs in S3 or network storage, not in Git.

Layer 2: Environment Isolation — Ensuring It Still Runs

Version control tells you what changed. Environment isolation tells you how to run the exact same software stack. The failure modes escalate in severity:

Degree	Example	Consequence
1st	library(DESeq2) installed but version unknown	Results may differ silently across machines
2nd	ggplot2 upgrade deprecates a function	Pipeline fails on new machine
3rd	Python 2 → 3: print is now a function	Code will not run at all

No two machines have the same software environment by default. A collaborator who clones your Git repository has the code, but not R 4.2.3, DESeq2 1.38.0, or the system-level libraries (zlib, OpenSSL, libcurl) that your pipeline requires. Environment isolation solves this by pinning every dependency to an exact version and making the environment reproducible as a single file.

Conda: dependency isolation. Conda creates isolated environments with pinned package versions. It handles both language-level packages (R, Python) and system-level libraries (zlib, hdf5) in a unified dependency resolver:

conda create -n rnaseq python=3.9 pandas=1.4.3 numpy=1.21.0
conda install -c bioconda salmon star samtools
conda env export > environment.yml  # Exact pinning for reproducibility

The environment.yml file is the minimum reproducibility requirement. An environment specified as python>=3.6 without pinned versions is scientific negligence: it will resolve to different versions on different dates, silently producing different results. A properly pinned environment.yml specifies exact versions and records transitive dependencies:

name: rnaseq
channels:
  - bioconda
  - conda-forge
  - defaults
dependencies:
  - python=3.9.13
  - salmon=1.9.0
  - star=2.7.10a
  - samtools=1.17
  - pandas=1.4.3

Why Conda, not pip. Python’s pip installs only Python packages. Bioinformatic tools like samtools, bwa, and star are C/C++ programs with system-level dependencies (zlib, hdf5, ncurses, libcurl). Conda resolves these alongside Python packages in a single dependency graph, preventing library version conflicts between system and language package managers.

But Conda is not absolute isolation. Conda environments share the host operating system kernel. A pipeline that worked on Ubuntu 20.04 may fail on CentOS 7 if it depends on a system library whose version differs between distributions.

Docker: operating system isolation. Docker freezes the entire operating system, dependencies, and code into an immutable image. The Dockerfile builds the environment from a base OS image layer by layer:

FROM continuumio/miniconda3:22.11.1
RUN conda install -c bioconda -c conda-forge bwa=0.7.17 samtools=1.17
COPY scripts/ /pipeline/scripts/
WORKDIR /pipeline
ENTRYPOINT ["bash", "scripts/run.sh"]

docker build -t my-pipeline:1.0 .
docker run --rm -v /data:/data my-pipeline:1.0

A Docker image contains the operating system, the package manager, every dependency, and the code — all frozen at known versions. If the image builds, it produces identical outputs on any host that can run Docker. This is why Johnson’s Rule says “a Dockerfile is the ultimate README”: it is a single file that, when executed, produces the entire computational environment.

Layer caching: why instruction order matters. Each Dockerfile instruction creates a layer that is cached by the build system. When a file changes, only the layers from that instruction forward are rebuilt — earlier layers are reused from cache. This is why the Dockerfile orders RUN conda install before COPY scripts/: changing scripts invalidates only the COPY layer and everything after it, not the expensive conda install step. Reversing the order (copying scripts before installing conda) means every code change triggers a full conda re-installation, turning seconds of rebuild time into minutes. Layer caching also applies to apt-get install, pip install, and any other expensive instruction — stable operations go first, frequently changing content goes last.

Singularity/Apptainer: HPC containerization. Docker containers run as root, which is banned on shared HPC systems for security reasons. Singularity (now Apptainer) converts Docker images to a single .sif file and runs them as the current user, without root privileges:

singularity build my-pipeline.sif docker://user/my-pipeline:1.0
singularity exec --bind /scratch:/data my-pipeline.sif bwa mem ref.fa reads.fq

Choosing the right isolation level:

Tool	Scope	Use Case
Conda	Language + system libraries	Local development, single-machine pipelines
Docker	Entire OS	Absolute reproducibility, cloud deployment
Singularity	Entire OS (user-space)	HPC clusters where Docker is banned

Most projects should use Conda for development and Docker (or Singularity for HPC) for production. The Conda environment.yml lives in the repository for quick local setup; the Dockerfile builds on top of it for deployment.

Layer 3: Workflow Orchestration — Making Pipelines Survive and Scale

Environment isolation ensures the pipeline can run. But a bash script that loops through 500 samples sequentially has two fatal problems: (1) if sample #423 fails, everything after it is lost, and the entire run must restart from the beginning; (2) independent steps cannot run in parallel.

A Directed Acyclic Graph (DAG) workflow system solves both. Each sample’s processing is an independent node in the graph. The system tracks which nodes completed successfully, which failed, and which inputs each node depends on. The pipeline can resume from the point of failure, and independent nodes execute in parallel.

Nextflow is the dominant DAG workflow system in bioinformatics. Its -resume flag caches successful process executions by their input hash:

nextflow.enable.dsl=2

process ALIGN {
    cpus 16
    memory 64.GB

    input:
    tuple val(sample_id), path(reads)
    path genome

    output:
    tuple val(sample_id), path("${sample_id}.sorted.bam")

    script:
    """
    bwa mem -t ${task.cpus} ${genome} ${reads} | \
        samtools sort -@ ${task.cpus} -o ${sample_id}.sorted.bam -
    """
}

workflow {
    Channel.fromFilePairs(params.reads).set { read_pairs_ch }
    ALIGN(read_pairs_ch, file(params.genome))
}

nextflow run pipeline.nf -params-file params.yml
# Resume after failure at sample #850:
nextflow run pipeline.nf -params-file params.yml -resume

The DAG approach is not specific to Nextflow. Snakemake provides equivalent semantics in Python syntax, and is the better choice for labs already embedded in the Python data science ecosystem. The choice between frameworks should be driven by the execution environment and community ecosystem — Nextflow dominates academic bioinformatics (nf-core), while WDL/Cromwell dominates clinical settings (GATK compatibility and FDA regulatory precedent).

When a DAG system is overkill. For pipelines with 2-3 steps and no batch looping, make provides dependency tracking and incremental rebuild without the learning curve of a workflow language. A Makefile declares targets and their prerequisites; make rebuilds only what changed based on file timestamps. This is sufficient for small projects and serves as a stepping stone to full DAG systems when complexity grows.

---

Biological Interpretation

Reproducibility infrastructure is not an afterthought — it is the boundary between a hypothesis and a result. An analysis published with a Dockerfile and a Nextflow pipeline can be re-executed by any reviewer, on any system, at any future date. An analysis published as “code available on request” is a claim that cannot be verified. The Nature survey finding that >50% of researchers cannot reproduce their own work after one year is a direct consequence of treating environment management as optional.

The corollary: reproducibility does not guarantee correctness. A pipeline that always produces the same output from the same input is reproducible. If that output is wrong — because of a parameter error, a normalization mistake, or a reference genome incompatibility — the pipeline is reproducible nonsense. Validation benchmarks (positive and negative controls) must be built into every workflow as assertions that the pipeline produces biologically plausible outputs. Reproducibility guarantees consistency; validation guarantees correctness.

The three-layer framework has a practical implication for project structure. Every bioinformatics project should contain, at minimum:

1. A git repository with meaningful commit history and a .gitignore that excludes data
2. An environment.yml or Dockerfile that builds the computational environment
3. A workflow definition (Nextflow, Snakemake, or equivalent Makefile) that encodes DAG dependencies

The presence of all three files — .git/, environment.yml, and main.nf — is a strong signal of professional-grade analysis. Their absence is a warning.

---

Current Landscape (Q2 2026)

• Container-native HPC: Podman and Apptainer are replacing Docker-in-HPC workflows, eliminating the need for Docker-to-Singularity conversion steps by supporting Docker images natively.
• WDL/Cromwell remains dominant in clinical settings due to GATK compatibility and FDA regulatory precedent for diagnostic pipelines.
• Cloud orchestration: Seqera Platform and Nextflow Tower add monitoring, cost tracking, and multi-cloud execution layers to Nextflow pipelines — useful for labs scaling beyond a single HPC cluster.
• AI-assisted debugging: LLMs for error log analysis are emerging but not yet reliable for production pipeline diagnostics. Treat them as a suggestion engine, not a debugger.
• Peer-to-peer image distribution: Harbor reduces container image download time on shared compute nodes through BitTorrent-based distribution.

---

Summary and Required Reading

1. Version Control (Git) — tracks what changed. Immutable history via Merkle DAG. Branching and pull requests enforce code review. Never commit raw data.
2. Environment Isolation (Conda → Docker → Singularity) — ensures the pipeline can run. Conda for local pinning, Docker for absolute OS-level isolation, Singularity for HPC. Each level solves a failure mode the previous one cannot.
3. Workflow Orchestration (Nextflow, Snakemake, WDL) — makes pipelines survive failure and scale. DAG-based execution provides automatic parallelization, resumption from failure, and dependency tracking.
4. Reproducibility does not guarantee correctness — validation controls are equally essential. Build positive and negative controls into every workflow.

Required Reading

• Di Tommaso et al.: “Nextflow enables reproducible computational workflows” (Nature Biotechnology, 2017).
• Nüst et al.: “Ten Simple Rules for Writing Dockerfiles for Reproducible Research” (PLOS Computational Biology, 2020).
• Chacon & Straub: Pro Git (Apress, 2014), Chapters 1-3 and 10 (object model).

Johnson’s Rule: A pipeline without version control, environment isolation, and workflow orchestration is a draft, not a manuscript.