Chapter 1: The Bioinformatic Computing Environment

Johnson’s First Principle: Compute is Physical, Not Magical

A modern CPU contains billions of transistors etched onto a silicon die. Every operation — reading a file, aligning a read, training a model — directs the flow of electrons through a physical lattice. Computation costs time and energy. A student who types wc -l and sees a number appear has understood nothing. A professional who types wc -l and estimates completion time from I/O throughput, RAM bandwidth, and CPU cache has understood everything.

The corollary is that every pipeline failure traces to one of three physical bottlenecks: the CPU is maxed out and needs parallelization; the disk cannot feed data fast enough and the CPU stalls waiting; or the data does not fit in RAM and the operating system kills the process. Identifying which of these three is happening — and knowing what to do about it — is the first skill a bioinformatician must develop.

Core Concepts

The Bottleneck Framework

A running process consumes three physical resources: CPU cycles (computation), RAM (working memory), and I/O bandwidth (data movement from storage). At any moment, one of these is the limiting factor — the bottleneck — and the other two are underutilized.

The diagnostic question is simple: - CPU at 100%, pipeline slow: the task is CPU-bound. More cores or a faster algorithm will help. - CPU below 50%, pipeline slow: the task is I/O-bound. The CPU spends most of its time waiting for data from storage. Faster storage or streaming will help. - Memory climbing steadily toward available RAM: the task is memory-bound. The process will be killed when it exceeds physical RAM. Streaming, memory-mapped files, or a lower-memory algorithm will help.

Every tool and concept in this chapter exists to answer one of these three diagnostic questions. Understanding which bottleneck you face determines which tool you reach for.

The Operating System: Hardware Traffic Cop

An Operating System (OS) sits between physical hardware and software applications, managing resource allocation. Modern OSes enforce a strict separation between the kernel (privileged hardware access) and user-space (where applications run). This architecture is a safety mechanism — if your Python script has a memory leak, the kernel kills it without crashing the machine.

When a program needs hardware access, it issues a system call:

  • read(fd, buf, count) — Read bytes from a file descriptor
  • write(fd, buf, count) — Write bytes to a file descriptor
  • mmap(addr, length, prot, flags, fd, offset) — Map a file into memory
  • fork() — Create a new process
  • execve(path, argv, envp) — Load a new program

Every bioinformatics operation — reading a FASTQ, writing a BAM, launching a subprocess — is a system call. The OS scheduler decides how to allocate CPU time, RAM pages, and I/O bandwidth among all running processes. If a program spends most of its time in read(), it is I/O-bound. If it spends time in sched_yield(), it is waiting for the CPU scheduler.

Why this matters for bottleneck diagnosis: Tools like strace or perf can reveal which system call dominates execution time. A process spending 80% of its time in read() is clearly I/O-bound — there is no point adding more CPU cores.

Why UNIX Dominates Bioinformatics

UNIX (specifically Linux) is the lingua franca of computational biology. POSIX (Portable Operating System Interface) guarantees compatibility across systems: a shell script written on macOS will run on a Linux HPC cluster, a cloud instance, and a supercomputer.

The core abstraction — everything is a file — is what makes stream processing possible. Data files, hard drives (/dev/sda), keyboard input (/dev/stdin), terminal output (/dev/stdout), and network sockets are all accessed via the same read()/write() system calls. This is why grep and awk can process piped data and files with identical syntax — the pipe operator simply connects one process’s stdout (a file descriptor) to another’s stdin.

100% of the TOP500 supercomputers run Linux. The bioinformatics ecosystem (BWA, GATK, SAMtools, DESeq2, Salmon, STAR) is developed and tested on Linux. Resistance to learning the UNIX command line is resistance to doing bioinformatics.

CPU: The Compute Resource

Cores and threads. A modern CPU contains multiple physical cores, each capable of running an independent instruction stream. Hyperthreading presents two logical threads per physical core, but a tool requesting 64 CPUs typically gets at most 32x throughput — the extra threads share the same execution units. Diagnosing whether hyperthreading is active is straightforward:

grep "cpu cores" /proc/cpuinfo | uniq      # Physical cores
grep "siblings" /proc/cpuinfo | uniq        # Logical threads

If siblings exceed cpu cores, hyperthreading is on. A pipeline that requests “all CPUs” on a hyperthreaded system may waste resources by oversubscribing physical execution units.

SIMD (Single Instruction, Multiple Data). Modern CPUs contain vector units that perform the same operation on multiple data points simultaneously. Tools like BWA-MEM2 and minimap2 leverage SIMD for significant speedups that naive Python implementations cannot match. This is why replacing bwa with bwa-mem2 can produce a 2x speedup without any change in output — the algorithm is identical, but the new version uses SIMD instructions.

The parallelization ceiling: Amdahl’s Law. Parallel speedup is bounded by the serial fraction of the workload. Amdahl’s Law formalizes this:

\[S = \frac{1}{(1-p) + p/n}\]

where \(S\) is speedup, \(p\) is the fraction that can be parallelized, and \(n\) is the number of cores. If 10% of the alignment step is serial (loading the reference index, writing the output header), \(p = 0.90\) and the maximum speedup is \(1/(0.10 + 0.90/n)\) — approaching \(10\times\) even with infinite cores. The first question when parallelizing is not “how many cores do I have” but “what fraction of this task is serial.”

When CPU is the bottleneck: htop shows all cores near 100%, but the pipeline is slower than desired. The solution is to parallelize across cores (use multi-threaded tools, split input files) or to use a more efficient algorithm (SIMD-optimized tools, C/Fortran libraries instead of Python loops).

RAM: The Working Memory

The memory hierarchy. Data moves through a series of progressively slower storage layers, with the CPU at the center:

Layer Latency Relative to L1
L1 cache 0.5 ns 1x
L2 cache 7 ns 14x
L3 cache ~40 ns 80x
RAM 100 ns 200x
NVMe SSD 20,000 ns 40,000x
HDD seek 10,000,000 ns 20,000,000x

The critical gap is between RAM and storage, but only for random access patterns — reading scattered locations across a file. Sequential access (reading a file from start to end) is fast because the hardware prefetcher anticipates the next cache line and the OS read-ahead buffer loads the next block before it is requested. A grep that scans a 50 GB FASTQ sequentially can saturate NVMe bandwidth (~5 GB/s). A samtools view without a BAI index that seeks to a specific genomic region makes thousands of random reads — each requiring a disk seek — and runs 100x slower. The access pattern, not just the data volume, determines whether the bottleneck is the storage device.

Loading a 100 GB FASTQ file with read.csv() reads the entire file into RAM. If the machine has 64 GB of RAM, the OS will swap to disk (using SSD as extremely slow RAM) and eventually invoke the OOM (Out-of-Memory) killer — the kernel process that terminates the offending program.

When memory is the bottleneck: htop shows RAM usage climbing steadily toward 100%, and the pipeline eventually crashes with a killed process. The forensic record of this event:

dmesg | grep -i oom   # Kernel log showing which process was killed

Preventive memory management:

ulimit -v 8000000     # Set 8 GB virtual memory limit — fail fast, not at hour 47

The ulimit command prevents the OOM scenario by failing immediately when the limit is exceeded, rather than letting the process thrash for hours before the kernel intervenes. Setting a memory limit based on a test dataset run is a hallmark of production-grade pipeline engineering.

The two professional solutions to memory-bound problems are streaming and memory-mapped files — both of which avoid loading the entire dataset into RAM at once. These are covered next.

Storage and I/O: The Data Movement Bottleneck

Storage speed varies by over three orders of magnitude depending on technology:

Device Sequential Read Random Access
HDD ~150 MB/s ~1 MB/s
SATA SSD ~500 MB/s ~50 MB/s
NVMe SSD ~3-7 GB/s ~500 MB/s

The gap between HDD and NVMe is 100x for sequential reads and 500x for random access. This difference explains why BAM files require .bai indexes. Without an index, samtools view must scan the entire BAM file to find reads in a region. With an index, it reads only the relevant compressed block — reducing a minutes-long random seek to milliseconds.

The page cache: why the second read is free. When the OS reads a file from disk, it keeps the data in a region of RAM called the page cache. A subsequent read of the same file returns instantly from memory — iostat shows 0% utilization because no disk access occurred. This creates a trap for pipeline development: the first run of samtools view is slow (reading from disk), but the second run appears fast (reading from cache), leading the developer to think the pipeline is faster than it will be on a cold cache. Always measure I/O performance on a cold cache (after echo 3 > /proc/sys/vm/drop_caches, or on the first run after a reboot) to get realistic benchmarks.

When I/O is the bottleneck: htop shows CPU below 50% and the pipeline is slow. The diagnostic tools:

iostat -x 2          # I/O statistics every 2 seconds — check %util
iotop -oP            # Per-process I/O activity — which process is reading?

If iostat shows a device at 100% utilization, the storage system is saturated. The solution is faster storage (NVMe over HDD), streaming (fewer intermediate files), or data reduction (filtering reads before expensive operations).

Stream Processing: The Pipe as I/O Bypass

The pipe operator (|) is the most important tool in the bioinformatician’s I/O optimization toolkit. It connects the stdout of one process directly to the stdin of another without writing to disk. Data flows through a kernel buffer — entirely in memory — and is consumed incrementally:

# Novice: writes ~20 GB of intermediate files to disk
grep "BRCA1" huge_file.fastq > temp.txt
sort temp.txt > sorted.txt
uniq -c sorted.txt > counts.txt

# Professional: streaming pipeline, one disk read, everything in memory
grep "BRCA1" huge_file.fastq | sort | uniq -c

The streaming version reads the file once, processes it through three tools in memory, and writes the final result. The novice version reads and writes the same data three times, tripling I/O time and wearing the disk.

Streaming also solves memory-bound problems. The essential streaming tools:

  • grep — pattern matching
  • awk — column math and text processing
  • sed — text replacement
  • sort | uniq -c — frequency counting
  • fold — line wrapping (useful for k-mer generation)

The generalization: Any pipeline step that writes a transient intermediate file should be examined for whether the I/O is necessary. If the next step can consume a stream, eliminate the file.

HPC Job Scheduling with SLURM

When analysis exceeds a single workstation, jobs are submitted to a cluster. SLURM (Simple Linux Utility for Resource Management) is the dominant scheduler in academic bioinformatics. Users write submission scripts that specify hardware requirements and submit them to queues (partitions):

#!/bin/bash
#SBATCH --job-name=alignment
#SBATCH --time=48:00:00
#SBATCH --cpus-per-task=16
#SBATCH --mem=128G

module load bwa/0.7.17
bwa mem -t 16 reference.fa sample_R1.fastq.gz sample_R2.fastq.gz \
  | samtools sort -@ 4 -o sample_sorted.bam -

The bottleneck framework applied to SLURM: The most common SLURM mistakes are failures to estimate resource requirements — which is the same diagnostic problem this chapter is built around. Over-requesting CPUs (requesting 64 on a hyperthreaded system with 32 physical cores leaves 32 cores idle that other users cannot use). Over-requesting time (delays scheduling because the job must wait for a longer time window). Under-requesting memory (OOM at hour 47 of a 48-hour job — the job fails, and the queue position is lost).

The solution is to profile the pipeline on a test dataset first, measure actual CPU, memory, and time usage, then request resources with a 2x safety margin:

sbatch my_script.sh              # Submit job
squeue -u $USER                  # Check job status
scancel JOBID                    # Cancel job
sacct -j JOBID --format=JobID,State,Elapsed,MaxRSS  # Actual resource usage

Persistent Sessions with tmux

A dropped SSH connection sends SIGHUP, terminating all child processes — including a pipeline that has been running for 46 hours. tmux decouples running processes from the terminal session, so the pipeline survives disconnection:

tmux new -s pipeline1            # Start session
# Ctrl-b, d                      # Detach (session continues after disconnect)
tmux attach -t pipeline1         # Reattach from anywhere

This is a small tool with an outsized impact: it prevents the “SSH disconnected, job lost” failure mode that every bioinformatician encounters. Use tmux (or screen) for any pipeline running longer than a few minutes on a remote machine.

Biological Interpretation

The bottleneck framework transforms how a bioinformatician approaches a new analysis. Before looking at algorithms or parameters, the professional asks: “What is the rate-limiting resource, and how do I measure it?”

CPU-bound diagnosis: htop shows all cores above 90%. The pipeline is compute-limited. Parallelize across samples, use multi-threaded tools, or switch to SIMD-optimized implementations. The gain from parallelization is bounded by Amdahl’s Law — if 10% of the pipeline is serial, maximum speedup is 10x regardless of cores.

I/O-bound diagnosis: htop shows CPU below 50%, iostat shows the disk at 100% utilization. The pipeline is data-movement-limited. Caveat: this heuristic assumes the tool is multi-threaded. A single-threaded aligner on a 32-core machine shows only ~3% CPU usage regardless of whether it is CPU-bound or I/O-bound — the low CPU percentage reflects idle cores, not a stalled pipeline. To distinguish, check whether a single core is saturated (htop per-core view, or mpstat -P ALL 1). If one core is at 100% and the rest are idle, the tool is single-threaded and CPU-bound. If all cores are below 50% and the tool is multi-threaded, then it is I/O-bound. Switch to streaming (pipes), use faster storage (NVMe over HDD), add BAM/CRAM indexes to reduce random access, or reduce data volume (filter reads before alignment).

Memory-bound diagnosis: htop shows RAM climbing to capacity, dmesg shows OOM kills. The pipeline is capacity-limited. Switch to streaming (generators in Python), use memory-mapped files (mmap via numpy.memmap or pandas.read_csv(chunksize=...)), estimate memory from a test dataset and provision appropriately.

A genome assembler requiring 400 GB of RAM on a 128 GB workstation is not “running slowly” — it is failing. The correct response is not to wait longer; it is to estimate memory requirements from a test dataset, provision appropriate resources, or use an algorithm with lower memory footprint.

Every intermediate file written to disk represents an I/O bottleneck that could have been avoided with a pipe. The question “did I write a file I could have streamed?” should be asked of every step in every pipeline.

Current Landscape (Q2 2026)

  • Cloud HPC: AWS Batch and Google Batch are increasingly replacing on-premise SLURM for burst workloads, but SLURM remains dominant in academic sequencing centers where capital equipment is already purchased.
  • ARM processors: AWS Graviton and NVIDIA Grace are entering HPC bioinformatics with competitive performance and lower power consumption, but software compatibility (particularly for SIMD-optimized tools) remains uneven.
  • GPU-accelerated bioinformatics: GATK, minimap2, and STAR-Fusion now offer GPU acceleration, changing the CPU vs. GPU resource planning equation — but GPU memory is still a bottleneck (40-80 GB vs. 256 GB+ RAM for CPU nodes).
  • NVMe-over-fabric: Enables remote direct memory access for multi-node genome assembly, reducing data movement bottlenecks in distributed computations by allowing nodes to access remote storage at near-local speeds.

Summary and Required Reading

  1. Compute is physical — every operation costs time and energy. Every pipeline failure traces to one of three bottlenecks: CPU, I/O, or memory.
  2. Diagnose before you optimize — htop tells you CPU vs. I/O. iostat tells you I/O saturation. dmesg tells you OOM kills. Measure the bottleneck before choosing the solution.
  3. Streaming eliminates disk I/O — pipes keep data flowing through memory. Never write transient intermediate files.
  4. Memory has a hierarchy — L1 cache → RAM → SSD → HDD. The gap between layers is the source of most performance problems.
  5. SLURM resource requests must be measured, not guessed — profile on a test dataset, request 2x safety margin via sacct.
  6. tmux prevents the disconnected-SSH failure — use it for any remote pipeline longer than a few minutes.

Required Reading

  • Tenenbaum & Bos: Modern Operating Systems (Chapters 1-3) — OS architecture, process management, memory hierarchy.
  • UNIX command-line fundamentals: pipes, redirection, stream processing with grep, awk, sed.

Johnson’s Rule: If you cannot navigate a UNIX terminal blindfolded, you cannot do bioinformatics at scale. A 100 GB file does not fit in read.csv() — but it fits in a pipe.