Biomedical data analysis workflows frequently break when migrating between local development servers and High-Performance Computing (HPC) environments. Varied versions of underlying tools like samtools, bedtools, or GATK introduce silent discrepancies in data output, compromising scientific reproducibility.

Combining Nextflow for reactive workflow orchestration with Singularity (Apptainer) for process isolation solves this dependency bottleneck while maximizing compute efficiency.

Why Singularity Over Docker for Bioinformatics?

While Docker dominates standard enterprise MLOps pipelines, it poses severe security risks in scientific HPC clusters. Docker requires root privileges to execute daemons, a privilege system administrators will not grant on shared multi-tenant supercomputers.

Singularity executes containers as the current host user, enforcing strict security compliance without sacrificing the benefits of immutable image layers.

Structure of an Optimized Nextflow Architecture

A clean Nextflow setup separates the workflow logic from the execution environment configurations. This decoupling allows the exact same code execution engine to run locally on a laptop or scale across thousands of cores on a Slurm-managed cluster.

project-root/
|
|-- main.nf               # Core workflow logic and channels
|-- nextflow.config       # Profile declarations (local, slurm, cloud)
`-- modules/
    |-- fastqc.nf         # Modular quality control process
    `-- alignment.nf      # Modular mapping process

Step 1: Writing Modular Nextflow Processes

Isolate each workflow step into an independent module. This layout ensures you can assign specific Singularity container tags and resource requirements directly to individual processes.

// modules/fastqc.nf
process FASTQC {
    tag "Quality Control on ${sample_id}"
    container 'https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0'

    input:
    tuple val(sample_id), path(reads)

    output:
    tuple val(sample_id), path("*.html"), emit: html
    tuple val(sample_id), path("*.zip"), emit: zip

    script:
    """
    fastqc --threads ${task.cpus} ${reads}
    """
}

Step 2: Centralizing Environment Management

Configure your nextflow.config file to handle container execution automatically based on runtime execution profiles. This configuration ensures Singularity caching is handled globally, preventing repetitive downloads during pipeline iterations.

// nextflow.config
profiles {
    local {
        process.executor = 'local'
    }

    hpc_slurm {
        process.executor = 'slurm'
        process.queue = 'standard'

        singularity.enabled = true
        singularity.autoMounts = true
        singularity.cacheDir = "${HOME}/.singularity_cache"
    }
}

process {
    cpus = { 1 * task.attempt }
    memory = { 2.GB * task.attempt }
    errorStrategy = 'retry'
    maxRetries = 3
}

Execution and Scalability Verification

To execute the pipeline on a local machine using container runtimes, run:

nextflow run main.nf -profile local

When shifting to the institutional HPC environment, switch the execution profile without touching a single line of your analysis code:

nextflow run main.nf -profile hpc_slurm

Nextflow automatically translates the execution steps into individual Slurm batch jobs, pulls the required Singularity containers securely into your cache directory, and mounts filesystems seamlessly. This approach reduces manual configuration work and ensures identical biological pipeline outputs regardless of the underlying hardware structure.