From 2929bb22474857540fab89764d7efa60e429fbbb Mon Sep 17 00:00:00 2001 From: Jonathan Manning Date: Fri, 15 May 2026 17:45:20 +0100 Subject: [PATCH 1/2] Split debugging side quest into mini-course; add -dump-hashes lesson Restructure the Troubleshooting Workflows side quest as a two-lesson mini-course (matching the plugin_development pattern), and add new coverage of cache-invalidation debugging using -dump-hashes. - index.md becomes a short landing page (overview, prerequisites, lesson plan, learning objectives split by lesson) - 01_common_errors.md is the existing catalog of syntax, channel and process errors (was sections 1-3 of the original) - 02_debugging_toolkit.md is rewritten around a single spine pipeline (sample_processing.nf) so each tool is applied in sequence: work-dir forensics, -preview, debug true, -stub-run, -dump-hashes, systematic method, and a practical exercise on buggy_workflow.nf - new -dump-hashes section walks three experiments (script comment, resource directive, channel map change) and shows in each case which hash component changes and why The old single-page index.md was 2659 lines; splitting and re-anchoring around one spine pipeline makes the second half navigable rather than a slog. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../side_quests/debugging/01_common_errors.md | 1819 +++++++++++ .../debugging/02_debugging_toolkit.md | 1019 +++++++ docs/en/docs/side_quests/debugging/index.md | 2675 +---------------- docs/en/mkdocs.yml | 5 +- side-quests/debugging/sample_processing.nf | 55 + 5 files changed, 2937 insertions(+), 2636 deletions(-) create mode 100644 docs/en/docs/side_quests/debugging/01_common_errors.md create mode 100644 docs/en/docs/side_quests/debugging/02_debugging_toolkit.md create mode 100644 side-quests/debugging/sample_processing.nf diff --git a/docs/en/docs/side_quests/debugging/01_common_errors.md b/docs/en/docs/side_quests/debugging/01_common_errors.md new file mode 100644 index 0000000000..216ddce369 --- /dev/null +++ b/docs/en/docs/side_quests/debugging/01_common_errors.md @@ -0,0 +1,1819 @@ +# Part 1: Common errors and how to fix them + +This lesson is a catalogue of the errors you'll meet most often when writing Nextflow workflows. +Each entry shows the error in action, the message Nextflow produces, and the fix. + +Read it through once to learn the shape of each error, or jump to a specific section when a matching message lands in your terminal. + +When you're done here, move on to [Part 2: The Nextflow debugging toolkit](02_debugging_toolkit.md) to learn the tools you reach for when an error isn't obvious from its message alone. + +--- + +## 0. Get started + +#### Open the training codespace + +If you haven't yet done so, make sure to open the training environment as described in the [Environment Setup](../../envsetup/index.md). + +[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/nextflow-io/training?quickstart=1&ref=master) + +#### Move into the project directory + +Let's move into the directory where the files for this tutorial are located. + +```bash +cd side-quests/debugging +``` + +You can set VSCode to focus on this directory: + +```bash +code . +``` + +#### Review the materials + +You'll find a set of example workflows with various types of bugs that we'll use for practice: + +??? abstract "Directory contents" + + ```console + . + ├── bad_bash_var.nf + ├── bad_channel_shape.nf + ├── bad_channel_shape_viewed_debug.nf + ├── bad_channel_shape_viewed.nf + ├── bad_number_inputs.nf + ├── badpractice_syntax.nf + ├── bad_resources.nf + ├── bad_syntax.nf + ├── buggy_workflow.nf + ├── data + │ ├── sample_001.fastq.gz + │ ├── sample_002.fastq.gz + │ ├── sample_003.fastq.gz + │ ├── sample_004.fastq.gz + │ ├── sample_005.fastq.gz + │ └── sample_data.csv + ├── exhausted.nf + ├── invalid_process.nf + ├── missing_output.nf + ├── missing_software.nf + ├── missing_software_with_stub.nf + ├── nextflow.config + ├── no_such_var.nf + └── sample_processing.nf + ``` + +These files represent common debugging scenarios you'll encounter in real-world development. + +#### Review the assignment + +Your challenge is to run each workflow, identify the error(s), and fix them. + +For each buggy workflow: + +1. **Run the workflow** and observe the error +2. **Analyze the error message**: what is Nextflow telling you? +3. **Locate the problem** in the code using the clues provided +4. **Fix the bug** and verify your solution works +5. **Reset the file** before moving to the next section (use `git checkout `) + +The exercises progress from simple syntax errors to more subtle runtime issues. +Solutions are discussed inline, but try to solve each one yourself before reading ahead. + +#### Readiness checklist + +Think you're ready to dive in? + +- [ ] My codespace is up and running +- [ ] I've set my working directory appropriately +- [ ] I understand the assignment + +If you can check all the boxes, you're good to go. + +--- + +## 1. Syntax Errors + +Syntax errors are the most common type of error you'll encounter when writing Nextflow code. They occur when the code does not conform to the expected syntax rules of the Nextflow DSL. These errors prevent your workflow from running at all, so it's important to learn how to identify and fix them quickly. + +### 1.1. Missing braces + +One of the most common syntax errors, and sometimes one of the more complex ones to debug is **missing or mismatched brackets**. + +Let's start with a practical example. + +#### Run the pipeline + +```bash +nextflow run bad_syntax.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_syntax.nf` [stupefied_bhabha] DSL2 - revision: ca6327fad2 + + Error bad_syntax.nf:24:1: Unexpected input: '' + + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details + ``` + +**Key elements of syntax error messages:** + +- **File and location**: Shows which file and line/column contain the error (`bad_syntax.nf:24:1`) +- **Error description**: Explains what the parser found that it didn't expect (`Unexpected input: ''`) +- **EOF indicator**: The `` (End Of File) message indicates the parser reached the end of the file while still expecting more content - a classic sign of unclosed braces + +#### Check the code + +Now, let's examine `bad_syntax.nf` to understand what's causing the error: + +```groovy title="bad_syntax.nf" hl_lines="14" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ +// Missing closing brace for the process + +workflow { + + // Create input channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + // Call the process with the input channel + PROCESS_FILES(input_ch) +} +``` + +For the purpose of this example we've left a comment for you to show where the error is. The Nextflow VSCode extension should also be giving you some hints about what might be wrong, putting the mismatched brace in red and highlighting the premature end of the file: + +![Bad syntax](../img/bad_syntax.png) + +**Debugging strategy for bracket errors:** + +1. Use VS Code's bracket matching (place cursor next to a bracket) +2. Check the Problems panel for bracket-related messages +3. Ensure each opening `{` has a corresponding closing `}` + +#### Fix the code + +Replace the comment with the missing closing brace: + +=== "After" + + ```groovy title="bad_syntax.nf" hl_lines="14" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } // Add the missing closing brace + + workflow { + + // Create input channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + // Call the process with the input channel + PROCESS_FILES(input_ch) + } + ``` + +=== "Before" + + ```groovy title="bad_syntax.nf" hl_lines="14" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + // Missing closing brace for the process + + workflow { + + // Create input channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + // Call the process with the input channel + PROCESS_FILES(input_ch) + } + ``` + +#### Run the pipeline + +Now run the workflow again to confirm it works: + +```bash +nextflow run bad_syntax.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_syntax.nf` [insane_faggin] DSL2 - revision: 961938ee2b + + executor > local (3) + [48/cd7f54] PROCESS_FILES (1) | 3 of 3 ✔ + ``` + +### 1.2. Using incorrect process keywords or directives + +Another common syntax error is an **invalid process definition**. This can happen if you forget to define required blocks or use incorrect directives in the process definition. + +#### Run the pipeline + +```bash +nextflow run invalid_process.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `invalid_process.nf` [nasty_jepsen] DSL2 - revision: da9758d614 + + Error invalid_process.nf:3:1: Invalid process definition -- check for missing or out-of-order section labels + │ 3 | process PROCESS_FILES { + │ | ^^^^^^^^^^^^^^^^^^^^^^^ + │ 4 | inputs: + │ 5 | val sample_name + │ 6 | + ╰ 7 | output: + + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details + ``` + +#### Check the code + +The error indicates an "Invalid process definition" and shows the context around the problem. Looking at lines 3-7, we can see `inputs:` on line 4, which is the issue. Let's examine `invalid_process.nf`: + +```groovy title="invalid_process.nf" hl_lines="4" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + inputs: // ERROR: Should be 'input' not 'inputs' + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ +} + +workflow { + + // Create input channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + // Call the process with the input channel + PROCESS_FILES(input_ch) +} +``` + +Looking at line 4 in the error context, we can spot the issue: we're using `inputs` instead of the correct `input` directive. The Nextflow VSCode extension will also flag this: + +![Invalid process message](../img/invalid_process_message.png) + +#### Fix the code + +Replace the incorrect keyword with the correct one by referencing [the documentation](https://www.nextflow.io/docs/latest/process.html#): + +=== "After" + + ```groovy title="invalid_process.nf" hl_lines="4" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: // Fixed: Changed 'inputs' to 'input' + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + + workflow { + + // Create input channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + // Call the process with the input channel + PROCESS_FILES(input_ch) + } + ``` + +=== "Before" + + ```groovy title="invalid_process.nf" hl_lines="4" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + inputs: // ERROR: Should be 'input' not 'inputs' + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + + workflow { + + // Create input channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + // Call the process with the input channel + PROCESS_FILES(input_ch) + } + ``` + +#### Run the pipeline + +Now run the workflow again to confirm it works: + +```bash +nextflow run invalid_process.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `invalid_process.nf` [silly_fermi] DSL2 - revision: 961938ee2b + + executor > local (3) + [b7/76cd9d] PROCESS_FILES (2) | 3 of 3 ✔ + ``` + +### 1.3. Using bad variable names + +The variable names you use in your script blocks must be valid, derived either from inputs or from groovy code inserted before the script. But when you're wrangling complexity at the start of pipeline development, it's easy to make mistakes in variable naming, and Nextflow will let you know quickly. + +#### Run the pipeline + +```bash +nextflow run no_such_var.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `no_such_var.nf` [gloomy_meninsky] DSL2 - revision: 0c4d3bc28c + + Error no_such_var.nf:17:39: `undefined_var` is not defined + │ 17 | echo "Using undefined variable: ${undefined_var}" >> ${output_pref + ╰ | ^^^^^^^^^^^^^ + + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details + ``` + +The error is caught at compile time and points directly to the undefined variable on line 17, with a caret indicating exactly where the problem is. + +#### Check the code + +Let's examine `no_such_var.nf`: + +```groovy title="no_such_var.nf" hl_lines="17" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_processed.txt" + + script: + // Define variables in Groovy code before the script + def output_prefix = "${sample_name}_processed" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + echo "Using undefined variable: ${undefined_var}" >> ${output_prefix}.txt // ERROR: undefined_var not defined + """ +} + +workflow { + input_ch = channel.of('sample1', 'sample2', 'sample3') + PROCESS_FILES(input_ch) +} +``` + +The error message indicates that the variable is not recognized in the script template, and there you go- you should be able to see `#!groovy ${undefined_var}` used in the script block, but not defined elsewhere. + +#### Fix the code + +If you get a 'No such variable' error, you can fix it by either defining the variable (by correcting input variable names or editing groovy code before the script), or by removing it from the script block if it's not needed: + +=== "After" + + ```groovy title="no_such_var.nf" hl_lines="15-17" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_processed.txt" + + script: + // Define variables in Groovy code before the script + def output_prefix = "${sample_name}_processed" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + """ // Removed the line with undefined_var + } + + workflow { + input_ch = channel.of('sample1', 'sample2', 'sample3') + PROCESS_FILES(input_ch) + } + ``` + +=== "Before" + + ```groovy title="no_such_var.nf" hl_lines="17" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + // Define variables in Groovy code before the script + def output_prefix = "${sample_name}_processed" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + echo "Using undefined variable: ${undefined_var}" >> ${output_prefix}.txt // ERROR: undefined_var not defined + """ + } + + workflow { + input_ch = channel.of('sample1', 'sample2', 'sample3') + PROCESS_FILES(input_ch) + } + ``` + +#### Run the pipeline + +Now run the workflow again to confirm it works: + +```bash +nextflow run no_such_var.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `no_such_var.nf` [suspicious_venter] DSL2 - revision: 6ba490f7c5 + + executor > local (3) + [21/237300] PROCESS_FILES (2) | 3 of 3 ✔ + ``` + +### 1.4. Bad use of Bash variables + +Starting out in Nextflow, it can be difficult to understand the difference between Nextflow (Groovy) and Bash variables. This can generate another form of the bad variable error that appears when trying to use variables in the Bash content of the script block. + +#### Run the pipeline + +```bash +nextflow run bad_bash_var.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_bash_var.nf` [infallible_mandelbrot] DSL2 - revision: 0853c11080 + + Error bad_bash_var.nf:13:42: `prefix` is not defined + │ 13 | echo "Processing ${sample_name}" > ${prefix}.txt + ╰ | ^^^^^^ + + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details + ``` + +#### Check the code + +The error points to line 13 where `#!groovy ${prefix}` is used. Let's examine `bad_bash_var.nf` to see what's causing the issue: + +```groovy title="bad_bash_var.nf" hl_lines="13" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + prefix="${sample_name}_output" + echo "Processing ${sample_name}" > ${prefix}.txt # ERROR: ${prefix} is Groovy syntax, not Bash + """ +} +``` + +In this example, we're defining the `prefix` variable in Bash, but in a Nextflow process the `$` syntax we used to refer to it (`#!groovy ${prefix}`) is interpreted as a Groovy variable, not Bash. The variable doesn't exist in the Groovy context, so we get a 'no such variable' error. + +#### Fix the code + +If you want to use a Bash variable, you must escape the dollar sign like this: + +=== "After" + + ```groovy title="bad_bash_var.nf" hl_lines="13" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + prefix="${sample_name}_output" + echo "Processing ${sample_name}" > \${prefix}.txt # Fixed: Escaped the dollar sign + """ + } + + workflow { + input_ch = channel.of('sample1', 'sample2', 'sample3') + PROCESS_FILES(input_ch) + } + ``` + +=== "Before" + + ```groovy title="bad_bash_var.nf" hl_lines="13" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + prefix="${sample_name}_output" + echo "Processing ${sample_name}" > ${prefix}.txt # ERROR: ${prefix} is Groovy syntax, not Bash + """ + } + ``` + +This tells Nextflow to interpret this as a Bash variable. + +#### Run the pipeline + +Now run the workflow again to confirm it works: + +```bash +nextflow run bad_bash_var.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_bash_var.nf` [naughty_franklin] DSL2 - revision: 58c1c83709 + + executor > local (3) + [4e/560285] PROCESS_FILES (2) | 3 of 3 ✔ + ``` + +!!! tip "Groovy vs Bash Variables" + + For simple variable manipulations like string concatenation or prefix/suffix operations, it's usually more readable to use Groovy variables in the script section rather than Bash variables in the script block: + + ```groovy linenums="1" + script: + def output_prefix = "${sample_name}_processed" + def output_file = "${output_prefix}.txt" + """ + echo "Processing ${sample_name}" > ${output_file} + """ + ``` + + This approach avoids the need to escape dollar signs and makes the code easier to read and maintain. + +### 1.5. Statements Outside Workflow Block + +The Nextflow VSCode extension highlights issues with code structure that will cause errors. A common example is defining channels outside of the `workflow {}` block - this is now enforced as a syntax error. + +#### Run the pipeline + +```bash +nextflow run badpractice_syntax.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `badpractice_syntax.nf` [intergalactic_colden] DSL2 - revision: 5e4b291bde + + Error badpractice_syntax.nf:3:1: Statements cannot be mixed with script declarations -- move statements into a process or workflow + │ 3 | input_ch = channel.of('sample1', 'sample2', 'sample3') + ╰ | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details + ``` + +The error message clearly indicates the problem: statements (like channel definitions) cannot be mixed with script declarations outside of a workflow or process block. + +#### Check the code + +Let's examine `badpractice_syntax.nf` to see what's causing the error: + +```groovy title="badpractice_syntax.nf" hl_lines="3" linenums="1" +#!/usr/bin/env nextflow + +input_ch = channel.of('sample1', 'sample2', 'sample3') // ERROR: Channel defined outside workflow + +process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_processed.txt" + + script: + // Define variables in Groovy code before the script + def output_prefix = "${sample_name}_processed" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + """ +} + +workflow { + PROCESS_FILES(input_ch) +} +``` + +The VSCode extension will also highlight the `input_ch` variable as being defined outside the workflow block: + +![Non-lethal syntax error](../img/nonlethal.png) + +#### Fix the code + +Move the channel definition inside the workflow block: + +=== "After" + + ```groovy title="badpractice_syntax.nf" hl_lines="21" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_processed.txt" + + script: + // Define variables in Groovy code before the script + def output_prefix = "${sample_name}_processed" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + """ + } + + workflow { + input_ch = channel.of('sample1', 'sample2', 'sample3') // Moved inside workflow block + PROCESS_FILES(input_ch) + } + ``` + +=== "Before" + + ```groovy title="badpractice_syntax.nf" hl_lines="3" linenums="1" + #!/usr/bin/env nextflow + + input_ch = channel.of('sample1', 'sample2', 'sample3') // ERROR: Channel defined outside workflow + + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_processed.txt" + + script: + // Define variables in Groovy code before the script + def output_prefix = "${sample_name}_processed" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + """ + } + + workflow { + PROCESS_FILES(input_ch) + } + ``` + +#### Run the pipeline + +Run the workflow again to confirm the fix works: + +```bash +nextflow run badpractice_syntax.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `badpractice_syntax.nf` [naughty_ochoa] DSL2 - revision: 5e4b291bde + + executor > local (3) + [6a/84a608] PROCESS_FILES (2) | 3 of 3 ✔ + ``` + +Keep your input channels defined within the workflow block, and in general follow any other recommendations the extension makes. + +### Takeaway + +You can systematically identify and fix syntax errors using Nextflow error messages and IDE visual indicators. Common syntax errors include missing braces, incorrect process keywords, undefined variables, and improper use of Bash vs. Nextflow variables. The VSCode extension helps catch many of these before runtime. With these syntax debugging skills in your toolkit, you'll be able to quickly resolve the most common Nextflow syntax errors and move on to tackling more complex runtime issues. + +### What's next? + +Learn to debug more complex channel structure errors that occur even when syntax is correct. + +--- + +## 2. Channel Structure Errors + +Channel structure errors are more subtle than syntax errors because the code is syntactically correct, but the data shapes don't match what processes expect. Nextflow will try to run the pipeline, but might find that the number of inputs doesn't match what it expects and fail. These errors typically only appear at runtime and require an understanding of the data flowing through your workflow. + +!!! tip "Debugging Channels with `.view()`" + + Throughout this section, remember that you can use the `.view()` operator to inspect channel content at any point in your workflow. This is one of the most powerful debugging tools for understanding channel structure issues. We'll explore this technique in detail in section 2.4, but feel free to use it as you work through the examples. + + ```groovy + my_channel.view() // Shows what's flowing through the channel + ``` + +### 2.1. Wrong Number of Input Channels + +This error occurs when you pass a different number of channels than a process expects. + +#### Run the pipeline + +```bash +nextflow run bad_number_inputs.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_number_inputs.nf` [happy_swartz] DSL2 - revision: d83e58dcd3 + + Error bad_number_inputs.nf:23:5: Incorrect number of call arguments, expected 1 but received 2 + │ 23 | PROCESS_FILES(samples_ch, files_ch) + ╰ | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details + ``` + +#### Check the code + +The error message clearly states that the call expected 1 argument but received 2, and points to line 23. Let's examine `bad_number_inputs.nf`: + +```groovy title="bad_number_inputs.nf" hl_lines="5 23" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + val sample_name // Process expects only 1 input + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ +} + +workflow { + + // Create two separate channels + samples_ch = channel.of('sample1', 'sample2', 'sample3') + files_ch = channel.of('file1.txt', 'file2.txt', 'file3.txt') + + // ERROR: Passing 2 channels but process expects only 1 + PROCESS_FILES(samples_ch, files_ch) +} +``` + +You should see the mismatched `PROCESS_FILES` call, supplying multiple input channels when the process only defines one. The VSCode extension will also under line process call in red, and supply a diagnostic message when you mouse over: + +![Incorrect number of args message](../img/incorrect_num_args.png) + +#### Fix the code + +For this specific example, the process expects a single channel and doesn't require the second channel, so we can fix it by passing only the `samples_ch` channel: + +=== "After" + + ```groovy title="bad_number_inputs.nf" hl_lines="23" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name // Process expects only 1 input + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + + workflow { + + // Create two separate channels + samples_ch = channel.of('sample1', 'sample2', 'sample3') + files_ch = channel.of('file1.txt', 'file2.txt', 'file3.txt') + + // Fixed: Pass only the channel the process expects + PROCESS_FILES(samples_ch) + } + ``` + +=== "Before" + + ```groovy title="bad_number_inputs.nf" hl_lines="5 23" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name // Process expects only 1 input + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + + workflow { + + // Create two separate channels + samples_ch = channel.of('sample1', 'sample2', 'sample3') + files_ch = channel.of('file1.txt', 'file2.txt', 'file3.txt') + + // ERROR: Passing 2 channels but process expects only 1 + PROCESS_FILES(samples_ch, files_ch) + } + ``` + +#### Run the pipeline + +```bash +nextflow run bad_number_inputs.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_number_inputs.nf` [big_euler] DSL2 - revision: e302bd87be + + executor > local (3) + [48/497f7b] PROCESS_FILES (3) | 3 of 3 ✔ + ``` + +More commonly than this example, you might add additional inputs to a process and forget to update the workflow call accordingly, which can lead to this type of error. Fortunately, this is one of the easier-to-understand and fix errors, as the error message is quite clear about the mismatch. + +### 2.2. Channel Exhaustion (Process Runs Fewer Times Than Expected) + +Some channel structure errors are much more subtle and produce no errors at all. Probably the most common of these reflects a challenge that new Nextflow users face in understanding that queue channels can be exhausted and run out of items, meaning the workflow finishes prematurely. + +#### Run the pipeline + +```bash +nextflow run exhausted.nf +``` + +??? success "Command output" + +```console title="Exhausted channel output" + N E X T F L O W ~ version 25.10.4 + +Launching `exhausted.nf` [extravagant_gauss] DSL2 - revision: 08cff7ba2a + +executor > local (1) +[bd/f61fff] PROCESS_FILES (1) [100%] 1 of 1 ✔ +``` + +This workflow completes without error, but it only processes a single sample! + +#### Check the code + +Let's examine `exhausted.nf` to see if that's right: + +```groovy title="exhausted.nf" hl_lines="23 24" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + val reference + val sample_name + + output: + path "${output_prefix}.txt" + + script: + // Define variables in Groovy code before the script + output_prefix = "${reference}_${sample_name}" + def timestamp = new Date().format("yyyy-MM-dd") + + """ + echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt + """ +} + +workflow { + + reference_ch = channel.of('baseline_reference') + input_ch = channel.of('sample1', 'sample2', 'sample3') + + PROCESS_FILES(reference_ch, input_ch) +} +``` + +The process only runs once instead of three times because the `reference_ch` channel is a queue channel that gets exhausted after the first process execution. When one channel is exhausted, the entire process stops, even if other channels still have items. + +This is a common pattern where you have a single reference file that needs to be reused across multiple samples. The solution is to convert the reference channel to a value channel that can be reused indefinitely. + +#### Fix the code + +There are a couple of ways to address this depending on how many files are affected. + +**Option 1**: You have a single reference file that you are re-using a lot. You can simply create a value channel type, which can be used over and over again. There are three ways to do this: + +**1a** Use `channel.value()`: + +```groovy title="exhausted.nf (fixed - Option 1a)" hl_lines="2" linenums="21" +workflow { + reference_ch = channel.value('baseline_reference') // Value channel can be reused + input_ch = channel.of('sample1', 'sample2', 'sample3') + + PROCESS_FILES(reference_ch, input_ch) +} +``` + +**1b** Use the `first()` [operator](https://www.nextflow.io/docs/latest/reference/operator.html#first): + +```groovy title="exhausted.nf (fixed - Option 1b)" hl_lines="2" linenums="21" +workflow { + reference_ch = channel.of('baseline_reference').first() // Convert to value channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + PROCESS_FILES(reference_ch, input_ch) +} +``` + +**1c.** Use the `collect()` [operator](https://www.nextflow.io/docs/latest/reference/operator.html#collect): + +```groovy title="exhausted.nf (fixed - Option 1c)" hl_lines="2" linenums="21" +workflow { + reference_ch = channel.of('baseline_reference').collect() // Convert to value channel + input_ch = channel.of('sample1', 'sample2', 'sample3') + + PROCESS_FILES(reference_ch, input_ch) +} +``` + +**Option 2**: In more complex scenarios, perhaps where you have multiple reference files for all samples in the sample channel, you can use the `combine` operator to create a new channel that combines the two channels into tuples: + +```groovy title="exhausted.nf (fixed - Option 2)" hl_lines="4" linenums="21" +workflow { + reference_ch = channel.of('baseline_reference','other_reference') + input_ch = channel.of('sample1', 'sample2', 'sample3') + combined_ch = reference_ch.combine(input_ch) // Creates cartesian product + + PROCESS_FILES(combined_ch) +} +``` + +The `.combine()` operator generates a cartesian product of the two channels, so each item in `reference_ch` will be paired with each item in `input_ch`. This allows the process to run for each sample while still using the reference. + +This requires the process input to be adjusted. In our example, the start of the process definition would need to be adjusted as follows: + +```groovy title="exhausted.nf (fixed - Option 2)" hl_lines="5" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + tuple val(reference), val(sample_name) +``` + +This approach may not be suitable in all situations. + +#### Run the pipeline + +Try one of the fixes above and run the workflow again: + +```bash +nextflow run exhausted.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `exhausted.nf` [maniac_leavitt] DSL2 - revision: f372a56a7d + + executor > local (3) + [80/0779e9] PROCESS_FILES (3) | 3 of 3 ✔ + ``` + +You should now see all three samples being processed instead of just one. + +### 2.3. Wrong Channel Content Structure + +When workflows reach a certain level of complexity, it can be a little difficult to keep track of the internal structures of each channel, and people commonly generate mismatches between what the process expects and what the channel actually contains. This is more subtle than the issue we discussed earlier, where the number of channels was incorrect. In this case, you can have the correct number of input channels, but the internal structure of one or more of those channels doesn't match what the process expects. + +#### Run the pipeline + +```bash +nextflow run bad_channel_shape.nf +``` + +??? failure "Command output" + + ```console + Launching `bad_channel_shape.nf` [hopeful_pare] DSL2 - revision: ffd66071a1 + + executor > local (3) + executor > local (3) + [3f/c2dcb3] PROCESS_FILES (3) [ 0%] 0 of 3 ✘ + ERROR ~ Error executing process > 'PROCESS_FILES (1)' + + Caused by: + Missing output file(s) `[sample1, file1.txt]_output.txt` expected by process `PROCESS_FILES (1)` + + + Command executed: + + echo "Processing [sample1, file1.txt]" > [sample1, file1.txt]_output.txt + + Command exit status: + 0 + + Command output: + (empty) + + Work dir: + /workspaces/training/side-quests/debugging/work/d6/1fb69d1d93300bbc9d42f1875b981e + + Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line + + -- Check '.nextflow.log' file for details + ``` + +#### Check the code + +The square brackets in the error message provide the clue here - the process is treating the tuple as a single value, which is not what we want. Let's examine `bad_channel_shape.nf`: + +```groovy title="bad_channel_shape.nf" hl_lines="5 20-22" linenums="1" +#!/usr/bin/env nextflow + +process PROCESS_FILES { + input: + val sample_name // Expects single value, gets tuple + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ +} + +workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'] + ) + PROCESS_FILES(input_ch) +} +``` + +You can see that we're generating a channel composed of tuples: `['sample1', 'file1.txt']`, but the process expects a single value, `val sample_name`. The command executed shows that the process is trying to create a file named `[sample3, file3.txt]_output.txt`, which is not the intended output. + +#### Fix the code + +To fix this, if the process requires both inputs we could adjust the process to accept a tuple: + +=== "Option 1: Accept tuple in process" + + === "After" + + ```groovy title="bad_channel_shape.nf" hl_lines="5" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + tuple val(sample_name), val(file_name) // Fixed: Accept tuple + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + + workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'] + ) + PROCESS_FILES(input_ch) + } + ``` + + === "Before" + + ```groovy title="bad_channel_shape.nf" hl_lines="5" linenums="1" + #!/usr/bin/env nextflow + + process PROCESS_FILES { + input: + val sample_name // Expects single value, gets tuple + + output: + path "${sample_name}_output.txt" + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + + workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'] + ) + PROCESS_FILES(input_ch) + } + ``` + +=== "Option 2: Extract first element" + + === "After" + + ```groovy title="bad_channel_shape.nf" hl_lines="9" linenums="16" + workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'] + ) + PROCESS_FILES(input_ch.map { it[0] }) // Fixed: Extract first element + } + ``` + + === "Before" + + ```groovy title="bad_channel_shape.nf" hl_lines="9" linenums="16" + workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'] + ) + PROCESS_FILES(input_ch) + } + ``` + +#### Run the pipeline + +Pick one of the solutions and re-run the workflow: + +```bash +nextflow run bad_channel_shape.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_channel_shape.nf` [clever_thompson] DSL2 - revision: 8cbcae3746 + + executor > local (3) + [bb/80a958] PROCESS_FILES (2) | 3 of 3 ✔ + ``` + +### 2.4. Channel Debugging Techniques + +#### Using `.view()` for Channel Inspection + +The most powerful debugging tool for channels is the `.view()` operator. With `.view()`, you can understand the shape of your channels at all stages to help with debugging. + +#### Run the pipeline + +Run `bad_channel_shape_viewed.nf` to see this in action: + +```bash +nextflow run bad_channel_shape_viewed.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_channel_shape_viewed.nf` [maniac_poisson] DSL2 - revision: b4f24dc9da + + executor > local (3) + [c0/db76b3] PROCESS_FILES (3) [100%] 3 of 3 ✔ + Channel content: [sample1, file1.txt] + Channel content: [sample2, file2.txt] + Channel content: [sample3, file3.txt] + After mapping: sample1 + After mapping: sample2 + After mapping: sample3 + ``` + +#### Check the code + +Let's examine `bad_channel_shape_viewed.nf` to see how `.view()` is used: + +```groovy title="bad_channel_shape_viewed.nf" linenums="16" hl_lines="9 11" +workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'] + ) + .view { "Channel content: $it" } // Debug: Show original channel content + .map { tuple -> tuple[0] } // Transform: Extract first element + .view { "After mapping: $it" } // Debug: Show transformed channel content + + PROCESS_FILES(input_ch) +} +``` + +#### Fix the code + +To save you from using `.view()` operations excessively in future to understand channel content, it's advisable to add some comments to help: + +```groovy title="bad_channel_shape_viewed.nf (with comments)" linenums="16" hl_lines="8 9" +workflow { + + // Channel emits tuples, but process expects single values + input_ch = channel.of( + ['sample1', 'file1.txt'], + ['sample2', 'file2.txt'], + ['sample3', 'file3.txt'], + ) // [sample_name, file_name] + .map { tuple -> tuple[0] } // sample_name + + PROCESS_FILES(input_ch) +} +``` + +This will become more important as your workflows grow in complexity and channel structure becomes more opaque. + +#### Run the pipeline + +```bash +nextflow run bad_channel_shape_viewed.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_channel_shape_viewed.nf` [marvelous_koch] DSL2 - revision: 03e79cdbad + + executor > local (3) + [ff/d67cec] PROCESS_FILES (2) | 3 of 3 ✔ + Channel content: [sample1, file1.txt] + Channel content: [sample2, file2.txt] + Channel content: [sample3, file3.txt] + After mapping: sample1 + After mapping: sample2 + After mapping: sample3 + ``` + +### Takeaway + +Many channel structure errors can be created with valid Nextflow syntax. You can debug channel structure errors by understanding data flow, using `.view()` operators for inspection, and recognizing error message patterns like square brackets indicating unexpected tuple structures. + +### What's next? + +Learn about errors created by process definitions. + +--- + +## 3. Process Structure Errors + +Most of the errors you encounter related to processes will related to mistakes you have made in forming the command, or to issues related to the underlying software. That said, similarly to the channel issues above, you can make mistakes in the process definition that don't quality as syntax errors, but which will cause errors at run time. + +### 3.1. Missing Output Files + +One common error when writing processes is to do something that generates a mismatch between what the process expects and what is generated. + +#### Run the pipeline + +```bash +nextflow run missing_output.nf +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `missing_output.nf` [zen_stone] DSL2 - revision: 37ff61f926 + + executor > local (3) + executor > local (3) + [fd/2642e9] process > PROCESS_FILES (2) [ 66%] 2 of 3, failed: 2 + ERROR ~ Error executing process > 'PROCESS_FILES (3)' + + Caused by: + Missing output file(s) `sample3.txt` expected by process `PROCESS_FILES (3)` + + + Command executed: + + echo "Processing sample3" > sample3_output.txt + + Command exit status: + 0 + + Command output: + (empty) + + Work dir: + /workspaces/training/side-quests/debugging/work/02/9604d49fb8200a74d737c72a6c98ed + + Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line + + -- Check '.nextflow.log' file for details + ``` + +#### Check the code + +The error message indicates that the process expected to produce an output file named `sample3.txt`, but the script actually creates `sample3_output.txt`. Let's examine the process definition in `missing_output.nf`: + +```groovy title="missing_output.nf" linenums="3" hl_lines="6 10" +process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}.txt" // Expects: sample3.txt + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt // Creates: sample3_output.txt + """ +} +``` + +You should see that there is a mismatch between the output file name in the `output:` block, and the one used in the script. This mismatch causes the process to fail. If you encounter this sort of error, go back and check that the outputs match between your process definition and your output block. + +If the problem still isn't clear, check the work directory itself to identify the actual output files created: + +```bash +❯ ls -h work/02/9604d49fb8200a74d737c72a6c98ed +sample3_output.txt +``` + +For this example this would highlight to us that a `_output` suffix is being incorporated into the output file name, contrary to our `output:` definition. + +#### Fix the code + +Fix the mismatch by making the output filename consistent: + +=== "After" + + ```groovy title="missing_output.nf" hl_lines="6 10" linenums="3" + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}_output.txt" // Fixed: Match the script output + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt + """ + } + ``` + +=== "Before" + + ```groovy title="missing_output.nf" hl_lines="6 10" linenums="3" + process PROCESS_FILES { + input: + val sample_name + + output: + path "${sample_name}.txt" // Expects: sample3.txt + + script: + """ + echo "Processing ${sample_name}" > ${sample_name}_output.txt // Creates: sample3_output.txt + """ + } + ``` + +#### Run the pipeline + +```bash +nextflow run missing_output.nf +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `missing_output.nf` [elated_hamilton] DSL2 - revision: 961938ee2b + + executor > local (3) + [16/1c437c] PROCESS_FILES (3) | 3 of 3 ✔ + ``` + +### 3.2. Missing software + +Another class of errors occurs due to mistakes in software provisioning. `missing_software.nf` is a syntactically valid workflow, but it depends on some external software to provide the `cowpy` command it uses. + +#### Run the pipeline + +```bash +nextflow run missing_software.nf +``` + +??? failure "Command output" + + ```console hl_lines="12 18" + ERROR ~ Error executing process > 'PROCESS_FILES (3)' + + Caused by: + Process `PROCESS_FILES (3)` terminated with an error exit status (127) + + + Command executed: + + cowpy sample3 > sample3_output.txt + + Command exit status: + 127 + + Command output: + (empty) + + Command error: + .command.sh: line 2: cowpy: command not found + + Work dir: + /workspaces/training/side-quests/debugging/work/82/42a5bfb60c9c6ee63ebdbc2d51aa6e + + Tip: you can try to figure out what's wrong by changing to the process work directory and showing the script file named `.command.sh` + + -- Check '.nextflow.log' file for details + ``` + +The process doesn't have access to the command we're specifying. Sometimes this is because a script is present in the workflow `bin` directory, but has not been made executable. Other times it is because the software is not installed in the container or environment where the workflow is running. + +#### Check the code + +Look out for that `127` exit code - it tells you exactly the problem. Let's examine `missing_software.nf`: + +```groovy title="missing_software.nf" linenums="3" hl_lines="3" +process PROCESS_FILES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + cowpy ${sample_name} > ${sample_name}_output.txt + """ +} +``` + +#### Fix the code + +We've been a little disingenuous here, and there's actually nothing wrong with the code. We just need to specify the necessary configuration to run the process in such a way that it has access to the command in question. In this case the process has a container definition, so all we need to do is run the workflow with Docker enabled. + +#### Run the pipeline + +We've set up a Docker profile for you in `nextflow.config`, so you can run the workflow with: + +```bash +nextflow run missing_software.nf -profile docker +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `missing_software.nf` [awesome_stonebraker] DSL2 - revision: 0296d12839 + + executor > local (3) + [38/ab20d1] PROCESS_FILES (1) | 3 of 3 ✔ + ``` + +!!! note + + To learn more about how Nextflow uses containers, see [Hello Nextflow](../../hello_nextflow/05_hello_containers.md) + +### 3.3. Bad resource configuration + +In production usage, you'll be configuring resources on your processes. For example `memory` defines the maximum amount of memory available to your process, and if the process exceeds that, your scheduler will typically kill the process and return an exit code of `137`. We can't demonstrate that here because we're using the `local` executor, but we can show something similar with `time`. + +#### Run the pipeline + +`bad_resources.nf` has process configuration with an unrealistic bound on time of 1 millisecond: + +```bash +nextflow run bad_resources.nf -profile docker +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_resources.nf` [disturbed_elion] DSL2 - revision: 27d2066e86 + + executor > local (3) + [c0/ded8e1] PROCESS_FILES (3) | 0 of 3 ✘ + ERROR ~ Error executing process > 'PROCESS_FILES (2)' + + Caused by: + Process exceeded running time limit (1ms) + + Command executed: + + cowpy sample2 > sample2_output.txt + + Command exit status: + - + + Command output: + (empty) + + Work dir: + /workspaces/training/side-quests/debugging/work/53/f0a4cc56d6b3dc2a6754ff326f1349 + + Container: + community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273 + + Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` + + -- Check '.nextflow.log' file for details + ``` + +#### Check the code + +Let's examine `bad_resources.nf`: + +```groovy title="bad_resources.nf" linenums="3" hl_lines="3" +process PROCESS_FILES { + + time '1 ms' // ERROR: Unrealistic time limit + + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + sleep 1 # Takes 1 second, but time limit is 1ms + cowpy ${sample_name} > ${sample_name}_output.txt + """ +} +``` + +We know the process will take longer than a second (we've added a sleep in there to make sure), but the process is set to time out after 1 millisecond. Someone has been a little unrealistic with their configuration! + +#### Fix the code + +Increase the time limit to a realistic value: + +=== "After" + + ```groovy title="bad_resources.nf" hl_lines="3" linenums="3" + process PROCESS_FILES { + + time '100 s' // Fixed: Realistic time limit + + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + sleep 1 + cowpy ${sample_name} > ${sample_name}_output.txt + """ + } + ``` + +=== "Before" + + ```groovy title="bad_resources.nf" hl_lines="3" linenums="3" + process PROCESS_FILES { + + time '1 ms' // ERROR: Unrealistic time limit + + input: + val sample_name + + output: + path "${sample_name}_output.txt" + + script: + """ + sleep 1 # Takes 1 second, but time limit is 1ms + cowpy ${sample_name} > ${sample_name}_output.txt + """ + } + ``` + +#### Run the pipeline + +```bash +nextflow run bad_resources.nf -profile docker +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `bad_resources.nf` [friendly_mcclintock] DSL2 - revision: 381567d2c1 + + executor > local (3) + [c2/9b4c41] PROCESS_FILES (3) | 3 of 3 ✔ + ``` + +If you make sure to read your error messages failures like this should not puzzle you for too long. But make sure you understand the resource requirements of the commands you are running so that you can configure your resource directives appropriately. + +### Takeaway + +Most process errors are accessible from the error message itself. +Missing output errors point to filename mismatches between `output:` declarations and what your script actually produces. +Exit code 127 means the command isn't on PATH (almost always a software provisioning problem). +Resource-related kills (commonly exit code 137 for memory, or "exceeded running time limit" for time) are about your configuration, not your code. + +### What's next? + +Move on to [Part 2: The Nextflow debugging toolkit](02_debugging_toolkit.md) to learn the tools you reach for when the error message isn't enough on its own. + +--- + +## Summary + +In this lesson, you learned to recognise and fix the most common categories of Nextflow error. + +| Category | Hallmark | Where to look | +| -------------------------- | ----------------------------------------------------------------------- | -------------------------------------- | +| Missing braces | `Unexpected input: ''` | The last process or workflow block | +| Bad keyword | `Invalid process definition` | The line cited in the error | +| Undefined variable | `\`name\` is not defined` | The script block or surrounding Groovy | +| Bash vs. Groovy variable | `is not defined` on a name defined in Bash | Add `\$` to escape Bash variables | +| Statement outside workflow | `Statements cannot be mixed with script declarations` | Move into a `workflow { }` block | +| Wrong channel cardinality | `Process requires N channels, M were specified` | The process call site | +| Wrong tuple shape | Output values with stray `[ ]` brackets, or `Path value cannot be null` | The `.map { }` feeding the process | +| Missing output | `Missing output file(s)` | Filename in `output:` vs. script | +| Missing software | Exit code `127`, `command not found` | The process container/conda | +| Resource exceeded | Exit code `137`, or `exceeded running time limit` | Process `memory` / `time` directives | + +When the error message doesn't pinpoint the problem this clearly, you'll need the tools covered in Part 2. + +[Continue to Part 2 :material-arrow-right:](02_debugging_toolkit.md){ .md-button .md-button--primary } diff --git a/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md b/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md new file mode 100644 index 0000000000..c631160704 --- /dev/null +++ b/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md @@ -0,0 +1,1019 @@ +# Part 2: The Nextflow debugging toolkit + +When an error message doesn't immediately tell you what's wrong, you reach for the toolkit. + +This lesson introduces six techniques, applied in order to the same small pipeline: + +1. **Work-directory forensics** when a process fails. +2. **`-preview`** to validate workflow logic before running anything. +3. **`debug true`** to stream process output as it runs. +4. **`-stub-run`** to iterate on workflow logic without running the real commands. +5. **`-dump-hashes`** to find out why `-resume` re-ran something you thought was cached. +6. **A systematic methodology** that ties the tools together. + +Each section uses the same pipeline so you can see how the tools fit together in a real development workflow. + +--- + +## 0. Get started + +Move into the working directory if you aren't already there: + +```bash +cd side-quests/debugging +``` + +The pipeline we'll use throughout this lesson is `sample_processing.nf`. +It reads a sample manifest, counts lines in each sample's gzipped FASTQ, and writes a small report: + +```groovy title="sample_processing.nf" linenums="1" +#!/usr/bin/env nextflow + +params.input = 'data/sample_data.csv' +params.outdir = 'results' + +process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + + input: + tuple val(sample_id), path(fastq) + + output: + path "${sample_id}.count.txt" + + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}_count.txt + """ + + stub: + """ + echo "${sample_id} has 0 lines" > ${sample_id}_count.txt + """ +} + +process REPORT { + + publishDir params.outdir, mode: 'copy' + + input: + path count_files + + output: + path 'report.txt' + + script: + """ + cat ${count_files} > report.txt + """ +} + +workflow { + + samples_ch = channel + .fromPath(params.input) + .splitCsv(header: true) + .map { row -> [row.sample_id, file(row.fastq_path)] } + + counts_ch = COUNT_LINES(samples_ch) + + REPORT(counts_ch.collect()) +} +``` + +The pipeline ships with one intentional bug, which we'll fix in section 1. +The rest of the sections build on the fixed pipeline. + +If at any point you want to start over, run: + +```bash +git checkout sample_processing.nf +``` + +to restore the original file. + +--- + +## 1. Work-directory forensics + +When a process fails, Nextflow creates a work directory containing everything that ran: the command, its output, its error stream, and its exit code. +This is the first place to look when an error message doesn't tell you enough on its own. + +### 1.1. Run the pipeline + +```bash +nextflow run sample_processing.nf -profile docker +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `sample_processing.nf` [chaotic_meninsky] DSL2 - revision: 4a8a16d3a2 + + executor > local (5) + [34/6278a1] COUNT_LINES (3) | 0 of 5 ✘ + [- ] REPORT - + ERROR ~ Error executing process > 'COUNT_LINES (5)' + + Caused by: + Missing output file(s) `sample_005.count.txt` expected by process `COUNT_LINES (5)` + + + Command executed: + + lines=$(zcat sample_005.fastq.gz | wc -l) + cowpy "sample_005 has ${lines} lines" > sample_005_count.txt + + Command exit status: + 0 + + Command output: + (empty) + + Work dir: + /workspaces/training/side-quests/debugging/work/18/8e48f60e7b68419a62909ca8c5bd3d + + Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh` + + -- Check '.nextflow.log' file for details + ``` + +The process exited cleanly (`Command exit status: 0`) but Nextflow couldn't find the output file it was promised. +The error already hints at the cause: the script writes `sample_005_count.txt`, but the process declares `sample_005.count.txt`. +Let's confirm that by looking inside the work directory. + +### 1.2. Walk the work directory + +The error message gives you the path. Copy it from your own terminal output (yours will have a different hash from ours). + +Every failed task has the same set of hidden files. Let's go through them in turn. + +#### 1.2.1. `.command.sh` — the executed command + +This is the exact script Nextflow ran, after variable substitution: + +```bash +cat work/18/8e48f60e7b68419a62909ca8c5bd3d/.command.sh +``` + +```console title="Output" +#!/bin/bash -ue +lines=$(zcat sample_005.fastq.gz | wc -l) +cowpy "sample_005 has ${lines} lines" > sample_005_count.txt +``` + +Notice that Nextflow has already substituted `${sample_id}` for `sample_005` and the redirect target is `sample_005_count.txt`. +The script ran exactly as written. + +#### 1.2.2. `.command.err` and `.command.out` — what the command printed + +```bash +cat work/18/8e48f60e7b68419a62909ca8c5bd3d/.command.err +``` + +```console title="Output" +(empty) +``` + +```bash +cat work/18/8e48f60e7b68419a62909ca8c5bd3d/.command.out +``` + +```console title="Output" +(empty) +``` + +Nothing on either stream, because `cowpy` wrote its output to the redirected file. + +#### 1.2.3. `.exitcode` — how the command exited + +```bash +cat work/18/8e48f60e7b68419a62909ca8c5bd3d/.exitcode +``` + +```console title="Output" +0 +``` + +Zero, as we already knew from the error. +This confirms the failure is on Nextflow's side, not the command's: the command succeeded, but it produced a file with a name Nextflow wasn't expecting. + +Common exit codes to recognise: + +- **0**: success (and yet here we are — usually means an output mismatch) +- **127**: command not found (software not installed in the container) +- **137**: killed by the scheduler (memory or time limit exceeded) + +#### 1.2.4. `ls` the directory — what files actually exist + +```bash +ls work/18/8e48f60e7b68419a62909ca8c5bd3d/ +``` + +```console title="Output" +sample_005.fastq.gz sample_005_count.txt +``` + +There it is: the file `sample_005_count.txt` exists, but the process expects `sample_005.count.txt`. + +### 1.3. Fix the bug + +The output declaration uses a `.` (dot), the script uses an `_` (underscore). Make them match. +Fix the script and the stub block: + +=== "After" + + ```groovy title="sample_processing.nf" hl_lines="11 16 21" linenums="6" + process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + + input: + tuple val(sample_id), path(fastq) + + output: + path "${sample_id}.count.txt" + + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + """ + + stub: + """ + echo "${sample_id} has 0 lines" > ${sample_id}.count.txt + """ + } + ``` + +=== "Before" + + ```groovy title="sample_processing.nf" hl_lines="11 16 21" linenums="6" + process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + + input: + tuple val(sample_id), path(fastq) + + output: + path "${sample_id}.count.txt" + + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}_count.txt + """ + + stub: + """ + echo "${sample_id} has 0 lines" > ${sample_id}_count.txt + """ + } + ``` + +### 1.4. Re-run and confirm + +```bash +nextflow run sample_processing.nf -profile docker -resume +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `sample_processing.nf` [friendly_tuckerman] DSL2 - revision: 62a54f6852 + + executor > local (6) + [65/82c6f9] COUNT_LINES (2) | 5 of 5 ✔ + [2d/d0078c] REPORT | 1 of 1 ✔ + ``` + +Note we used `-resume`. With this single-fault example resume doesn't save you anything, but in real pipelines it lets you retry just the failed steps after a fix. + +### Takeaway + +The work directory contains the complete record of what the process did. +`.command.sh` shows the executed command, `.command.err` and `.command.out` show its streams, `.exitcode` reveals how it finished, and a plain `ls` shows what files exist. +For any process failure: read the error, find the work directory, walk through these files until the cause is obvious. + +### What's next? + +Learn how to validate workflow logic before running anything, with `-preview`. + +--- + +## 2. Validate workflow logic with `-preview` + +Before running an expensive workflow, you can ask Nextflow to _compile_ it and report what _would_ execute, without running any tasks. +This catches syntax errors, missing files, and bad workflow structure in seconds rather than minutes. + +### 2.1. Preview the working pipeline + +```bash +nextflow run sample_processing.nf -preview +``` + +??? success "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `sample_processing.nf` [ecstatic_curran] DSL2 - revision: 62a54f6852 + + [- ] COUNT_LINES - + [- ] REPORT - + ``` + +The dashes mean "this process would run, but nothing was executed". +The script parsed cleanly, the workflow DAG is well-formed, and the two processes show up as expected. + +### 2.2. Catch a syntax error before running + +Now intentionally break the file. +Open `sample_processing.nf` and delete the closing brace at the end of the workflow block (the `}` on line 55): + +=== "Broken" + + ```groovy title="sample_processing.nf" hl_lines="10" linenums="45" + workflow { + + samples_ch = channel + .fromPath(params.input) + .splitCsv(header: true) + .map { row -> [row.sample_id, file(row.fastq_path)] } + + counts_ch = COUNT_LINES(samples_ch) + + REPORT(counts_ch.collect()) + ``` + +=== "Working" + + ```groovy title="sample_processing.nf" hl_lines="10" linenums="45" + workflow { + + samples_ch = channel + .fromPath(params.input) + .splitCsv(header: true) + .map { row -> [row.sample_id, file(row.fastq_path)] } + + counts_ch = COUNT_LINES(samples_ch) + + REPORT(counts_ch.collect()) + } + ``` + +Try `-preview` again: + +```bash +nextflow run sample_processing.nf -preview +``` + +??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `sample_processing.nf` [...] DSL2 - revision: ... + + ERROR ~ Script compilation error + - file : /workspaces/training/side-quests/debugging/sample_processing.nf + - cause: Unexpected input: '{' @ line 45, column 10. + workflow { + ^ + + 1 error + ``` + +The error is identical to what you'd see if you ran the workflow normally, but it took milliseconds and didn't pull a container or run any tasks. + +Restore the brace before moving on: + +```bash +git checkout sample_processing.nf +``` + +This also re-introduces the output-mismatch bug from section 1, so re-apply your earlier fix. + +### Takeaway + +`-preview` parses your workflow and shows the planned execution without running anything. +Use it as a fast sanity check before launching expensive pipelines or after refactoring workflow logic. + +### What's next? + +Learn how to see what a process is actually receiving as input at runtime, with `debug true`. + +--- + +## 3. Stream process output with `debug true` + +The work directory is a great forensic tool _after_ a failure, but sometimes you want to see what a process is doing _while_ it's running. +The `debug true` directive streams `stdout` and `stderr` to your terminal as the process runs. +Combined with a strategic `echo`, this gives you the same kind of visibility you'd get from `print` statements during local development. + +### 3.1. Add `debug true` and an `echo` + +Edit `sample_processing.nf` and add two lines to the COUNT_LINES process: + +=== "After" + + ```groovy title="sample_processing.nf" hl_lines="5 18" linenums="6" + process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + debug true + + input: + tuple val(sample_id), path(fastq) + + output: + path "${sample_id}.count.txt" + + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + echo "DEBUG: processed ${sample_id} (\${lines} lines)" + """ + ``` + +=== "Before" + + ```groovy title="sample_processing.nf" linenums="6" + process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + + input: + tuple val(sample_id), path(fastq) + + output: + path "${sample_id}.count.txt" + + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + """ + ``` + +The cache from section 1 will be invalidated because the script body changed (more on that in section 5), so the tasks will re-run. + +### 3.2. Run the pipeline + +```bash +nextflow run sample_processing.nf -profile docker -resume +``` + +??? success "Command output" + + ```console + executor > local (6) + [a3/5fe041] COUNT_LINES (3) | 5 of 5 ✔ + [bd/16c4b2] REPORT | 1 of 1 ✔ + DEBUG: processed sample_001 (12 lines) + + DEBUG: processed sample_002 (12 lines) + + DEBUG: processed sample_003 (12 lines) + + DEBUG: processed sample_004 (12 lines) + + DEBUG: processed sample_005 (12 lines) + ``` + +The `DEBUG:` lines come from inside the running process. +You can use this to print the value of any variable Nextflow has substituted, or any value computed by your script, at the moment the process actually sees it. +This is invaluable when the issue is "the process is doing something, but not what I expected". + +### 3.3. Reset + +`debug true` is a development tool. Remove it (and the `echo`) before continuing: + +```bash +git checkout sample_processing.nf +``` + +And re-apply the section 1 fix once more. + +### Takeaway + +`debug true` plus an `echo` inside the script gives you live visibility into what a process actually receives and computes. +Reach for it when a process completes successfully but produces something you didn't expect. + +### What's next? + +Learn how to test workflow logic when running the real commands is slow or impossible, with `-stub-run`. + +--- + +## 4. Iterate on logic with `-stub-run` + +Sometimes the real work a process does is slow, expensive, or depends on software you don't have available locally. +The `stub:` directive lets you declare a "fake" command that produces files of the right shape without doing the real computation. +Run the workflow with `-stub-run` and Nextflow uses the stubs instead of the real scripts. + +This is invaluable when you're iterating on downstream logic and don't want to wait for the upstream heavy lifting every time. + +### 4.1. Look at the stub directive + +The COUNT_LINES process in our pipeline already has one: + +```groovy title="sample_processing.nf" hl_lines="1-4" linenums="23" + stub: + """ + echo "${sample_id} has 0 lines" > ${sample_id}.count.txt + """ +``` + +The stub produces a file with the same name as the real output, but with placeholder content and no `cowpy` dependency. + +### 4.2. Run with `-stub-run` (no Docker required) + +Notice that you don't need `-profile docker`. Stubs run on the host, so they bypass the container declaration entirely: + +```bash +nextflow run sample_processing.nf -stub-run +``` + +??? success "Command output" + + ```console + executor > local (6) + [af/649b28] COUNT_LINES (1) | 5 of 5 ✔ + [ff/8fddbe] REPORT | 1 of 1 ✔ + ``` + +The pipeline runs end-to-end in seconds, producing the same output shape as the real run. +You can now iterate on REPORT (or any downstream change) without waiting on `cowpy`. + +### Takeaway + +`-stub-run` is the fastest way to validate a workflow change end-to-end. +Treat stubs as part of process design: every process you write should ship with a stub that produces the right output files. + +### What's next? + +Learn how to investigate why `-resume` didn't hit the cache when you expected it to, with `-dump-hashes`. + +--- + +## 5. Debug cache invalidation with `-dump-hashes` + +You make a small change to a pipeline, run it with `-resume`, and watch every task re-run. +Why? +Nextflow decides whether to reuse a cached task by hashing its inputs: script body, container, input values, and so on. +If any of those change, the hash changes and the cache misses. + +The `-dump-hashes` flag writes every component of every task hash to `.nextflow.log`, so you can see exactly which input changed. + +We'll run three experiments on the working pipeline, each making one small change and looking at the hash output to see what happened. + +### 5.1. Establish a baseline + +Make sure your pipeline is the working version from section 1 (output declarations and script both using `.count.txt`). +Run the pipeline once with `-dump-hashes` to populate the cache and the log: + +```bash +nextflow run sample_processing.nf -profile docker -dump-hashes +``` + +Look at the cache hash entries that were written to `.nextflow.log`: + +```bash +grep "cache hash" .nextflow.log | head +``` + +```console title="Output" +... [COUNT_LINES (4)] cache hash: 766733e20df3908928ff9e5f7cd2ba11; mode: STANDARD; entries: +... [COUNT_LINES (2)] cache hash: 5ba7de0fa92670ce4e414e3468df82ac; mode: STANDARD; entries: +... [COUNT_LINES (1)] cache hash: 1eef4a425bc5dc2ae4e06059a08404e4; mode: STANDARD; entries: +... [COUNT_LINES (5)] cache hash: 1d05640ac563902c089065adb578c616; mode: STANDARD; entries: +... [COUNT_LINES (3)] cache hash: 60025c26c50d23e7af16d9ca35b1b664; mode: STANDARD; entries: +``` + +Each task has a single hash, but it's computed from many components. +Look at the full entry for one task: + +```bash +grep -A 18 "COUNT_LINES (1)" .nextflow.log +``` + +```console title="Output" +[COUNT_LINES (1)] cache hash: 1eef4a425bc5dc2ae4e06059a08404e4; mode: STANDARD; entries: + ...UUID... [java.util.UUID] 326d1c67-b353-45b9-afda-38018e57a635 + ... [java.lang.String] COUNT_LINES + 87b5860a [java.lang.String] """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + """ + 1b554354 [java.lang.String] community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273 + ... [java.lang.String] sample_id + e245add3 [java.lang.String] sample_001 + ... [java.lang.String] fastq + 23747ba3 [nextflow.util.ArrayBag] [FileHolder(sourceObj:.../sample_001.fastq.gz, ...)] + ... [java.lang.String] $ + ... [java.lang.Boolean] true +``` + +The components include the session UUID, the process name, the full script body, the container, each input variable name and value, and a few framework constants. +Note the script body in particular: its hash (`87b5860a...`) is computed from the exact text shown. +Any change to that text - including comments and whitespace - will produce a different hash. + +!!! tip "JSON output" + + For longer pipelines, `-dump-hashes json` writes the entries in JSON format which is easier to diff with `jq` or a script. + +### 5.2. Experiment 1 — a "harmless" comment + +Add a single-line comment inside the COUNT_LINES script: + +=== "After" + + ```groovy title="sample_processing.nf" hl_lines="3" linenums="17" + script: + """ + # Count lines in the gzipped fastq + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + """ + ``` + +=== "Before" + + ```groovy title="sample_processing.nf" linenums="17" + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + """ + ``` + +Re-run with `-resume`: + +```bash +nextflow run sample_processing.nf -profile docker -resume -dump-hashes +``` + +??? "Command output" + + ```console + executor > local (6) + [da/470bba] COUNT_LINES (1) | 5 of 5 ✔ + [43/4659bf] REPORT | 1 of 1 ✔ + ``` + +Despite using `-resume`, every COUNT_LINES task ran again. REPORT ran too, because its inputs depend on COUNT_LINES outputs. + +Look at the new hash for COUNT_LINES (1): + +```bash +grep -A 18 "COUNT_LINES (1)" .nextflow.log +``` + +```console title="Output" +[COUNT_LINES (1)] cache hash: 1a1e1e398f69e0ed797c7859c6c31b90; mode: STANDARD; entries: + ... + 179bc8d7 [java.lang.String] """ + # Count lines in the gzipped fastq + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}.count.txt + """ + ... +``` + +The overall cache hash changed (`1eef4a42...` → `1a1e1e39...`). +The script hash changed too (`87b5860a...` → `179bc8d7...`) - because Nextflow hashes the script body literally, including the comment. + +**Lesson:** any change inside `script:`, `stub:`, or `shell:` blocks invalidates the cache, even comments and whitespace. + +Remove the comment before the next experiment. + +### 5.3. Experiment 2 — a resource directive + +Add a `memory` directive to COUNT_LINES: + +=== "After" + + ```groovy title="sample_processing.nf" hl_lines="5" linenums="6" + process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + memory '2.GB' + ``` + +=== "Before" + + ```groovy title="sample_processing.nf" linenums="6" + process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + ``` + +Re-run with `-resume`: + +```bash +nextflow run sample_processing.nf -profile docker -resume +``` + +??? success "Command output" + + ```console + [da/470bba] COUNT_LINES (1) | 5 of 5, cached: 5 ✔ + [43/4659bf] REPORT | 1 of 1, cached: 1 ✔ + ``` + +Everything cached. The hashes are unchanged because resource directives like `memory`, `cpus`, and `time` are not part of the cache key. + +**Lesson:** tuning resources never busts the cache. +That makes resource adjustment cheap, but it also means you can't force a re-run by changing memory or CPUs - you'd need to touch something that _is_ hashed (such as the script body) or pass `-resume ` from an earlier session. + +Remove the `memory` directive before the next experiment. + +### 5.4. Experiment 3 — a channel transformation + +Change the workflow's `.map { }` to upper-case the sample ID: + +=== "After" + + ```groovy title="sample_processing.nf" hl_lines="4" linenums="45" + workflow { + + samples_ch = channel + .fromPath(params.input) + .splitCsv(header: true) + .map { row -> [row.sample_id.toUpperCase(), file(row.fastq_path)] } + + counts_ch = COUNT_LINES(samples_ch) + + REPORT(counts_ch.collect()) + } + ``` + +=== "Before" + + ```groovy title="sample_processing.nf" hl_lines="4" linenums="45" + workflow { + + samples_ch = channel + .fromPath(params.input) + .splitCsv(header: true) + .map { row -> [row.sample_id, file(row.fastq_path)] } + + counts_ch = COUNT_LINES(samples_ch) + + REPORT(counts_ch.collect()) + } + ``` + +Re-run with `-resume`: + +```bash +nextflow run sample_processing.nf -profile docker -resume -dump-hashes +``` + +??? "Command output" + + ```console + executor > local (6) + [06/01db08] COUNT_LINES (1) | 5 of 5 ✔ + [d7/4c994c] REPORT | 1 of 1 ✔ + ``` + +Every task ran again - including REPORT, which we didn't touch. +Look at one of the new COUNT_LINES hash entries: + +```console title="Excerpt" +... [java.lang.String] sample_id +... [java.lang.String] SAMPLE_001 <-- was 'sample_001' before +... [java.lang.String] fastq +... [nextflow.util.ArrayBag] [FileHolder(...sample_001.fastq.gz...)] +``` + +The input value for `sample_id` is now `SAMPLE_001` rather than `sample_001`, so the COUNT_LINES task hash changed. +But why did REPORT re-run? +Because REPORT's input is the _files_ COUNT_LINES produced - and those files now live in different work directories (the new COUNT_LINES tasks). +A change upstream cascaded into a cache miss downstream, even though REPORT's own script and directives were untouched. + +**Lesson:** when you trace a cache miss, look upstream too. +A change to a `.map { }` or input file can quietly invalidate every downstream process whose inputs depend on the changed task's outputs. + +Restore the original `.map`: + +```bash +git checkout sample_processing.nf +``` + +(and re-apply your section 1 fix once more). + +### Takeaway + +`-dump-hashes` shows the full input set Nextflow used to compute each task's cache key. +When a `-resume` re-runs more than you expected, find the affected task in `.nextflow.log` and check which component changed. +Comments, whitespace and any text inside the script block bust the cache. Resource directives don't. Upstream input changes cascade downstream. + +### What's next? + +Tie all the tools together into a systematic methodology. + +--- + +## 6. A systematic debugging approach + +With each technique covered individually, here is how they combine into a workflow you can apply to any pipeline failure. + +### 6.1. The four-phase method + +**Phase 1 — Parse first (seconds).** +Run `nextflow run workflow.nf -preview`. +Catches syntax errors, missing process definitions, and bad workflow structure before anything expensive runs. + +**Phase 2 — Read the error (minutes).** +For runtime failures, the Nextflow error message names the failing process and includes the work directory. +Decide whether the error is structural (channel shape, missing software, wrong configuration) or process-internal (the command itself failed). + +**Phase 3 — Investigate (minutes to hours).** +For process-internal failures, walk the work directory: `.command.sh`, `.command.err`, `.command.out`, `.exitcode`, `ls`. +For structural issues, use `.view()` on the relevant channel, or add `debug true` plus an `echo` inside a process to see what's actually flowing through. +If running the real commands is slow, switch to `-stub-run` while you iterate. + +**Phase 4 — Fix and verify.** +Make the smallest change that addresses the root cause. +Re-run with `-resume`. +If the fix worked but tasks you didn't change also re-ran, use `-dump-hashes` to find out why. + +### 6.2. A debugging profile + +You can bake several of these tools into a profile so they're a single flag away: + +```groovy title="nextflow.config (debug profile)" +profiles { + debug { + process { + debug = true + cleanup = false + maxForks = 1 + } + } +} +``` + +Then run with `-profile debug` whenever you need maximum visibility. +The `cleanup = false` keeps work directories around for inspection, and `maxForks = 1` makes parallel output easier to follow. + +### Takeaway + +Reach for tools in order of cost: `-preview` is free, work-directory inspection is the first step for any process failure, `debug true` and `.view()` give you channel and runtime visibility, `-stub-run` lets you iterate fast, and `-dump-hashes` is your last resort for cache mysteries. + +### What's next? + +Apply the toolkit to an unfamiliar pipeline. + +--- + +## 7. Practical exercise + +`buggy_workflow.nf` is a different pipeline that contains several intentional bugs covering all the categories from Part 1 and this lesson. +Use the four-phase method to fix it. + +!!! exercise + + Run the workflow and start the debugging loop: + + ```bash + nextflow run buggy_workflow.nf + ``` + + ??? failure "Command output" + + ```console + N E X T F L O W ~ version 25.10.4 + + Launching `buggy_workflow.nf` [wise_ramanujan] DSL2 - revision: d51a8e83fd + + ERROR ~ Range [11, 12) out of bounds for length 11 + + -- Check '.nextflow.log' file for details + ``` + + This cryptic error indicates a parsing problem in the `params{}` block. Apply Phase 1 (preview) first. + + **Suggested approach:** + + 1. **Phase 1 — Parse first.** Use `-preview` to identify syntax issues. Fix them. + 2. **Phase 2 — Read each runtime error.** Decide whether the cause is structural or process-internal. + 3. **Phase 3 — Investigate.** For process failures, walk the work directory. For channel-shape problems, use `.view()` or `debug true`. + 4. **Phase 4 — Fix and verify.** After each fix, re-run with `-resume`. + + Stop when the workflow runs to completion. There are roughly 9 or 10 bugs depending on how you count. + + ??? solution + + The bugs in `buggy_workflow.nf`, in the order you'll typically encounter them: + + **Bug 1 — Trailing comma in output declaration** + + ```groovy linenums="21" + output: + path "${sample_id}_result.txt", // remove the trailing comma + ``` + + **Bug 2 — Missing closing brace on `processFiles`** + + Add the missing `}` after the script block of `processFiles`. + + **Bug 3 — Variable name mismatch** + + ```groovy linenums="26" + echo "Processing: ${sample}" // should be ${sample_id} + cat ${input_file} > ${sample}_result.txt // should be ${sample_id} + ``` + + **Bug 4 — Undefined channel reference** + + ```groovy linenums="87" + heavy_ch = heavyProcess(sample_ids) // sample_ids doesn't exist; use input_ch + ``` + + **Bug 5 — Wrong channel shape for `processFiles`** + + ```groovy linenums="83" + .map { row -> row.sample_id } // processFiles expects a tuple + // fix: + .map { row -> [row.sample_id, file(row.fastq_path)] } + ``` + + **Bug 6 — `heavyProcess` now receives two-element tuples it doesn't want** + + ```groovy linenums="87" + heavy_ch = heavyProcess(input_ch.map { it[0] }) + ``` + + **Bug 7 — Unescaped Bash variable** + + ```groovy linenums="48" + echo "Heavy computation \${i} for ${sample_id}" + ``` + + **Bug 8 — Unrealistic time limit** + + ```groovy linenums="36" + time '100 s' // not '1 ms' + ``` + + **Bug 9 — Output filename mismatch in `heavyProcess`** + + ```groovy linenums="49" + done > ${sample_id}_heavy.txt // not ${sample_id}.txt + ``` + + **Bug 10 — `handleFiles` is reading from `pwd` instead of an upstream channel** + + ```groovy linenums="88" + file_ch = handleFiles(heavy_ch) + ``` + + Once all are fixed, the workflow runs to completion. + +### Takeaway + +The four-phase method scales: start with cheap checks (`-preview`), let error messages drive your next move, drop into forensic detail when needed, and verify each fix with `-resume`. + +--- + +## Summary + +You learned a sequence of debugging tools, each applied to the same small pipeline. + +| Tool | Best for | +| ----------------- | ----------------------------------------------------------- | +| Work directory | Any process failure - the complete record of what ran | +| `-preview` | Catching syntax and structural issues without running tasks | +| `debug true` | Seeing what a process actually receives at runtime | +| `-stub-run` | Iterating on workflow logic without real commands | +| `-dump-hashes` | Diagnosing unexpected cache invalidation | +| Four-phase method | A repeatable order in which to apply the tools | + +The single biggest debugging skill is matching the tool to the problem. +Most failures don't need every tool - they need the right one. + +--- + +## What's next? + +Return to the [menu of Side Quests](../index.md) or click the button in the bottom right of the page to move on to the next topic in the list. diff --git a/docs/en/docs/side_quests/debugging/index.md b/docs/en/docs/side_quests/debugging/index.md index 6c23af0c25..71bbd8bf06 100644 --- a/docs/en/docs/side_quests/debugging/index.md +++ b/docs/en/docs/side_quests/debugging/index.md @@ -1,2659 +1,64 @@ -# Troubleshooting Workflows - -Debugging is a critical skill that can save you hours of frustration and help you become a more effective Nextflow developer. Throughout your career, especially when you're starting out, you'll encounter bugs while building and maintaining your workflows. Learning systematic debugging approaches will help you identify and resolve issues quickly. - -### Learning goals - -In this side quest, we'll explore **systematic debugging techniques** for Nextflow workflows: - -- **Syntax error debugging**: Using IDE features and Nextflow error messages effectively -- **Channel debugging**: Diagnosing data flow issues and channel structure problems -- **Process debugging**: Investigating execution failures and resource issues -- **Built-in debugging tools**: Leveraging Nextflow's preview mode, stub running, and work directories -- **Systematic approaches**: A four-phase methodology for efficient debugging - -By the end, you'll have a robust debugging methodology that transforms frustrating error messages into clear roadmaps for solutions. - -### Prerequisites - -Before taking on this side quest, you should: - -- Have completed the [Hello Nextflow](../../hello_nextflow/index.md) tutorial or equivalent beginner's course. -- Be comfortable using basic Nextflow concepts and mechanisms (processes, channels, operators) - -**Optional:** We recommend completing the [IDE Features for Nextflow Development](../dev_environment/index.md) side quest first. -That covers comprehensive coverage of IDE features that support debugging (syntax highlighting, error detection, etc.), which we'll use heavily here. - ---- - -## 0. Get started - -#### Open the training codespace - -If you haven't yet done so, make sure to open the training environment as described in the [Environment Setup](../../envsetup/index.md). - -[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/nextflow-io/training?quickstart=1&ref=master) - -#### Move into the project directory - -Let's move into the directory where the files for this tutorial are located. - -```bash -cd side-quests/debugging -``` - -You can set VSCode to focus on this directory: - -```bash -code . -``` - -#### Review the materials - -You'll find a set of example workflows with various types of bugs that we'll use for practice: - -??? abstract "Directory contents" - - ```console - . - ├── bad_bash_var.nf - ├── bad_channel_shape.nf - ├── bad_channel_shape_viewed_debug.nf - ├── bad_channel_shape_viewed.nf - ├── bad_number_inputs.nf - ├── badpractice_syntax.nf - ├── bad_resources.nf - ├── bad_syntax.nf - ├── buggy_workflow.nf - ├── data - │ ├── sample_001.fastq.gz - │ ├── sample_002.fastq.gz - │ ├── sample_003.fastq.gz - │ ├── sample_004.fastq.gz - │ ├── sample_005.fastq.gz - │ └── sample_data.csv - ├── exhausted.nf - ├── invalid_process.nf - ├── missing_output.nf - ├── missing_software.nf - ├── missing_software_with_stub.nf - ├── nextflow.config - └── no_such_var.nf - ``` - -These files represent common debugging scenarios you'll encounter in real-world development. - -#### Review the assignment - -Your challenge is to run each workflow, identify the error(s), and fix them. - -For each buggy workflow: - -1. **Run the workflow** and observe the error -2. **Analyze the error message**: what is Nextflow telling you? -3. **Locate the problem** in the code using the clues provided -4. **Fix the bug** and verify your solution works -5. **Reset the file** before moving to the next section (use `git checkout `) - -The exercises progress from simple syntax errors to more subtle runtime issues. -Solutions are discussed inline, but try to solve each one yourself before reading ahead. - -#### Readiness checklist - -Think you're ready to dive in? - -- [ ] I understand the goal of this course and its prerequisites -- [ ] My codespace is up and running -- [ ] I've set my working directory appropriately -- [ ] I understand the assignment - -If you can check all the boxes, you're good to go. - ---- - -## 1. Syntax Errors - -Syntax errors are the most common type of error you'll encounter when writing Nextflow code. They occur when the code does not conform to the expected syntax rules of the Nextflow DSL. These errors prevent your workflow from running at all, so it's important to learn how to identify and fix them quickly. - -### 1.1. Missing braces - -One of the most common syntax errors, and sometimes one of the more complex ones to debug is **missing or mismatched brackets**. - -Let's start with a practical example. - -#### Run the pipeline - -```bash -nextflow run bad_syntax.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_syntax.nf` [stupefied_bhabha] DSL2 - revision: ca6327fad2 - - Error bad_syntax.nf:24:1: Unexpected input: '' - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -**Key elements of syntax error messages:** - -- **File and location**: Shows which file and line/column contain the error (`bad_syntax.nf:24:1`) -- **Error description**: Explains what the parser found that it didn't expect (`Unexpected input: ''`) -- **EOF indicator**: The `` (End Of File) message indicates the parser reached the end of the file while still expecting more content - a classic sign of unclosed braces - -#### Check the code - -Now, let's examine `bad_syntax.nf` to understand what's causing the error: - -```groovy title="bad_syntax.nf" hl_lines="14" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ -// Missing closing brace for the process - -workflow { - - // Create input channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - // Call the process with the input channel - PROCESS_FILES(input_ch) -} -``` - -For the purpose of this example we've left a comment for you to show where the error is. The Nextflow VSCode extension should also be giving you some hints about what might be wrong, putting the mismatched brace in red and highlighting the premature end of the file: - -![Bad syntax](../img/bad_syntax.png) - -**Debugging strategy for bracket errors:** - -1. Use VS Code's bracket matching (place cursor next to a bracket) -2. Check the Problems panel for bracket-related messages -3. Ensure each opening `{` has a corresponding closing `}` - -#### Fix the code - -Replace the comment with the missing closing brace: - -=== "After" - - ```groovy title="bad_syntax.nf" hl_lines="14" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } // Add the missing closing brace - - workflow { - - // Create input channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - // Call the process with the input channel - PROCESS_FILES(input_ch) - } - ``` - -=== "Before" - - ```groovy title="bad_syntax.nf" hl_lines="14" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - // Missing closing brace for the process - - workflow { - - // Create input channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - // Call the process with the input channel - PROCESS_FILES(input_ch) - } - ``` - -#### Run the pipeline - -Now run the workflow again to confirm it works: - -```bash -nextflow run bad_syntax.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_syntax.nf` [insane_faggin] DSL2 - revision: 961938ee2b - - executor > local (3) - [48/cd7f54] PROCESS_FILES (1) | 3 of 3 ✔ - ``` - -### 1.2. Using incorrect process keywords or directives - -Another common syntax error is an **invalid process definition**. This can happen if you forget to define required blocks or use incorrect directives in the process definition. - -#### Run the pipeline - -```bash -nextflow run invalid_process.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `invalid_process.nf` [nasty_jepsen] DSL2 - revision: da9758d614 - - Error invalid_process.nf:3:1: Invalid process definition -- check for missing or out-of-order section labels - │ 3 | process PROCESS_FILES { - │ | ^^^^^^^^^^^^^^^^^^^^^^^ - │ 4 | inputs: - │ 5 | val sample_name - │ 6 | - ╰ 7 | output: - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -#### Check the code - -The error indicates an "Invalid process definition" and shows the context around the problem. Looking at lines 3-7, we can see `inputs:` on line 4, which is the issue. Let's examine `invalid_process.nf`: - -```groovy title="invalid_process.nf" hl_lines="4" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - inputs: // ERROR: Should be 'input' not 'inputs' - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ -} - -workflow { - - // Create input channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - // Call the process with the input channel - PROCESS_FILES(input_ch) -} -``` - -Looking at line 4 in the error context, we can spot the issue: we're using `inputs` instead of the correct `input` directive. The Nextflow VSCode extension will also flag this: - -![Invalid process message](../img/invalid_process_message.png) - -#### Fix the code - -Replace the incorrect keyword with the correct one by referencing [the documentation](https://www.nextflow.io/docs/latest/process.html#): - -=== "After" - - ```groovy title="invalid_process.nf" hl_lines="4" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: // Fixed: Changed 'inputs' to 'input' - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - - workflow { - - // Create input channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - // Call the process with the input channel - PROCESS_FILES(input_ch) - } - ``` - -=== "Before" - - ```groovy title="invalid_process.nf" hl_lines="4" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - inputs: // ERROR: Should be 'input' not 'inputs' - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - - workflow { - - // Create input channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - // Call the process with the input channel - PROCESS_FILES(input_ch) - } - ``` - -#### Run the pipeline - -Now run the workflow again to confirm it works: - -```bash -nextflow run invalid_process.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `invalid_process.nf` [silly_fermi] DSL2 - revision: 961938ee2b - - executor > local (3) - [b7/76cd9d] PROCESS_FILES (2) | 3 of 3 ✔ - ``` - -### 1.3. Using bad variable names - -The variable names you use in your script blocks must be valid, derived either from inputs or from groovy code inserted before the script. But when you're wrangling complexity at the start of pipeline development, it's easy to make mistakes in variable naming, and Nextflow will let you know quickly. - -#### Run the pipeline - -```bash -nextflow run no_such_var.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `no_such_var.nf` [gloomy_meninsky] DSL2 - revision: 0c4d3bc28c - - Error no_such_var.nf:17:39: `undefined_var` is not defined - │ 17 | echo "Using undefined variable: ${undefined_var}" >> ${output_pref - ╰ | ^^^^^^^^^^^^^ - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -The error is caught at compile time and points directly to the undefined variable on line 17, with a caret indicating exactly where the problem is. - -#### Check the code - -Let's examine `no_such_var.nf`: - -```groovy title="no_such_var.nf" hl_lines="17" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_processed.txt" - - script: - // Define variables in Groovy code before the script - def output_prefix = "${sample_name}_processed" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - echo "Using undefined variable: ${undefined_var}" >> ${output_prefix}.txt // ERROR: undefined_var not defined - """ -} - -workflow { - input_ch = channel.of('sample1', 'sample2', 'sample3') - PROCESS_FILES(input_ch) -} -``` - -The error message indicates that the variable is not recognized in the script template, and there you go- you should be able to see `#!groovy ${undefined_var}` used in the script block, but not defined elsewhere. - -#### Fix the code - -If you get a 'No such variable' error, you can fix it by either defining the variable (by correcting input variable names or editing groovy code before the script), or by removing it from the script block if it's not needed: - -=== "After" - - ```groovy title="no_such_var.nf" hl_lines="15-17" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_processed.txt" - - script: - // Define variables in Groovy code before the script - def output_prefix = "${sample_name}_processed" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - """ // Removed the line with undefined_var - } - - workflow { - input_ch = channel.of('sample1', 'sample2', 'sample3') - PROCESS_FILES(input_ch) - } - ``` - -=== "Before" - - ```groovy title="no_such_var.nf" hl_lines="17" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - // Define variables in Groovy code before the script - def output_prefix = "${sample_name}_processed" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - echo "Using undefined variable: ${undefined_var}" >> ${output_prefix}.txt // ERROR: undefined_var not defined - """ - } - - workflow { - input_ch = channel.of('sample1', 'sample2', 'sample3') - PROCESS_FILES(input_ch) - } - ``` - -#### Run the pipeline - -Now run the workflow again to confirm it works: - -```bash -nextflow run no_such_var.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `no_such_var.nf` [suspicious_venter] DSL2 - revision: 6ba490f7c5 - - executor > local (3) - [21/237300] PROCESS_FILES (2) | 3 of 3 ✔ - ``` - -### 1.4. Bad use of Bash variables - -Starting out in Nextflow, it can be difficult to understand the difference between Nextflow (Groovy) and Bash variables. This can generate another form of the bad variable error that appears when trying to use variables in the Bash content of the script block. - -#### Run the pipeline - -```bash -nextflow run bad_bash_var.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_bash_var.nf` [infallible_mandelbrot] DSL2 - revision: 0853c11080 - - Error bad_bash_var.nf:13:42: `prefix` is not defined - │ 13 | echo "Processing ${sample_name}" > ${prefix}.txt - ╰ | ^^^^^^ - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -#### Check the code - -The error points to line 13 where `#!groovy ${prefix}` is used. Let's examine `bad_bash_var.nf` to see what's causing the issue: - -```groovy title="bad_bash_var.nf" hl_lines="13" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - prefix="${sample_name}_output" - echo "Processing ${sample_name}" > ${prefix}.txt # ERROR: ${prefix} is Groovy syntax, not Bash - """ -} -``` - -In this example, we're defining the `prefix` variable in Bash, but in a Nextflow process the `$` syntax we used to refer to it (`#!groovy ${prefix}`) is interpreted as a Groovy variable, not Bash. The variable doesn't exist in the Groovy context, so we get a 'no such variable' error. - -#### Fix the code - -If you want to use a Bash variable, you must escape the dollar sign like this: - -=== "After" - - ```groovy title="bad_bash_var.nf" hl_lines="13" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - prefix="${sample_name}_output" - echo "Processing ${sample_name}" > \${prefix}.txt # Fixed: Escaped the dollar sign - """ - } - - workflow { - input_ch = channel.of('sample1', 'sample2', 'sample3') - PROCESS_FILES(input_ch) - } - ``` - -=== "Before" - - ```groovy title="bad_bash_var.nf" hl_lines="13" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - prefix="${sample_name}_output" - echo "Processing ${sample_name}" > ${prefix}.txt # ERROR: ${prefix} is Groovy syntax, not Bash - """ - } - ``` - -This tells Nextflow to interpret this as a Bash variable. - -#### Run the pipeline - -Now run the workflow again to confirm it works: - -```bash -nextflow run bad_bash_var.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_bash_var.nf` [naughty_franklin] DSL2 - revision: 58c1c83709 - - executor > local (3) - [4e/560285] PROCESS_FILES (2) | 3 of 3 ✔ - ``` - -!!! tip "Groovy vs Bash Variables" - - For simple variable manipulations like string concatenation or prefix/suffix operations, it's usually more readable to use Groovy variables in the script section rather than Bash variables in the script block: - - ```groovy linenums="1" - script: - def output_prefix = "${sample_name}_processed" - def output_file = "${output_prefix}.txt" - """ - echo "Processing ${sample_name}" > ${output_file} - """ - ``` - - This approach avoids the need to escape dollar signs and makes the code easier to read and maintain. - -### 1.5. Statements Outside Workflow Block - -The Nextflow VSCode extension highlights issues with code structure that will cause errors. A common example is defining channels outside of the `workflow {}` block - this is now enforced as a syntax error. - -#### Run the pipeline - -```bash -nextflow run badpractice_syntax.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `badpractice_syntax.nf` [intergalactic_colden] DSL2 - revision: 5e4b291bde - - Error badpractice_syntax.nf:3:1: Statements cannot be mixed with script declarations -- move statements into a process or workflow - │ 3 | input_ch = channel.of('sample1', 'sample2', 'sample3') - ╰ | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -The error message clearly indicates the problem: statements (like channel definitions) cannot be mixed with script declarations outside of a workflow or process block. - -#### Check the code - -Let's examine `badpractice_syntax.nf` to see what's causing the error: - -```groovy title="badpractice_syntax.nf" hl_lines="3" linenums="1" -#!/usr/bin/env nextflow - -input_ch = channel.of('sample1', 'sample2', 'sample3') // ERROR: Channel defined outside workflow - -process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_processed.txt" - - script: - // Define variables in Groovy code before the script - def output_prefix = "${sample_name}_processed" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - """ -} - -workflow { - PROCESS_FILES(input_ch) -} -``` - -The VSCode extension will also highlight the `input_ch` variable as being defined outside the workflow block: - -![Non-lethal syntax error](../img/nonlethal.png) - -#### Fix the code - -Move the channel definition inside the workflow block: - -=== "After" - - ```groovy title="badpractice_syntax.nf" hl_lines="21" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_processed.txt" - - script: - // Define variables in Groovy code before the script - def output_prefix = "${sample_name}_processed" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - """ - } - - workflow { - input_ch = channel.of('sample1', 'sample2', 'sample3') // Moved inside workflow block - PROCESS_FILES(input_ch) - } - ``` - -=== "Before" - - ```groovy title="badpractice_syntax.nf" hl_lines="3" linenums="1" - #!/usr/bin/env nextflow - - input_ch = channel.of('sample1', 'sample2', 'sample3') // ERROR: Channel defined outside workflow - - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_processed.txt" - - script: - // Define variables in Groovy code before the script - def output_prefix = "${sample_name}_processed" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - """ - } - - workflow { - PROCESS_FILES(input_ch) - } - ``` - -#### Run the pipeline - -Run the workflow again to confirm the fix works: - -```bash -nextflow run badpractice_syntax.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `badpractice_syntax.nf` [naughty_ochoa] DSL2 - revision: 5e4b291bde - - executor > local (3) - [6a/84a608] PROCESS_FILES (2) | 3 of 3 ✔ - ``` - -Keep your input channels defined within the workflow block, and in general follow any other recommendations the extension makes. - -### Takeaway - -You can systematically identify and fix syntax errors using Nextflow error messages and IDE visual indicators. Common syntax errors include missing braces, incorrect process keywords, undefined variables, and improper use of Bash vs. Nextflow variables. The VSCode extension helps catch many of these before runtime. With these syntax debugging skills in your toolkit, you'll be able to quickly resolve the most common Nextflow syntax errors and move on to tackling more complex runtime issues. - -### What's next? - -Learn to debug more complex channel structure errors that occur even when syntax is correct. - ---- - -## 2. Channel Structure Errors - -Channel structure errors are more subtle than syntax errors because the code is syntactically correct, but the data shapes don't match what processes expect. Nextflow will try to run the pipeline, but might find that the number of inputs doesn't match what it expects and fail. These errors typically only appear at runtime and require an understanding of the data flowing through your workflow. - -!!! tip "Debugging Channels with `.view()`" - - Throughout this section, remember that you can use the `.view()` operator to inspect channel content at any point in your workflow. This is one of the most powerful debugging tools for understanding channel structure issues. We'll explore this technique in detail in section 2.4, but feel free to use it as you work through the examples. - - ```groovy - my_channel.view() // Shows what's flowing through the channel - ``` - -### 2.1. Wrong Number of Input Channels - -This error occurs when you pass a different number of channels than a process expects. - -#### Run the pipeline - -```bash -nextflow run bad_number_inputs.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_number_inputs.nf` [happy_swartz] DSL2 - revision: d83e58dcd3 - - Error bad_number_inputs.nf:23:5: Incorrect number of call arguments, expected 1 but received 2 - │ 23 | PROCESS_FILES(samples_ch, files_ch) - ╰ | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -#### Check the code - -The error message clearly states that the call expected 1 argument but received 2, and points to line 23. Let's examine `bad_number_inputs.nf`: - -```groovy title="bad_number_inputs.nf" hl_lines="5 23" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - val sample_name // Process expects only 1 input - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ -} - -workflow { - - // Create two separate channels - samples_ch = channel.of('sample1', 'sample2', 'sample3') - files_ch = channel.of('file1.txt', 'file2.txt', 'file3.txt') - - // ERROR: Passing 2 channels but process expects only 1 - PROCESS_FILES(samples_ch, files_ch) -} -``` - -You should see the mismatched `PROCESS_FILES` call, supplying multiple input channels when the process only defines one. The VSCode extension will also under line process call in red, and supply a diagnostic message when you mouse over: - -![Incorrect number of args message](../img/incorrect_num_args.png) - -#### Fix the code - -For this specific example, the process expects a single channel and doesn't require the second channel, so we can fix it by passing only the `samples_ch` channel: - -=== "After" - - ```groovy title="bad_number_inputs.nf" hl_lines="23" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name // Process expects only 1 input - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - - workflow { - - // Create two separate channels - samples_ch = channel.of('sample1', 'sample2', 'sample3') - files_ch = channel.of('file1.txt', 'file2.txt', 'file3.txt') - - // Fixed: Pass only the channel the process expects - PROCESS_FILES(samples_ch) - } - ``` - -=== "Before" - - ```groovy title="bad_number_inputs.nf" hl_lines="5 23" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name // Process expects only 1 input - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - - workflow { - - // Create two separate channels - samples_ch = channel.of('sample1', 'sample2', 'sample3') - files_ch = channel.of('file1.txt', 'file2.txt', 'file3.txt') - - // ERROR: Passing 2 channels but process expects only 1 - PROCESS_FILES(samples_ch, files_ch) - } - ``` - -#### Run the pipeline - -```bash -nextflow run bad_number_inputs.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_number_inputs.nf` [big_euler] DSL2 - revision: e302bd87be - - executor > local (3) - [48/497f7b] PROCESS_FILES (3) | 3 of 3 ✔ - ``` - -More commonly than this example, you might add additional inputs to a process and forget to update the workflow call accordingly, which can lead to this type of error. Fortunately, this is one of the easier-to-understand and fix errors, as the error message is quite clear about the mismatch. - -### 2.2. Channel Exhaustion (Process Runs Fewer Times Than Expected) - -Some channel structure errors are much more subtle and produce no errors at all. Probably the most common of these reflects a challenge that new Nextflow users face in understanding that queue channels can be exhausted and run out of items, meaning the workflow finishes prematurely. - -#### Run the pipeline - -```bash -nextflow run exhausted.nf -``` - -??? success "Command output" - -```console title="Exhausted channel output" - N E X T F L O W ~ version 25.10.4 - -Launching `exhausted.nf` [extravagant_gauss] DSL2 - revision: 08cff7ba2a - -executor > local (1) -[bd/f61fff] PROCESS_FILES (1) [100%] 1 of 1 ✔ -``` - -This workflow completes without error, but it only processes a single sample! - -#### Check the code - -Let's examine `exhausted.nf` to see if that's right: - -```groovy title="exhausted.nf" hl_lines="23 24" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - val reference - val sample_name - - output: - path "${output_prefix}.txt" - - script: - // Define variables in Groovy code before the script - output_prefix = "${reference}_${sample_name}" - def timestamp = new Date().format("yyyy-MM-dd") - - """ - echo "Processing ${sample_name} on ${timestamp}" > ${output_prefix}.txt - """ -} - -workflow { - - reference_ch = channel.of('baseline_reference') - input_ch = channel.of('sample1', 'sample2', 'sample3') - - PROCESS_FILES(reference_ch, input_ch) -} -``` - -The process only runs once instead of three times because the `reference_ch` channel is a queue channel that gets exhausted after the first process execution. When one channel is exhausted, the entire process stops, even if other channels still have items. - -This is a common pattern where you have a single reference file that needs to be reused across multiple samples. The solution is to convert the reference channel to a value channel that can be reused indefinitely. - -#### Fix the code - -There are a couple of ways to address this depending on how many files are affected. - -**Option 1**: You have a single reference file that you are re-using a lot. You can simply create a value channel type, which can be used over and over again. There are three ways to do this: - -**1a** Use `channel.value()`: - -```groovy title="exhausted.nf (fixed - Option 1a)" hl_lines="2" linenums="21" -workflow { - reference_ch = channel.value('baseline_reference') // Value channel can be reused - input_ch = channel.of('sample1', 'sample2', 'sample3') - - PROCESS_FILES(reference_ch, input_ch) -} -``` - -**1b** Use the `first()` [operator](https://www.nextflow.io/docs/latest/reference/operator.html#first): - -```groovy title="exhausted.nf (fixed - Option 1b)" hl_lines="2" linenums="21" -workflow { - reference_ch = channel.of('baseline_reference').first() // Convert to value channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - PROCESS_FILES(reference_ch, input_ch) -} -``` - -**1c.** Use the `collect()` [operator](https://www.nextflow.io/docs/latest/reference/operator.html#collect): - -```groovy title="exhausted.nf (fixed - Option 1c)" hl_lines="2" linenums="21" -workflow { - reference_ch = channel.of('baseline_reference').collect() // Convert to value channel - input_ch = channel.of('sample1', 'sample2', 'sample3') - - PROCESS_FILES(reference_ch, input_ch) -} -``` - -**Option 2**: In more complex scenarios, perhaps where you have multiple reference files for all samples in the sample channel, you can use the `combine` operator to create a new channel that combines the two channels into tuples: - -```groovy title="exhausted.nf (fixed - Option 2)" hl_lines="4" linenums="21" -workflow { - reference_ch = channel.of('baseline_reference','other_reference') - input_ch = channel.of('sample1', 'sample2', 'sample3') - combined_ch = reference_ch.combine(input_ch) // Creates cartesian product - - PROCESS_FILES(combined_ch) -} -``` - -The `.combine()` operator generates a cartesian product of the two channels, so each item in `reference_ch` will be paired with each item in `input_ch`. This allows the process to run for each sample while still using the reference. - -This requires the process input to be adjusted. In our example, the start of the process definition would need to be adjusted as follows: - -```groovy title="exhausted.nf (fixed - Option 2)" hl_lines="5" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - tuple val(reference), val(sample_name) -``` - -This approach may not be suitable in all situations. - -#### Run the pipeline - -Try one of the fixes above and run the workflow again: - -```bash -nextflow run exhausted.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `exhausted.nf` [maniac_leavitt] DSL2 - revision: f372a56a7d - - executor > local (3) - [80/0779e9] PROCESS_FILES (3) | 3 of 3 ✔ - ``` - -You should now see all three samples being processed instead of just one. - -### 2.3. Wrong Channel Content Structure - -When workflows reach a certain level of complexity, it can be a little difficult to keep track of the internal structures of each channel, and people commonly generate mismatches between what the process expects and what the channel actually contains. This is more subtle than the issue we discussed earlier, where the number of channels was incorrect. In this case, you can have the correct number of input channels, but the internal structure of one or more of those channels doesn't match what the process expects. - -#### Run the pipeline - -```bash -nextflow run bad_channel_shape.nf -``` - -??? failure "Command output" - - ```console - Launching `bad_channel_shape.nf` [hopeful_pare] DSL2 - revision: ffd66071a1 - - executor > local (3) - executor > local (3) - [3f/c2dcb3] PROCESS_FILES (3) [ 0%] 0 of 3 ✘ - ERROR ~ Error executing process > 'PROCESS_FILES (1)' - - Caused by: - Missing output file(s) `[sample1, file1.txt]_output.txt` expected by process `PROCESS_FILES (1)` - - - Command executed: - - echo "Processing [sample1, file1.txt]" > [sample1, file1.txt]_output.txt - - Command exit status: - 0 - - Command output: - (empty) - - Work dir: - /workspaces/training/side-quests/debugging/work/d6/1fb69d1d93300bbc9d42f1875b981e - - Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line - - -- Check '.nextflow.log' file for details - ``` - -#### Check the code - -The square brackets in the error message provide the clue here - the process is treating the tuple as a single value, which is not what we want. Let's examine `bad_channel_shape.nf`: - -```groovy title="bad_channel_shape.nf" hl_lines="5 20-22" linenums="1" -#!/usr/bin/env nextflow - -process PROCESS_FILES { - input: - val sample_name // Expects single value, gets tuple - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ -} - -workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'] - ) - PROCESS_FILES(input_ch) -} -``` - -You can see that we're generating a channel composed of tuples: `['sample1', 'file1.txt']`, but the process expects a single value, `val sample_name`. The command executed shows that the process is trying to create a file named `[sample3, file3.txt]_output.txt`, which is not the intended output. - -#### Fix the code - -To fix this, if the process requires both inputs we could adjust the process to accept a tuple: - -=== "Option 1: Accept tuple in process" - - === "After" - - ```groovy title="bad_channel_shape.nf" hl_lines="5" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - tuple val(sample_name), val(file_name) // Fixed: Accept tuple - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - - workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'] - ) - PROCESS_FILES(input_ch) - } - ``` - - === "Before" - - ```groovy title="bad_channel_shape.nf" hl_lines="5" linenums="1" - #!/usr/bin/env nextflow - - process PROCESS_FILES { - input: - val sample_name // Expects single value, gets tuple - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - - workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'] - ) - PROCESS_FILES(input_ch) - } - ``` - -=== "Option 2: Extract first element" - - === "After" - - ```groovy title="bad_channel_shape.nf" hl_lines="9" linenums="16" - workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'] - ) - PROCESS_FILES(input_ch.map { it[0] }) // Fixed: Extract first element - } - ``` - - === "Before" - - ```groovy title="bad_channel_shape.nf" hl_lines="9" linenums="16" - workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'] - ) - PROCESS_FILES(input_ch) - } - ``` - -#### Run the pipeline - -Pick one of the solutions and re-run the workflow: - -```bash -nextflow run bad_channel_shape.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_channel_shape.nf` [clever_thompson] DSL2 - revision: 8cbcae3746 - - executor > local (3) - [bb/80a958] PROCESS_FILES (2) | 3 of 3 ✔ - ``` - -### 2.4. Channel Debugging Techniques - -#### Using `.view()` for Channel Inspection - -The most powerful debugging tool for channels is the `.view()` operator. With `.view()`, you can understand the shape of your channels at all stages to help with debugging. - -#### Run the pipeline - -Run `bad_channel_shape_viewed.nf` to see this in action: - -```bash -nextflow run bad_channel_shape_viewed.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_channel_shape_viewed.nf` [maniac_poisson] DSL2 - revision: b4f24dc9da - - executor > local (3) - [c0/db76b3] PROCESS_FILES (3) [100%] 3 of 3 ✔ - Channel content: [sample1, file1.txt] - Channel content: [sample2, file2.txt] - Channel content: [sample3, file3.txt] - After mapping: sample1 - After mapping: sample2 - After mapping: sample3 - ``` - -#### Check the code - -Let's examine `bad_channel_shape_viewed.nf` to see how `.view()` is used: - -```groovy title="bad_channel_shape_viewed.nf" linenums="16" hl_lines="9 11" -workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'] - ) - .view { "Channel content: $it" } // Debug: Show original channel content - .map { tuple -> tuple[0] } // Transform: Extract first element - .view { "After mapping: $it" } // Debug: Show transformed channel content - - PROCESS_FILES(input_ch) -} -``` - -#### Fix the code - -To save you from using `.view()` operations excessively in future to understand channel content, it's advisable to add some comments to help: - -```groovy title="bad_channel_shape_viewed.nf (with comments)" linenums="16" hl_lines="8 9" -workflow { - - // Channel emits tuples, but process expects single values - input_ch = channel.of( - ['sample1', 'file1.txt'], - ['sample2', 'file2.txt'], - ['sample3', 'file3.txt'], - ) // [sample_name, file_name] - .map { tuple -> tuple[0] } // sample_name - - PROCESS_FILES(input_ch) -} -``` - -This will become more important as your workflows grow in complexity and channel structure becomes more opaque. - -#### Run the pipeline - -```bash -nextflow run bad_channel_shape_viewed.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_channel_shape_viewed.nf` [marvelous_koch] DSL2 - revision: 03e79cdbad - - executor > local (3) - [ff/d67cec] PROCESS_FILES (2) | 3 of 3 ✔ - Channel content: [sample1, file1.txt] - Channel content: [sample2, file2.txt] - Channel content: [sample3, file3.txt] - After mapping: sample1 - After mapping: sample2 - After mapping: sample3 - ``` - -### Takeaway - -Many channel structure errors can be created with valid Nextflow syntax. You can debug channel structure errors by understanding data flow, using `.view()` operators for inspection, and recognizing error message patterns like square brackets indicating unexpected tuple structures. - -### What's next? - -Learn about errors created by process definitions. - ---- - -## 3. Process Structure Errors - -Most of the errors you encounter related to processes will related to mistakes you have made in forming the command, or to issues related to the underlying software. That said, similarly to the channel issues above, you can make mistakes in the process definition that don't quality as syntax errors, but which will cause errors at run time. - -### 3.1. Missing Output Files - -One common error when writing processes is to do something that generates a mismatch between what the process expects and what is generated. - -#### Run the pipeline - -```bash -nextflow run missing_output.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `missing_output.nf` [zen_stone] DSL2 - revision: 37ff61f926 - - executor > local (3) - executor > local (3) - [fd/2642e9] process > PROCESS_FILES (2) [ 66%] 2 of 3, failed: 2 - ERROR ~ Error executing process > 'PROCESS_FILES (3)' - - Caused by: - Missing output file(s) `sample3.txt` expected by process `PROCESS_FILES (3)` - - - Command executed: - - echo "Processing sample3" > sample3_output.txt - - Command exit status: - 0 - - Command output: - (empty) - - Work dir: - /workspaces/training/side-quests/debugging/work/02/9604d49fb8200a74d737c72a6c98ed - - Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line - - -- Check '.nextflow.log' file for details - ``` - -#### Check the code - -The error message indicates that the process expected to produce an output file named `sample3.txt`, but the script actually creates `sample3_output.txt`. Let's examine the process definition in `missing_output.nf`: - -```groovy title="missing_output.nf" linenums="3" hl_lines="6 10" -process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}.txt" // Expects: sample3.txt - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt // Creates: sample3_output.txt - """ -} -``` - -You should see that there is a mismatch between the output file name in the `output:` block, and the one used in the script. This mismatch causes the process to fail. If you encounter this sort of error, go back and check that the outputs match between your process definition and your output block. - -If the problem still isn't clear, check the work directory itself to identify the actual output files created: - -```bash -❯ ls -h work/02/9604d49fb8200a74d737c72a6c98ed -sample3_output.txt -``` - -For this example this would highlight to us that a `_output` suffix is being incorporated into the output file name, contrary to our `output:` definition. - -#### Fix the code - -Fix the mismatch by making the output filename consistent: - -=== "After" - - ```groovy title="missing_output.nf" hl_lines="6 10" linenums="3" - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}_output.txt" // Fixed: Match the script output - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ - } - ``` - -=== "Before" - - ```groovy title="missing_output.nf" hl_lines="6 10" linenums="3" - process PROCESS_FILES { - input: - val sample_name - - output: - path "${sample_name}.txt" // Expects: sample3.txt - - script: - """ - echo "Processing ${sample_name}" > ${sample_name}_output.txt // Creates: sample3_output.txt - """ - } - ``` - -#### Run the pipeline - -```bash -nextflow run missing_output.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `missing_output.nf` [elated_hamilton] DSL2 - revision: 961938ee2b - - executor > local (3) - [16/1c437c] PROCESS_FILES (3) | 3 of 3 ✔ - ``` - -### 3.2. Missing software - -Another class of errors occurs due to mistakes in software provisioning. `missing_software.nf` is a syntactically valid workflow, but it depends on some external software to provide the `cowpy` command it uses. - -#### Run the pipeline - -```bash -nextflow run missing_software.nf -``` - -??? failure "Command output" - - ```console hl_lines="12 18" - ERROR ~ Error executing process > 'PROCESS_FILES (3)' - - Caused by: - Process `PROCESS_FILES (3)` terminated with an error exit status (127) - - - Command executed: - - cowpy sample3 > sample3_output.txt - - Command exit status: - 127 - - Command output: - (empty) - - Command error: - .command.sh: line 2: cowpy: command not found - - Work dir: - /workspaces/training/side-quests/debugging/work/82/42a5bfb60c9c6ee63ebdbc2d51aa6e - - Tip: you can try to figure out what's wrong by changing to the process work directory and showing the script file named `.command.sh` - - -- Check '.nextflow.log' file for details - ``` - -The process doesn't have access to the command we're specifying. Sometimes this is because a script is present in the workflow `bin` directory, but has not been made executable. Other times it is because the software is not installed in the container or environment where the workflow is running. - -#### Check the code - -Look out for that `127` exit code - it tells you exactly the problem. Let's examine `missing_software.nf`: - -```groovy title="missing_software.nf" linenums="3" hl_lines="3" -process PROCESS_FILES { - - container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' - - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - cowpy ${sample_name} > ${sample_name}_output.txt - """ -} -``` - -#### Fix the code - -We've been a little disingenuous here, and there's actually nothing wrong with the code. We just need to specify the necessary configuration to run the process in such a way that it has access to the command in question. In this case the process has a container definition, so all we need to do is run the workflow with Docker enabled. - -#### Run the pipeline - -We've set up a Docker profile for you in `nextflow.config`, so you can run the workflow with: - -```bash -nextflow run missing_software.nf -profile docker -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `missing_software.nf` [awesome_stonebraker] DSL2 - revision: 0296d12839 - - executor > local (3) - [38/ab20d1] PROCESS_FILES (1) | 3 of 3 ✔ - ``` - -!!! note - - To learn more about how Nextflow uses containers, see [Hello Nextflow](../../hello_nextflow/05_hello_containers.md) - -### 3.3. Bad resource configuration - -In production usage, you'll be configuring resources on your processes. For example `memory` defines the maximum amount of memory available to your process, and if the process exceeds that, your scheduler will typically kill the process and return an exit code of `137`. We can't demonstrate that here because we're using the `local` executor, but we can show something similar with `time`. - -#### Run the pipeline - -`bad_resources.nf` has process configuration with an unrealistic bound on time of 1 millisecond: - -```bash -nextflow run bad_resources.nf -profile docker -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_resources.nf` [disturbed_elion] DSL2 - revision: 27d2066e86 - - executor > local (3) - [c0/ded8e1] PROCESS_FILES (3) | 0 of 3 ✘ - ERROR ~ Error executing process > 'PROCESS_FILES (2)' - - Caused by: - Process exceeded running time limit (1ms) - - Command executed: - - cowpy sample2 > sample2_output.txt - - Command exit status: - - - - Command output: - (empty) - - Work dir: - /workspaces/training/side-quests/debugging/work/53/f0a4cc56d6b3dc2a6754ff326f1349 - - Container: - community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273 - - Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` - - -- Check '.nextflow.log' file for details - ``` - -#### Check the code - -Let's examine `bad_resources.nf`: - -```groovy title="bad_resources.nf" linenums="3" hl_lines="3" -process PROCESS_FILES { - - time '1 ms' // ERROR: Unrealistic time limit - - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - sleep 1 # Takes 1 second, but time limit is 1ms - cowpy ${sample_name} > ${sample_name}_output.txt - """ -} -``` - -We know the process will take longer than a second (we've added a sleep in there to make sure), but the process is set to time out after 1 millisecond. Someone has been a little unrealistic with their configuration! - -#### Fix the code - -Increase the time limit to a realistic value: - -=== "After" - - ```groovy title="bad_resources.nf" hl_lines="3" linenums="3" - process PROCESS_FILES { - - time '100 s' // Fixed: Realistic time limit - - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - sleep 1 - cowpy ${sample_name} > ${sample_name}_output.txt - """ - } - ``` - -=== "Before" - - ```groovy title="bad_resources.nf" hl_lines="3" linenums="3" - process PROCESS_FILES { - - time '1 ms' // ERROR: Unrealistic time limit - - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - sleep 1 # Takes 1 second, but time limit is 1ms - cowpy ${sample_name} > ${sample_name}_output.txt - """ - } - ``` - -#### Run the pipeline - -```bash -nextflow run bad_resources.nf -profile docker -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_resources.nf` [friendly_mcclintock] DSL2 - revision: 381567d2c1 - - executor > local (3) - [c2/9b4c41] PROCESS_FILES (3) | 3 of 3 ✔ - ``` - -If you make sure to read your error messages failures like this should not puzzle you for too long. But make sure you understand the resource requirements of the commands you are running so that you can configure your resource directives appropriately. - -### 3.4. Process Debugging Techniques - -When processes fail or behave unexpectedly, you need systematic techniques to investigate what went wrong. The work directory contains all the information you need to debug process execution. - -#### Using Work Directory Inspection - -The most powerful debugging tool for processes is examining the work directory. When a process fails, Nextflow creates a work directory for that specific process execution containing all the files needed to understand what happened. - -#### Run the pipeline - -Let's use the `missing_output.nf` example from earlier to demonstrate work directory inspection (re-generate an output naming mismatch if you need to): - -```bash -nextflow run missing_output.nf -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `missing_output.nf` [irreverent_payne] DSL2 - revision: 3d5117f7e2 - - executor > local (3) - [5d/d544a4] PROCESS_FILES (2) | 0 of 3 ✘ - ERROR ~ Error executing process > 'PROCESS_FILES (1)' - - Caused by: - Missing output file(s) `sample1.txt` expected by process `PROCESS_FILES (1)` - - Command executed: - - echo "Processing sample1" > sample1_output.txt - - Command exit status: - 0 - - Command output: - (empty) - - Work dir: - /workspaces/training/side-quests/debugging/work/1e/2011154d0b0f001cd383d7364b5244 - - Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` - - -- Check '.nextflow.log' file for details - ``` - -#### Check the work directory - -When you get this error, the work directory contains all the debugging information. Find the work directory path from the error message and examine its contents: - -```bash -# Find the work directory from the error message -ls work/02/9604d49fb8200a74d737c72a6c98ed/ -``` - -You can then examine the key files: - -##### Check the Command Script - -The `.command.sh` file shows exactly what command was executed: - -```bash -# View the executed command -cat work/02/9604d49fb8200a74d737c72a6c98ed/.command.sh -``` - -This reveals: - -- **Variable substitution**: Whether Nextflow variables were properly expanded -- **File paths**: Whether input files were correctly located -- **Command structure**: Whether the script syntax is correct - -Common issues to look for: - -- **Missing quotes**: Variables containing spaces need proper quoting -- **Wrong file paths**: Input files that don't exist or are in wrong locations -- **Incorrect variable names**: Typos in variable references -- **Missing environment setup**: Commands that depend on specific environments - -##### Check Error Output - -The `.command.err` file contains the actual error messages: - -```bash -# View error output -cat work/02/9604d49fb8200a74d737c72a6c98ed/.command.err -``` - -This file will show: - -- **Exit codes**: 127 (command not found), 137 (killed), etc. -- **Permission errors**: File access issues -- **Software errors**: Application-specific error messages -- **Resource errors**: Memory/time limit exceeded - -##### Check Standard Output - -The `.command.out` file shows what your command produced: - -```bash -# View standard output -cat work/02/9604d49fb8200a74d737c72a6c98ed/.command.out -``` - -This helps verify: - -- **Expected output**: Whether the command produced the right results -- **Partial execution**: Whether the command started but failed partway through -- **Debug information**: Any diagnostic output from your script - -##### Check the Exit Code - -The `.exitcode` file contains the exit code for the process: - -```bash -# View exit code -cat work/*/*/.exitcode -``` - -Common exit codes and their meanings: - -- **Exit code 127**: Command not found - check software installation -- **Exit code 137**: Process killed - check memory/time limits - -##### Check File Existence - -When processes fail due to missing output files, check what files were actually created: - -```bash -# List all files in the work directory -ls -la work/02/9604d49fb8200a74d737c72a6c98ed/ -``` - -This helps identify: - -- **File naming mismatches**: Output files with different names than expected -- **Permission issues**: Files that couldn't be created -- **Path problems**: Files created in wrong directories - -In our example earlier, this confirmed to us that while our expected `sample3.txt` wasn't present, `sample3_output.txt` was: - -```bash -❯ ls -h work/02/9604d49fb8200a74d737c72a6c98ed -sample3_output.txt -``` - -### Takeaway - -Process debugging requires examining work directories to understand what went wrong. Key files include `.command.sh` (the executed script), `.command.err` (error messages), and `.command.out` (standard output). Exit codes like 127 (command not found) and 137 (process killed) provide immediate diagnostic clues about the type of failure. - -### What's next? - -Learn about Nextflow's built-in debugging tools and systematic approaches to troubleshooting. - --- - -## 4. Built-in Debugging Tools and Advanced Techniques - -Nextflow provides several powerful built-in tools for debugging and analyzing workflow execution. These tools help you understand what went wrong, where it went wrong, and how to fix it efficiently. - -### 4.1. Real-time Process Output - -Sometimes you need to see what's happening inside running processes. You can enable real-time process output, which shows you exactly what each task is doing as it executes. - -#### Run the pipeline - -`bad_channel_shape_viewed.nf` from our earlier examples printed channel content using `.view()`, but we can also use the `debug` directive to echo variables from within the process itself, which we demonstrate in `bad_channel_shape_viewed_debug.nf`. Run the workflow: - -```bash -nextflow run bad_channel_shape_viewed_debug.nf -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_channel_shape_viewed_debug.nf` [agitated_crick] DSL2 - revision: ea3676d9ec - - executor > local (3) - [c6/2dac51] process > PROCESS_FILES (3) [100%] 3 of 3 ✔ - Channel content: [sample1, file1.txt] - Channel content: [sample2, file2.txt] - Channel content: [sample3, file3.txt] - After mapping: sample1 - After mapping: sample2 - After mapping: sample3 - Sample name inside process is sample2 - - Sample name inside process is sample1 - - Sample name inside process is sample3 - ``` - -#### Check the code - -Let's examine `bad_channel_shape_viewed_debug.nf` to see how the `debug` directive works: - -```groovy title="bad_channel_shape_viewed_debug.nf" linenums="3" hl_lines="2" -process PROCESS_FILES { - debug true // Enable real-time output - - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - echo "Sample name inside process is ${sample_name}" - echo "Processing ${sample_name}" > ${sample_name}_output.txt - """ -} -``` - -The `debug` directive can be a quick and convenient way to understand the environment of a process. - -### 4.2. Preview Mode - -Sometimes you want to catch problems before any processes run. Nextflow provides a flag for this kind of proactive debugging: `-preview`. - -#### Run the pipeline - -The preview mode lets you test workflow logic without executing commands. This can be quite useful for quickly checking the structure of your workflow and ensuring that processes are connected correctly without running any actual commands. - -!!! note - - If you fixed `bad_syntax.nf` earlier, reintroduce the syntax error by removing the closing brace after the script block before running this command. - -Run this command: - -```bash -nextflow run bad_syntax.nf -preview -``` - -??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `bad_syntax.nf` [magical_mercator] DSL2 - revision: 550b9a8873 - - Error bad_syntax.nf:24:1: Unexpected input: '' - - ERROR ~ Script compilation failed - - -- Check '.nextflow.log' file for details - ``` - -Preview mode is particularly useful for catching syntax errors early without running any processes. It validates the workflow structure and process connections before execution. - -### 4.3. Stub Running for Logic Testing - -Sometimes errors are difficult to debug because commands take too long, require special software, or fail for complex reasons. Stub running lets you test workflow logic without executing the actual commands. - -#### Run the pipeline - -When you're developing a Nextflow process, you can use the `stub` directive to define 'dummy' commands that generate outputs of the correct form without running the real command. This approach is particularly valuable when you want to verify that your workflow logic is correct before dealing with the complexities of the actual software. - -For example, remember our `missing_software.nf` from earlier? The one where we had missing software that prevented the workflow running until we added `-profile docker`? `missing_software_with_stub.nf` is a very similar workflow. If we run it in the same way, we will generate the same error: - -```bash -nextflow run missing_software_with_stub.nf -``` - -??? failure "Command output" - - ```console hl_lines="12 18" - ERROR ~ Error executing process > 'PROCESS_FILES (3)' - - Caused by: - Process `PROCESS_FILES (3)` terminated with an error exit status (127) - - - Command executed: - - cowpy sample3 > sample3_output.txt - - Command exit status: - 127 - - Command output: - (empty) - - Command error: - .command.sh: line 2: cowpy: command not found - - Work dir: - /workspaces/training/side-quests/debugging/work/82/42a5bfb60c9c6ee63ebdbc2d51aa6e - - Tip: you can try to figure out what's wrong by changing to the process work directory and showing the script file named `.command.sh` - - -- Check '.nextflow.log' file for details - ``` - -However, this workflow will not produce errors if we run it with `-stub-run`, even without the `docker` profile: - -```bash -nextflow run missing_software_with_stub.nf -stub-run -``` - -??? success "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `missing_software_with_stub.nf` [astonishing_shockley] DSL2 - revision: f1f4f05d7d - - executor > local (3) - [b5/2517a3] PROCESS_FILES (3) | 3 of 3 ✔ - ``` - -#### Check the code - -Let's examine `missing_software_with_stub.nf`: - -```groovy title="missing_software.nf (with stub)" hl_lines="16-19" linenums="3" -process PROCESS_FILES { - - container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' - - input: - val sample_name - - output: - path "${sample_name}_output.txt" - - script: - """ - cowpy ${sample_name} > ${sample_name}_output.txt - """ - - stub: - """ - touch ${sample_name}_output.txt - """ -} -``` - -Relative to `missing_software.nf`, this process has a `stub:` directive specifying a command to be used instead of the one specified in `script:`, in the event that that Nextflow is run in stub mode. - -The `touch` command we're using here doesn't depend on any software or appropriate inputs, and will run in all situations, allowing us to debug workflow logic without worrying about the process internals. - -**Stub running helps debug:** - -- Channel structure and data flow -- Process connections and dependencies -- Parameter propagation -- Workflow logic without software dependencies - -### 4.4. Systematic Debugging Approach - -Now that you've learned individual debugging techniques - from trace files and work directories to preview mode, stub running, and resource monitoring - let's tie them together into a systematic methodology. Having a structured approach prevents you from getting overwhelmed by complex errors and ensures you don't miss important clues. - -This methodology combines all the tools we've covered into an efficient workflow: - -**Four-Phase Debugging Method:** - -**Phase 1: Syntax Error Resolution (5 minutes)** - -1. Check for red underlines in VSCode or your IDE -2. Run `nextflow run workflow.nf -preview` to identify syntax issues -3. Fix all syntax errors (missing braces, trailing commas, etc.) -4. Ensure the workflow parses successfully before proceeding - -**Phase 2: Quick Assessment (5 minutes)** - -1. Read runtime error messages carefully -2. Check if it's a runtime, logic, or resource error -3. Use preview mode to test basic workflow logic - -**Phase 3: Detailed Investigation (15-30 minutes)** - -1. Find the work directory of the failed task -2. Examine log files -3. Add `.view()` operators to inspect channels -4. Use `-stub-run` to test workflow logic without execution - -**Phase 4: Fix and Validate (15 minutes)** - -1. Make minimal targeted fixes -2. Test with resume: `nextflow run workflow.nf -resume` -3. Verify complete workflow execution - -!!! tip "Using Resume for Efficient Debugging" - - Once you've identified a problem, you need an efficient way to test your fixes without wasting time re-running successful parts of your workflow. Nextflow's `-resume` functionality is invaluable for debugging. - - You will have encountered `-resume` if you've worked through [Hello Nextflow](../../hello_nextflow/index.md), and it's important that you make good use of it when debugging to save yourself waiting while the processes before your problem process run. - - **Resume debugging strategy:** - - 1. Run workflow until failure - 2. Examine work directory for failed task - 3. Fix the specific issue - 4. Resume to test only the fix - 5. Repeat until workflow completes - -#### Debugging Configuration Profile - -To make this systematic approach even more efficient, you can create a dedicated debugging configuration that automatically enables all the tools you need: - -```groovy title="nextflow.config (debug profile)" linenums="1" -profiles { - debug { - process { - debug = true - cleanup = false - - // Conservative resources for debugging - maxForks = 1 - memory = '2.GB' - cpus = 1 - } - } -} -``` - -Then you can run the pipeline with this profile enabled: - -```bash -nextflow run workflow.nf -profile debug -``` - -This profile enables real-time output, preserves work directories, and limits parallelization for easier debugging. - -### 4.5. Practical Debugging Exercise - -Now it's time to put the systematic debugging approach into practice. The workflow `buggy_workflow.nf` contains several common errors that represent the types of issues you'll encounter in real-world development. - -!!! exercise - - Use the systematic debugging approach to identify and fix all errors in `buggy_workflow.nf`. This workflow attempts to process sample data from a CSV file but contains multiple intentional bugs representing common debugging scenarios. - - Start by running the workflow to see the first error: - - ```bash - nextflow run buggy_workflow.nf - ``` - - ??? failure "Command output" - - ```console - N E X T F L O W ~ version 25.10.4 - - Launching `buggy_workflow.nf` [wise_ramanujan] DSL2 - revision: d51a8e83fd - - ERROR ~ Range [11, 12) out of bounds for length 11 - - -- Check '.nextflow.log' file for details - ``` - - This cryptic error indicates a parsing problem around line 11-12 in the `params{}` block. The v2 parser catches structural issues early. - - Apply the four-phase debugging method you've learned: - - **Phase 1: Syntax Error Resolution** - - Check for red underlines in VSCode or your IDE - - Run `nextflow run workflow.nf -preview` to identify syntax issues - - Fix all syntax errors (missing braces, trailing commas, etc.) - - Ensure the workflow parses successfully before proceeding - - **Phase 2: Quick Assessment** - - Read runtime error messages carefully - - Identify whether errors are runtime, logic, or resource-related - - Use `-preview` mode to test basic workflow logic - - **Phase 3: Detailed Investigation** - - Examine work directories for failed tasks - - Add `.view()` operators to inspect channels - - Check log files in work directories - - Use `-stub-run` to test workflow logic without execution - - **Phase 4: Fix and Validate** - - Make targeted fixes - - Use `-resume` to test fixes efficiently - - Verify complete workflow execution - - **Debugging Tools at Your Disposal:** - ```bash - # Preview mode for syntax checking - nextflow run buggy_workflow.nf -preview - - # Debug profile for detailed output - nextflow run buggy_workflow.nf -profile debug - - # Stub running for logic testing - nextflow run buggy_workflow.nf -stub-run - - # Resume after fixes - nextflow run buggy_workflow.nf -resume - ``` - - ??? solution - The `buggy_workflow.nf` contains 9 or 10 distinct errors (depending how you count) covering all major debugging categories. Here's a systematic breakdown of each error and how to fix it - - Let's start with those syntax errors: - - **Error 1: Syntax Error - Trailing Comma** - ```groovy linenums="21" - output: - path "${sample_id}_result.txt", // ERROR: Trailing comma - ``` - **Fix:** Remove the trailing comma - ```groovy linenums="21" - output: - path "${sample_id}_result.txt" - ``` - - **Error 2: Syntax Error - Missing Closing Brace** - ```groovy linenums="24" - script: - """ - echo "Processing: ${sample}" - cat ${input_file} > ${sample}_result.txt - """ - // ERROR: Missing closing brace for processFiles process - ``` - **Fix:** Add the missing closing brace - ```groovy linenums="29" - """ - echo "Processing: ${sample_id}" - cat ${input_file} > ${sample_id}_result.txt - """ - } // Add missing closing brace - ``` - - **Error 3: Variable Name Error** - ```groovy linenums="26" - echo "Processing: ${sample}" // ERROR: should be sample_id - cat ${input_file} > ${sample}_result.txt // ERROR: should be sample_id - ``` - **Fix:** Use the correct input variable name - ```groovy linenums="26" - echo "Processing: ${sample_id}" - cat ${input_file} > ${sample_id}_result.txt - ``` - - **Error 4: Undefined Variable Error** - ```groovy linenums="87" - heavy_ch = heavyProcess(sample_ids) // ERROR: sample_ids undefined - ``` - **Fix:** Use the correct channel and extract sample IDs - ```groovy linenums="87" - heavy_ch = heavyProcess(input_ch) - ``` - - At this point the workflow will run, but we'll still be getting errors (e.g. `Path value cannot be null` in `processFiles`), caused by bad channel structure. - - **Error 5: Channel Structure Error - Wrong Map Output** - ```groovy linenums="83" - .map { row -> row.sample_id } // ERROR: processFiles expects tuple - ``` - **Fix:** Return the tuple structure that processFiles expects - ```groovy linenums="83" - .map { row -> [row.sample_id, file(row.fastq_path)] } - ``` - - But this will break our for for running `heavyProcess()` above, so we'll need to use a map to pass just the sample IDs to that process: - - **Error 6: Bad channel structure for heavyProcess** - ```groovy linenums="87" - heavy_ch = heavyProcess(input_ch) // ERROR: input_ch now has 2 elements per emission- heavyProcess only needs 1 (the first) - ``` - **Fix:** Use the correct channel and extract sample IDs - ```groovy linenums="87" - heavy_ch = heavyProcess(input_ch.map{it[0]}) - ``` - - Now we get a but further but receive an error about `No such variable: i`, because we didn't escape a Bash variable. - - **Error 7: Bash Variable Escaping Error** - ```groovy linenums="48" - echo "Heavy computation $i for ${sample_id}" // ERROR: $i not escaped - ``` - **Fix:** Escape the bash variable - ```groovy linenums="48" - echo "Heavy computation \${i} for ${sample_id}" - ``` - - Now we get `Process exceeded running time limit (1ms)`, so we fix the run time limit for the relevant process: - - **Error 8: Resource Configuration Error** - ```groovy linenums="36" - time '1 ms' // ERROR: Unrealistic time limit - ``` - **Fix:** Increase to a realistic time limit - ```groovy linenums="36" - time '100 s' - ``` - - Next we have a `Missing output file(s)` error to resolve: - - **Error 9: Output File Name Mismatch** - ```groovy linenums="49" - done > ${sample_id}.txt // ERROR: Wrong filename, should match output declaration - ``` - **Fix:** Match the output declaration - ```groovy linenums="49" - done > ${sample_id}_heavy.txt - ``` - - The first two processes ran, but not the third. - - **Error 10: Output File Name Mismatch** - ```groovy linenums="88" - file_ch = channel.fromPath("*.txt") // Error: attempting to take input from the pwd rather than a process - handleFiles(file_ch) - ``` - **Fix:** Take the output from the previous process - ```groovy linenums="88" - file_ch = handleFiles(heavy_ch) - ``` - - With that, the whole workflow should run. - - **Complete Corrected Workflow:** - ```groovy linenums="1" - #!/usr/bin/env nextflow - - /* - * Buggy workflow for debugging exercises - * This workflow contains several intentional bugs for learning purposes - */ - - params{ - // Parameters with missing validation - input: Path = 'data/sample_data.csv' - output: String = 'results' - } - - /* - * Process with input/output mismatch - */ - process processFiles { - - input: - tuple val(sample_id), path(input_file) - - output: - path "${sample_id}_result.txt" - - script: - """ - echo "Processing: ${sample_id}" - cat ${input_file} > ${sample_id}_result.txt - """ - } - - /* - * Process with resource issues - */ - process heavyProcess { - - time '100 s' - - input: - val sample_id - - output: - path "${sample_id}_heavy.txt" - - script: - """ - # Simulate heavy computation - for i in {1..1000000}; do - echo "Heavy computation \$i for ${sample_id}" - done > ${sample_id}_heavy.txt - """ - } - - /* - * Process with file handling issues - */ - process handleFiles { - - input: - path input_file - - output: - path "processed_${input_file}" - - script: - """ - if [ -f "${input_file}" ]; then - cp ${input_file} processed_${input_file} - fi - """ - } - - /* - * Main workflow with channel issues - */ - workflow { - main: - // Channel with incorrect usage - input_ch = channel - .fromPath(params.input) - .splitCsv(header: true) - .map { row -> [row.sample_id, file(row.fastq_path)] } - - processed_ch = processFiles(input_ch) - - heavy_ch = heavyProcess(input_ch.map{it[0]}) - - file_ch = handleFiles(heavy_ch) - - publish: - processed = processed_ch - heavy = heavy_ch - files = file_ch - } - - output { - processed { - path 'processed' - } - heavy { - path 'heavy' - } - files { - path 'files' - } - } - ``` - -**Error Categories Covered:** - -- **Syntax errors**: Missing braces, trailing commas, undefined variables -- **Channel structure errors**: Wrong data shapes, undefined channels -- **Process errors**: Output file mismatches, variable escaping -- **Resource errors**: Unrealistic time limits - -**Key Debugging Lessons:** - -1. **Read error messages carefully** - they often point directly to the problem -2. **Use systematic approaches** - fix one error at a time and test with `-resume` -3. **Understand data flow** - channel structure errors are often the most subtle -4. **Check work directories** - when processes fail, the logs tell you exactly what went wrong - +title: Troubleshooting Workflows +hide: + - toc --- -## Summary - -In this side quest, you've learned a set of systematic techniques for debugging Nextflow workflows. -Applying these techniques in your own work will enable you to spend less time fighting your computer, solve problems faster and protect yourself from future issues. - -### Key patterns - -**1. How to identify and fix syntax errors**: - -- Interpreting Nextflow error messages and locating problems -- Common syntax errors: missing braces, incorrect keywords, undefined variables -- Distinguishing between Nextflow (Groovy) and Bash variables -- Using VS Code extension features for early error detection - -```groovy -// Missing brace - look for red underlines in IDE -process FOO { - script: - """ - echo "hello" - """ -// } <-- missing! - -// Wrong keyword -inputs: // Should be 'input:' - -// Undefined variable - escape with backslash for Bash variables -echo "${undefined_var}" // Nextflow variable (error if not defined) -echo "\${bash_var}" // Bash variable (escaped) -``` - -**2. How to debug channel structure issues**: - -- Understanding channel cardinality and exhaustion issues -- Debugging channel content structure mismatches -- Using `.view()` operators for channel inspection -- Recognizing error patterns like square brackets in output - -```groovy -// Inspect channel content -my_channel.view { "Content: $it" } - -// Convert queue to value channel (prevents exhaustion) -reference_ch = channel.value('ref.fa') -// or -reference_ch = channel.of('ref.fa').first() -``` - -**3. How to troubleshoot process execution problems**: - -- Diagnosing missing output file errors -- Understanding exit codes (127 for missing software, 137 for memory issues) -- Investigating work directories and command files -- Configuring resources appropriately +# Troubleshooting Workflows -```bash -# Check what was actually executed -cat work/ab/cdef12/.command.sh +Debugging is a critical skill that can save you hours of frustration and help you become a more effective Nextflow developer. +Throughout your career, especially when you're starting out, you'll encounter bugs while building and maintaining your workflows. -# Check error output -cat work/ab/cdef12/.command.err +This mini-course covers two complementary skills. +The first is recognising the shape of common errors, so that the error message in your terminal becomes a signpost rather than a wall. +The second is the toolkit of techniques you reach for when an error isn't immediately obvious from its message. -# Exit code 127 = command not found -# Exit code 137 = killed (memory/time limit) -``` +## Audience & prerequisites -**4. How to use Nextflow's built-in debugging tools**: +These lessons are aimed at Nextflow users who have completed the basics and want to debug their own workflows confidently. -- Leveraging preview mode and real-time debugging -- Implementing stub running for logic testing -- Applying resume for efficient debugging cycles -- Following a four-phase systematic debugging methodology +**Prerequisites** -!!! tip "Quick Debugging Reference" +- Completed the [Hello Nextflow](../../hello_nextflow/index.md) tutorial or equivalent. +- Comfortable with basic Nextflow concepts (processes, channels, operators). +- Docker installed (the examples use containerised processes). - **Syntax errors?** → Check VSCode warnings, run `nextflow run workflow.nf -preview` +**Optional:** We recommend completing the [IDE Features for Nextflow Development](../dev_environment/index.md) side quest first. +That covers comprehensive coverage of IDE features that support debugging (syntax highlighting, error detection, etc.), which we use heavily here. - **Channel issues?** → Use `.view()` to inspect content: `my_channel.view()` +**Working directory:** `side-quests/debugging` - **Process failures?** → Check work directory files: +## Learning objectives - - `.command.sh` - the executed script - - `.command.err` - error messages - - `.exitcode` - exit status (127 = command not found, 137 = killed) +By the end of this mini-course, you will be able to: - **Mysterious behavior?** → Run with `-stub-run` to test workflow logic +**Recognising common errors (Part 1):** - **Made fixes?** → Use `-resume` to save time testing: `nextflow run workflow.nf -resume` +- Read Nextflow error messages and locate the relevant code +- Recognise syntax errors, channel structure errors, and process errors +- Apply targeted fixes for each error category ---- +**The Nextflow debugging toolkit (Part 2):** -### Additional resources +- Inspect a process work directory to find out what actually ran +- Validate workflow logic before execution with `-preview` +- Stream process output in real time with `debug true` +- Iterate on workflow logic without running real commands using `-stub-run` +- Diagnose cache invalidation problems with `-dump-hashes` +- Apply a systematic four-phase debugging method -- [Nextflow troubleshooting guide](https://www.nextflow.io/docs/latest/troubleshooting.html): Official troubleshooting documentation -- [Understanding Nextflow channels](https://www.nextflow.io/docs/latest/channel.html): Deep dive into channel types and behavior -- [Process directives reference](https://www.nextflow.io/docs/latest/process.html#directives): All available process configuration options -- [nf-test](https://www.nf-test.com/): Testing framework for Nextflow pipelines -- [Nextflow Slack community](https://www.nextflow.io/slack-invite.html): Get help from the community +## Lesson plan -For production workflows, consider: +#### Part 1: Common errors and how to fix them -- Setting up [Seqera Platform](https://seqera.io/platform/) for monitoring and debugging at scale -- Using [Wave containers](https://seqera.io/wave/) for reproducible software environments +A catalogue of the most common errors you'll meet, with an example for each. +Read this one through, or use it as a reference when a specific error message lands in your terminal. -**Remember:** Effective debugging is a skill that improves with practice. The systematic methodology and comprehensive toolkit you've acquired here will serve you well throughout your Nextflow development journey. +#### Part 2: The Nextflow debugging toolkit ---- +Every tool is applied to the same small pipeline, so you can see how each tool fits into a real development workflow. +The lesson culminates with a `-dump-hashes` walkthrough of three cache-invalidation experiments, and a practical debugging exercise on an unfamiliar pipeline. -## What's next? +Ready to start? -Return to the [menu of Side Quests](../index.md) or click the button in the bottom right of the page to move on to the next topic in the list. +[Start with Part 1 :material-arrow-right:](01_common_errors.md){ .md-button .md-button--primary } diff --git a/docs/en/mkdocs.yml b/docs/en/mkdocs.yml index 41ab59ce40..f938fb51d6 100644 --- a/docs/en/mkdocs.yml +++ b/docs/en/mkdocs.yml @@ -70,7 +70,10 @@ nav: - side_quests/metadata/index.md - side_quests/splitting_and_grouping/index.md - side_quests/nf_test/index.md - - side_quests/debugging/index.md + - Troubleshooting Workflows: + - side_quests/debugging/index.md + - side_quests/debugging/01_common_errors.md + - side_quests/debugging/02_debugging_toolkit.md - side_quests/workflows_of_workflows/index.md - Plugin Development: - side_quests/plugin_development/index.md diff --git a/side-quests/debugging/sample_processing.nf b/side-quests/debugging/sample_processing.nf new file mode 100644 index 0000000000..82807b13de --- /dev/null +++ b/side-quests/debugging/sample_processing.nf @@ -0,0 +1,55 @@ +#!/usr/bin/env nextflow + +params.input = 'data/sample_data.csv' +params.outdir = 'results' + +process COUNT_LINES { + + container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' + publishDir params.outdir, mode: 'copy' + + input: + tuple val(sample_id), path(fastq) + + output: + path "${sample_id}.count.txt" + + script: + """ + lines=\$(zcat ${fastq} | wc -l) + cowpy "${sample_id} has \${lines} lines" > ${sample_id}_count.txt + """ + + stub: + """ + echo "${sample_id} has 0 lines" > ${sample_id}_count.txt + """ +} + +process REPORT { + + publishDir params.outdir, mode: 'copy' + + input: + path count_files + + output: + path 'report.txt' + + script: + """ + cat ${count_files} > report.txt + """ +} + +workflow { + + samples_ch = channel + .fromPath(params.input) + .splitCsv(header: true) + .map { row -> [row.sample_id, file(row.fastq_path)] } + + counts_ch = COUNT_LINES(samples_ch) + + REPORT(counts_ch.collect()) +} From 044db7af5e92ee1cf0798661affd2f3f9eff0af6 Mon Sep 17 00:00:00 2001 From: Jonathan Manning Date: Fri, 15 May 2026 18:44:43 +0100 Subject: [PATCH 2/2] Walkthrough fixes; add summary.md + next_steps.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found during /run-tutorial walkthrough of the new mini-course: Documentation fixes: - hl_lines corrections in lesson 2 (§1.3, §3.1, §2.2, §5.4): some were off by 1-5 lines, highlighting blank lines or closing quotes instead of the meaningful content. Recounted from line 1 of each snippet. - §2.2 syntax error block: updated from v1 parser format to v2 format matching what training environments actually produce (NXF_SYNTAX_PARSER=v2 / strict syntax parser default) - §2.2 "Broken" block: added placeholder comment so it matches "Working" line count, letting both highlight the same conceptual position - Process names (COUNT_LINES, REPORT) and directives (memory, cpus) backticked in prose (was bare in several places) - Two bare ??? "Command output" admonitions in §5 changed to ??? success "Command output" for consistency Mini-course structure (matching plugin_development pattern): - Added summary.md - course-wide recap with error/tool cheat sheets and three "easy to forget" reminders - Added next_steps.md - what to do next, resources, links to other training - Lesson 2 tail trimmed to a "continue to summary" button rather than repeating the recap inline - mkdocs.yml nav updated to include summary.md and next_steps.md Verified end-to-end via Docker (DooD): - L2 §1: missing-output failure + work-dir inspection + fix + success - L2 §2: -preview on working pipeline + preview catching missing brace - L2 §3: debug true + echo streams to terminal - L2 §4: -stub-run runs without docker - L2 §5: baseline -dump-hashes, comment busts cache, memory directive preserves cache, upstream .map change cascades downstream - L2 §7: buggy_workflow.nf produces the documented initial error - L1 spot-checks: bad_syntax, invalid_process, no_such_var, bad_resources, bad_channel_shape all produce their documented errors Co-Authored-By: Claude Opus 4.7 (1M context) --- .../debugging/02_debugging_toolkit.md | 76 +++++++------------ .../docs/side_quests/debugging/next_steps.md | 48 ++++++++++++ docs/en/docs/side_quests/debugging/summary.md | 71 +++++++++++++++++ docs/en/mkdocs.yml | 2 + 4 files changed, 150 insertions(+), 47 deletions(-) create mode 100644 docs/en/docs/side_quests/debugging/next_steps.md create mode 100644 docs/en/docs/side_quests/debugging/summary.md diff --git a/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md b/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md index c631160704..381c922c1e 100644 --- a/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md +++ b/docs/en/docs/side_quests/debugging/02_debugging_toolkit.md @@ -228,7 +228,7 @@ Fix the script and the stub block: === "After" - ```groovy title="sample_processing.nf" hl_lines="11 16 21" linenums="6" + ```groovy title="sample_processing.nf" hl_lines="10 15 20" linenums="6" process COUNT_LINES { container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' @@ -255,7 +255,7 @@ Fix the script and the stub block: === "Before" - ```groovy title="sample_processing.nf" hl_lines="11 16 21" linenums="6" + ```groovy title="sample_processing.nf" hl_lines="10 15 20" linenums="6" process COUNT_LINES { container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' @@ -344,7 +344,7 @@ Open `sample_processing.nf` and delete the closing brace at the end of the workf === "Broken" - ```groovy title="sample_processing.nf" hl_lines="10" linenums="45" + ```groovy title="sample_processing.nf" hl_lines="11" linenums="45" workflow { samples_ch = channel @@ -355,11 +355,12 @@ Open `sample_processing.nf` and delete the closing brace at the end of the workf counts_ch = COUNT_LINES(samples_ch) REPORT(counts_ch.collect()) + // missing closing brace ``` === "Working" - ```groovy title="sample_processing.nf" hl_lines="10" linenums="45" + ```groovy title="sample_processing.nf" hl_lines="11" linenums="45" workflow { samples_ch = channel @@ -386,13 +387,11 @@ nextflow run sample_processing.nf -preview Launching `sample_processing.nf` [...] DSL2 - revision: ... - ERROR ~ Script compilation error - - file : /workspaces/training/side-quests/debugging/sample_processing.nf - - cause: Unexpected input: '{' @ line 45, column 10. - workflow { - ^ + Error sample_processing.nf:53:1: Unexpected input: '' - 1 error + ERROR ~ Script compilation failed + + -- Check '.nextflow.log' file for details ``` The error is identical to what you'd see if you ran the workflow normally, but it took milliseconds and didn't pull a container or run any tasks. @@ -424,11 +423,11 @@ Combined with a strategic `echo`, this gives you the same kind of visibility you ### 3.1. Add `debug true` and an `echo` -Edit `sample_processing.nf` and add two lines to the COUNT_LINES process: +Edit `sample_processing.nf` and add two lines to the `COUNT_LINES` process: === "After" - ```groovy title="sample_processing.nf" hl_lines="5 18" linenums="6" + ```groovy title="sample_processing.nf" hl_lines="5 17" linenums="6" process COUNT_LINES { container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' @@ -530,7 +529,7 @@ This is invaluable when you're iterating on downstream logic and don't want to w ### 4.1. Look at the stub directive -The COUNT_LINES process in our pipeline already has one: +The `COUNT_LINES` process in our pipeline already has one: ```groovy title="sample_processing.nf" hl_lines="1-4" linenums="23" stub: @@ -558,7 +557,7 @@ nextflow run sample_processing.nf -stub-run ``` The pipeline runs end-to-end in seconds, producing the same output shape as the real run. -You can now iterate on REPORT (or any downstream change) without waiting on `cowpy`. +You can now iterate on `REPORT` (or any downstream change) without waiting on `cowpy`. ### Takeaway @@ -639,7 +638,7 @@ Any change to that text - including comments and whitespace - will produce a dif ### 5.2. Experiment 1 — a "harmless" comment -Add a single-line comment inside the COUNT_LINES script: +Add a single-line comment inside the `COUNT_LINES` script: === "After" @@ -668,7 +667,7 @@ Re-run with `-resume`: nextflow run sample_processing.nf -profile docker -resume -dump-hashes ``` -??? "Command output" +??? success "Command output" ```console executor > local (6) @@ -676,9 +675,9 @@ nextflow run sample_processing.nf -profile docker -resume -dump-hashes [43/4659bf] REPORT | 1 of 1 ✔ ``` -Despite using `-resume`, every COUNT_LINES task ran again. REPORT ran too, because its inputs depend on COUNT_LINES outputs. +Despite using `-resume`, every `COUNT_LINES` task ran again. `REPORT` ran too, because its inputs depend on `COUNT_LINES` outputs. -Look at the new hash for COUNT_LINES (1): +Look at the new hash for `COUNT_LINES (1)`: ```bash grep -A 18 "COUNT_LINES (1)" .nextflow.log @@ -704,7 +703,7 @@ Remove the comment before the next experiment. ### 5.3. Experiment 2 — a resource directive -Add a `memory` directive to COUNT_LINES: +Add a `memory` directive to `COUNT_LINES`: === "After" @@ -741,7 +740,7 @@ nextflow run sample_processing.nf -profile docker -resume Everything cached. The hashes are unchanged because resource directives like `memory`, `cpus`, and `time` are not part of the cache key. **Lesson:** tuning resources never busts the cache. -That makes resource adjustment cheap, but it also means you can't force a re-run by changing memory or CPUs - you'd need to touch something that _is_ hashed (such as the script body) or pass `-resume ` from an earlier session. +That makes resource adjustment cheap, but it also means you can't force a re-run by changing `memory` or `cpus` - you'd need to touch something that _is_ hashed (such as the script body) or pass `-resume ` from an earlier session. Remove the `memory` directive before the next experiment. @@ -751,7 +750,7 @@ Change the workflow's `.map { }` to upper-case the sample ID: === "After" - ```groovy title="sample_processing.nf" hl_lines="4" linenums="45" + ```groovy title="sample_processing.nf" hl_lines="6" linenums="45" workflow { samples_ch = channel @@ -767,7 +766,7 @@ Change the workflow's `.map { }` to upper-case the sample ID: === "Before" - ```groovy title="sample_processing.nf" hl_lines="4" linenums="45" + ```groovy title="sample_processing.nf" hl_lines="6" linenums="45" workflow { samples_ch = channel @@ -787,7 +786,7 @@ Re-run with `-resume`: nextflow run sample_processing.nf -profile docker -resume -dump-hashes ``` -??? "Command output" +??? success "Command output" ```console executor > local (6) @@ -796,7 +795,7 @@ nextflow run sample_processing.nf -profile docker -resume -dump-hashes ``` Every task ran again - including REPORT, which we didn't touch. -Look at one of the new COUNT_LINES hash entries: +Look at one of the new `COUNT_LINES` hash entries: ```console title="Excerpt" ... [java.lang.String] sample_id @@ -805,9 +804,9 @@ Look at one of the new COUNT_LINES hash entries: ... [nextflow.util.ArrayBag] [FileHolder(...sample_001.fastq.gz...)] ``` -The input value for `sample_id` is now `SAMPLE_001` rather than `sample_001`, so the COUNT_LINES task hash changed. -But why did REPORT re-run? -Because REPORT's input is the _files_ COUNT_LINES produced - and those files now live in different work directories (the new COUNT_LINES tasks). +The input value for `sample_id` is now `SAMPLE_001` rather than `sample_001`, so the `COUNT_LINES` task hash changed. +But why did `REPORT` re-run? +Because `REPORT`'s input is the _files_ `COUNT_LINES` produced - and those files now live in different work directories (the new `COUNT_LINES` tasks). A change upstream cascaded into a cache miss downstream, even though REPORT's own script and directives were untouched. **Lesson:** when you trace a cache miss, look upstream too. @@ -996,24 +995,7 @@ The four-phase method scales: start with cheap checks (`-preview`), let error me --- -## Summary - -You learned a sequence of debugging tools, each applied to the same small pipeline. - -| Tool | Best for | -| ----------------- | ----------------------------------------------------------- | -| Work directory | Any process failure - the complete record of what ran | -| `-preview` | Catching syntax and structural issues without running tasks | -| `debug true` | Seeing what a process actually receives at runtime | -| `-stub-run` | Iterating on workflow logic without real commands | -| `-dump-hashes` | Diagnosing unexpected cache invalidation | -| Four-phase method | A repeatable order in which to apply the tools | - -The single biggest debugging skill is matching the tool to the problem. -Most failures don't need every tool - they need the right one. - ---- - -## What's next? +You've reached the end of Part 2. +For a recap and quick-reference cheat sheet, continue to the summary. -Return to the [menu of Side Quests](../index.md) or click the button in the bottom right of the page to move on to the next topic in the list. +[Continue to the summary :material-arrow-right:](summary.md){ .md-button .md-button--primary } diff --git a/docs/en/docs/side_quests/debugging/next_steps.md b/docs/en/docs/side_quests/debugging/next_steps.md new file mode 100644 index 0000000000..1e0c5c0169 --- /dev/null +++ b/docs/en/docs/side_quests/debugging/next_steps.md @@ -0,0 +1,48 @@ +# Next Steps + +Congratulations on completing the **Troubleshooting Workflows** mini-course. + +--- + +## 1. Apply the toolkit to your own pipelines + +The fastest way to internalise these techniques is to reach for them on your own work. +Next time a pipeline of yours fails, before reaching for the search engine: + +- Read the error message all the way through, then find the work directory it cites. +- Walk through `.command.sh`, `.command.err`, `.command.out`, `.exitcode`, and `ls` before changing any code. +- If the error is opaque or absent, run with `-preview` and then `debug true` to confirm what your code actually sees. + +You'll find that the toolkit answers most questions on its own. + +--- + +## 2. Build debugging into your workflow design + +A few habits make pipelines easier to debug from the start: + +- Ship every process with a `stub:` directive so `-stub-run` always works. +- Add a `debug` profile to your `nextflow.config` (real-time output, `cleanup = false`, `maxForks = 1`) so a single flag gets you full visibility. +- Use `.view()` liberally during development. Remove it once channels are stable. + +--- + +## 3. Continue your Nextflow training + +If you haven't already, check out our other training material: + +- **[Hello Nextflow](../../hello_nextflow/index.md)** — Foundational Nextflow concepts +- **[Hello nf-core](../../hello_nf-core/index.md)** — nf-core pipelines and best practices +- **[nf-test](../nf_test/index.md)** — Add tests so future bugs are caught before they hit production +- **[Other Side Quests](../index.md)** — Deep dives into specific topics + +--- + +## Additional resources + +- [Nextflow troubleshooting guide](https://www.nextflow.io/docs/latest/troubleshooting.html) — official troubleshooting documentation +- [Nextflow channels reference](https://www.nextflow.io/docs/latest/channel.html) — channel types and behaviour +- [Process directives reference](https://www.nextflow.io/docs/latest/process.html#directives) — all available process configuration options +- [nf-test](https://www.nf-test.com/) — testing framework for Nextflow pipelines +- [Nextflow Slack community](https://www.nextflow.io/slack-invite.html) — help from the community +- [Seqera Platform](https://seqera.io/platform/) — monitoring and debugging at scale for production workflows diff --git a/docs/en/docs/side_quests/debugging/summary.md b/docs/en/docs/side_quests/debugging/summary.md new file mode 100644 index 0000000000..50d9ad1c76 --- /dev/null +++ b/docs/en/docs/side_quests/debugging/summary.md @@ -0,0 +1,71 @@ +# Summary + +You have completed the Troubleshooting Workflows mini-course. +This page recaps what you learned in each part and serves as a quick reference for future debugging sessions. + +--- + +## What you learned + +### Part 1: Common errors and how to fix them + +You worked through the most frequent Nextflow error categories, learning the shape of each error message and the typical fix. +You covered syntax errors (missing braces, wrong keywords, undefined variables, Bash vs Groovy variable handling, statements outside the workflow block), channel structure errors, and process structure errors (missing outputs, missing software, bad resource configuration). + +### Part 2: The Nextflow debugging toolkit + +Using a single spine pipeline you applied six techniques in sequence: + +- **Work-directory forensics** — `.command.sh`, `.command.err`, `.command.out`, `.exitcode` and a plain `ls` to reconstruct exactly what a failed process did +- **`-preview`** — fast parse-time validation that catches syntax errors and bad workflow structure before anything expensive runs +- **`debug true`** — streaming process output and echoes to your terminal as the process runs +- **`-stub-run`** — using stub directives to iterate on workflow logic without the real commands or their containers +- **`-dump-hashes`** — finding out exactly which input component changed when `-resume` re-runs more than you expected +- **A four-phase systematic method** that combines all of the above + +--- + +## Quick reference + +### Error shapes (Part 1) + +| Hallmark | Likely cause | +| ----------------------------------------------------- | ------------------------------------------- | +| `Unexpected input: ''` | Missing closing brace | +| `Invalid process definition` | Wrong section keyword (`inputs` vs `input`) | +| `` `name` is not defined `` | Typo or missing Groovy variable | +| `is not defined` on a name set in Bash | Bash variable not escaped (`\$name`) | +| `Statements cannot be mixed with script declarations` | Channel definition outside `workflow { }` | +| `Process requires N channels, M were specified` | Cardinality mismatch at process call | +| Stray `[ ]` or `Path value cannot be null` | Channel emits wrong tuple shape | +| `Missing output file(s)` | Output declaration vs script filename | +| Exit code 127, `command not found` | Software not in container/conda | +| Exit code 137, `exceeded running time limit` | Resource directive too tight | + +### Tool cheat sheet (Part 2) + +| Tool | Reach for it when… | +| ------------------------ | ------------------------------------------------------------------------------------------ | +| `work//.command.*` | A process failed and you need to see exactly what ran | +| `-preview` | You just edited the workflow and want a fast structural sanity check | +| `debug true` + `echo` | A process completes but produces something unexpected | +| `-stub-run` | You're iterating downstream and don't want to wait on slow or containerised upstream tasks | +| `-dump-hashes` | `-resume` re-ran tasks you thought were cached | +| `nextflow.log` | Anything that doesn't show up in the terminal output | + +### The four-phase method + +1. **Parse first** — `nextflow run workflow.nf -preview` +2. **Read the error** — categorise as structural, runtime, or resource +3. **Investigate** — walk the work directory, add `.view()` / `debug true`, use `-stub-run` if iterating +4. **Fix and verify** — minimal change, re-run with `-resume`, use `-dump-hashes` if cache misses surprise you + +--- + +## What's most worth remembering + +Three things that are easy to forget: + +1. **The work directory has everything.** Any time a process fails, your first move is to find the work directory in the error message and `ls` it. +2. **Any text change inside `script:` busts the cache, including comments and whitespace.** Resource directives don't. +3. **A change upstream can cascade into cache misses downstream**, even when the downstream process is untouched, because its inputs are now different files. diff --git a/docs/en/mkdocs.yml b/docs/en/mkdocs.yml index f938fb51d6..5f0ac69202 100644 --- a/docs/en/mkdocs.yml +++ b/docs/en/mkdocs.yml @@ -74,6 +74,8 @@ nav: - side_quests/debugging/index.md - side_quests/debugging/01_common_errors.md - side_quests/debugging/02_debugging_toolkit.md + - side_quests/debugging/summary.md + - side_quests/debugging/next_steps.md - side_quests/workflows_of_workflows/index.md - Plugin Development: - side_quests/plugin_development/index.md