Skip to content

increase ulimit for cutadapt #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vmikk wants to merge 1 commit into
base: test-v1.2.0
Choose a base branch
from
Open

increase ulimit for cutadapt #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/pipecraft-core/service_scripts/demux_paired_end_data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ output_dir=$"/input/demultiplex_out"
#python module for assigning sample names as based on indexes file
run_python_module=$"python3 /scripts/submodules/assign_sample_names.demuxModule.py $indexes_file"

# Increase the number of open files limit
ulimit -S -n 6000
Copy link

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command ulimit -S -n 6000 only raises the soft limit for open files. The soft limit can only be raised up to the current hard limit. If the container's hard limit is lower than 6000 (which on some Docker/OS configurations may equal the default of 1024), this command will fail with an error like bash: ulimit: open files: cannot modify limit: Operation not permitted or silently do nothing.

To ensure this works reliably across all deployment environments, consider also setting the hard limit (ulimit -H -n 6000), which requires root/privileged access (common in containers), or — more robustly — setting the nofile ulimit at the Docker container creation level in HostConfig.Ulimits. Additionally, if users run large multiplexing experiments (e.g., 96×96 adapters for paired-end), they would need 96×96×2 ≈ 18,432 file descriptors, which would exceed the hardcoded 6000 limit. Consider making this value dynamically computed based on the number of adapters or using a higher ceiling such as 65536.

Suggested change
ulimit -S -n 6000
# Allow override via DESIRED_NOFILE; default to a higher ceiling suitable for large multiplexing
DESIRED_NOFILE="${DESIRED_NOFILE:-65536}"
# Get current hard limit
current_hard_limit=$(ulimit -H -n 2>/dev/null || echo "")
if [ -n "$current_hard_limit" ]; then
# Try to raise hard limit to desired value (may fail without sufficient privileges)
if [ "$current_hard_limit" != "unlimited" ] && [ "$current_hard_limit" -lt "$DESIRED_NOFILE" ] 2>/dev/null; then
ulimit -H -n "$DESIRED_NOFILE" 2>/dev/null || true
current_hard_limit=$(ulimit -H -n 2>/dev/null || echo "")
fi
# Clamp soft limit to the (possibly updated) hard limit
target_soft_limit="$DESIRED_NOFILE"
if [ "$current_hard_limit" != "unlimited" ] && [ "$current_hard_limit" -lt "$target_soft_limit" ] 2>/dev/null; then
target_soft_limit="$current_hard_limit"
fi
ulimit -S -n "$target_soft_limit" 2>/dev/null || true
fi

Copilot uses AI. Check for mistakes.

#############################
### Start of the workflow ###
#############################
Expand Down
3 changes: 3 additions & 0 deletions src/pipecraft-core/service_scripts/demux_single_end_data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ source /scripts/submodules/framework.functions.sh
#output dir
output_dir=$"/input/demultiplex_out"

# Increase the number of open files limit
ulimit -S -n 6000

Comment on lines +44 to +46
Copy link

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same concern as in the paired-end script: ulimit -S -n 6000 only raises the soft limit. If the container's hard limit is below 6000, this command will fail. For a single-end scenario with many adapters (e.g., 96+), even 6000 might not be sufficient (96 adapters × 1 output file each + communication overhead could be fine, but the risk grows with scale).

Consider using a higher, more future-proof value (e.g., 65536) or dynamically computing the required value based on the adapter count.

Suggested change
# Increase the number of open files limit
ulimit -S -n 6000
# Increase the number of open files limit (respect hard limit and avoid lowering current soft limit)
desired_nofiles=65536
hard_limit=$(ulimit -H -n 2>/dev/null)
soft_limit=$(ulimit -S -n 2>/dev/null)
# Determine target_limit: do not exceed hard limit (if numeric), and do not go below current soft limit
if [ "$hard_limit" = "unlimited" ] || [ "$hard_limit" = "unlimited\n" ]; then
target_limit=$desired_nofiles
else
target_limit=$desired_nofiles
if [ "$hard_limit" -lt "$desired_nofiles" ] 2>/dev/null; then
target_limit=$hard_limit
fi
fi
# Never lower the existing soft limit
if [ "$soft_limit" -gt "$target_limit" ] 2>/dev/null; then
target_limit=$soft_limit
fi
ulimit -S -n "$target_limit" 2>/dev/null || {
printf 'Warning: unable to increase open file limit; continuing with current limits.\n' >&2
}

Copilot uses AI. Check for mistakes.
#############################
### Start of the workflow ###
#############################
Expand Down
Loading