Skip to content

Thread scaling improvements#123

Open
BenLangmead wants to merge 44 commits into
masterfrom
batch_parsing_output
Open

Thread scaling improvements#123
BenLangmead wants to merge 44 commits into
masterfrom
batch_parsing_output

Conversation

@BenLangmead

@BenLangmead BenLangmead commented Jul 18, 2017

Copy link
Copy Markdown
Owner

This branch has many improvements related to thread scaling. Among them:

  • Move away from thread groups, and to individually constructed & detached threads
  • Remove heap allocation in ThreadSafe constructor
  • Remove other unnecessary heap allocations

ChristopherWilks and others added 29 commits February 18, 2016 21:47
@ChristopherWilks
added working version of load balancing for Rail in EMR, note hardcod…
49a6a48 
…ed pid_dir and thread_ceiling is set to ncpus-2
@ChristopherWilks
applying the latest bug fix from the fix_parallel_build_support_for_t…
e0959b8 
…bb branch
@ChristopherWilks
added rlb to VERSION
a416046
@ChristopherWilks
changed to using ps to get pids, seeing an abort at the very end, aft…
9cd4c99 
…er the ends have been produced
@ChristopherWilks
merged the paralell fix branch code
ef0159a
@ChristopherWilks
added TBB support for the load balancing code, fixed ending segfault …
975cef4 
…due to queue not being thread safe
@ChristopherWilks
re-enabled atomic support for thread_count when WITH_TBB=1, cleaned u…
b0f8fdd 
…p code, re-enabled del_pid when process done
@BenLangmead
merged master into rail load balancing
04855cf
@BenLangmead
fixing some errors in the merge
84fdb20
@BenLangmead
fixing more errors in merging; switched from stat to kill to be mac c…
9f195e7 
…ompatible
@BenLangmead
several formatting improvements and bug fixes
d3ac765
@BenLangmead
removing unneeded includes and replacing nthreads > 0 conditionals
4a22f90
@BenLangmead
addressing warnings and error in WITH_TBB=1 mode
60e1f46
@BenLangmead
small cleanups
c13d06a
@BenLangmead
added --thread-piddir and got rid of hard coded directory for thread …
988b355 
…stealing
@ChristopherWilks
first commit of tbb::threads + 10ms delay
8a1afe0
@ChristopherWilks
output buffer with user defined size in reads
c1ca69d
@ChristopherWilks
mem page pool used by AlignmentCache now allocated in one big chunk
c10f3ab
@ChristopherWilks
added master travis config
882ce84
@ChristopherWilks
commenting out readline and gzip compression to allow for easy portab…
b2216a9 
…ility
moved ThreadSafe to outside the loop
1e1c8d6
@ChristopherWilks
ported stack mutex; removed readsPerBatchOutput option; fixed nthread…
b6cb7b0 
…s+1 bug
@ChristopherWilks
fixed thread timing no-output race condition
dfeaf84
@BenLangmead
merge master into rail_load_balancing
d4debcd
@BenLangmead
merge 2.3.2 commits into rail_load_balancing
1450dce
@BenLangmead
stability and error checking enhancements for thread stealing
ff9a750
@BenLangmead
merging in master
d53e053
@BenLangmead
another master merge
017bbb8 
Merge branch 'master' into batch_parsing_output
@BenLangmead
merging
6d01f37 
Merge branch 'master' into rail_load_balancing
@BenLangmead

BenLangmead commented Jul 18, 2017

Copy link
Copy Markdown
Owner Author

One of the areas where we changed the use of ThreadSafe a lot was when compiling metrics, as controlled by the --met* options. Some effort should be put into testing that we haven't created race conditions there.

BenLangmead and others added 15 commits July 25, 2017 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@BenLangmead @ChristopherWilks @ch4rr0