Repository navigation
Question about memory explosion in my R package #662
Replies: 1 comment 1 reply
|
Hi @mattwarkentin, I'm going to provide the general patterns for dealing with memory usage. By way of explanation, mirai with default
Of course, a simpler solution may just be to break up your map and manually chunk the work to keep memory usage more manageable. |
Uh oh!
There was an error while loading. Please reload this page.
Hi @shikokuchuo,
I have used
miraiin my R package,orthanc, to be able to perform API requests/queries in parallel but I am noticing an issue of the memory use exploding when running in parallel (i.e., settingmirai::daemons()versus a main process sequential run.miraiis used by way ofpurrr::in_parallel()inside apurrr::map(). So whendaemons()is not set, it performs the map sequential in the main process and, for this example I'm running right this second, the memory is very low (~1GB, lets say) according to the Positron memory profiler. However, when I setdaemons()and it runs in separate child processes, the main process shoots up to 60+ GB of memory (even though each child process doing the actual work is seemingly low footprint).I don't know if you have a suggestion for how I can profile this better to find out where the memory issue is happening in my code. I can't understand why this would happen.
The code for this parallelized function is here, for what that's worth: https://github.com/mattwarkentin/orthanc/blob/main/R/filtering.R
The job of the function is fairly straightforward. We have a list of objects (
Patients) and we loop over them (potentially in parallel) to apply predicate functions to remove certain patients/studies/series/instances that do not meet the logical condition.Here is some example code:
Any guidance would be greatly appreciated!
All reactions