Maximum LCIO event size / file size of 4 GB?

[alexandertuna]

Hi LCIOers,

Do you know if there's a maximum file/event size of 4 GB? I am generating some muon collider events with a large background, and I noticed things start to break when the event size (I have one event per file) surpasses 4-ish GB:

Singularity> du -hs neutrinoGun_digi_3900MB.slcio neutrinoGun_digi_4200MB.slcio
3.9G	neutrinoGun_digi_3900MB.slcio
4.2G	neutrinoGun_digi_4200MB.slcio
Singularity>
Singularity> lcio_event_counter neutrinoGun_digi_3900MB.slcio
1
Singularity> lcio_event_counter neutrinoGun_digi_4200MB.slcio
1
Singularity> lcio_check_col_elements MCParticle neutrinoGun_digi_3900MB.slcio
 found 1 MCParticle element(s) in 1 event(s)
 [ blah blah ]
Singularity> lcio_check_col_elements MCParticle neutrinoGun_digi_4200MB.slcio
terminate called after throwing an instance of 'sio::exception'
  what():  /tmp/root/spack-stage/spack-stage-sio-0.2-2n4u2dalexze6gcn7qhp5buf5kc6kkdo/spack-src/source/src/io_device.cc (l.34) in seek: Can't seek device cursor: out of range! [out_of_range]
/tmp/root/spack-stage/spack-stage-lcio-2.22.5-jygurrobznbsj3xhqvkxva4ariufnr2c/spack-src/src/cpp/src/MT/LCReader.cc (l.201) in readNextEvent: Couldn't read next event! [out_of_range]
Aborted (core dumped)
Singularity>

Maybe this is related to #71 but I'm not sure

Thanks for any insight!
Alex

[tmadlener]

Did the writing of the event work without issues? If that is the case this might be related to #71 but it seems to be a different issue, as that happens during output, but you seem to be running into issues while reading. It would be interesting to step through the whole thing with a debugger to see what pos and _buffer.size() report here:

https://github.com/iLCSoft/SIO/blob/9368e91bb3f4a99e8286d26dc7a425f216ff97d7/source/src/io_device.cc#L33

[alexandertuna]

I believe writing the 4.2GB event worked normally. There are no error messages, and there are success messages like:

MyAIDAProcessor      INFO Finalising Output_REC
[ MESSAGE4 "Output_REC"]
[ MESSAGE4 "Output_REC"] LCIOOutputProcessor::end()  Output_REC: 1 events in 0 runs written to file ./neutrinoGun_digi_0.slcio
[ MESSAGE4 "Output_REC"]

[ ... ]

[ MESSAGE "EventNumber"] Statusmonitor::end()  EventNumber processed 1 events in 0 runs
Output_REC           INFO Finalising MyAIDAProcessor
ApplicationMgr       INFO Application Manager Finalized successfully
ApplicationMgr       INFO Application Manager Terminated successfully with a user requested ScheduledStop

(Output_REC is the file writer)

[alexandertuna]

@tmadlener is it helpful to have access to the 4GB file? If so, it's here:

https://uaf-3.t2.ucsd.edu/~atuna/muoncollider/data/debug/neutrinoGun_digi_4200MB.slcio

[dudarboh]

I think I managed to reproduce it.
I have created a file with a lot of MCParticles in the single collection. And at some points, I cannot read the file anymore. However, the error message is slightly different for me.
But both errors come from LCReader.cc (l.201), if it is not the same bug, at least they are connected.

Didn't have time to debug further with gdb for now

Error

terminate called after throwing an instance of 'sio::exception'
  what():  /cvmfs/sw-nightlies.hsf.org/key4hep/releases/2025-12-21/x86_64-almalinux9-gcc14.2.0-dbg/sio/9368e91bb3f4a99e8286d26dc7a425f216ff97d7_develop-djl7xp/include/sio/api.h (l.375) in read: Can't read 24 bytes out of buffer (pos=1990746984) [invalid_argument]
/tmp/root/spack-stage/spack-stage-lcio-d45d898b7571ba7a2eff5353616f9651f1530937_develop-atr5oylkvctupauufwdfmlsvjtcgk74b/spack-src/src/cpp/src/SIO/SIOCollectionHandler.cc (l.63) in read: Couldn't read out object of type 'MCParticle' at index 22622124 [io_failure]
/tmp/root/spack-stage/spack-stage-sio-9368e91bb3f4a99e8286d26dc7a425f216ff97d7_develop-djl7xppxzam62ey5d26wg2jk3dhfbkmk/spack-src/source/src/api.cc (l.305) in read_blocks: Failed to decode block buffer (MyCollection) [io_failure]
/tmp/root/spack-stage/spack-stage-lcio-d45d898b7571ba7a2eff5353616f9651f1530937_develop-atr5oylkvctupauufwdfmlsvjtcgk74b/spack-src/src/cpp/src/MT/LCReader.cc (l.201) in readNextEvent: Couldn't read next event! [io_failure]

Lazy reproducer (using my file)

# note: unzipped it weights around 6GB
wget https://cernbox.cern.ch/remote.php/dav/public-files/Pb622mQxertfvBr/out.tar.xz
tar -xf out.tar.xz
source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh -r 2026-01-23 -d
# root main.cpp
# or
g++ -g -o main main.cpp $(root-config --libs --cflags) -I${LCIO}/include -llcio -L${LCIO}/lib64
gdb main

// main.cpp
#include "IOIMPL/LCFactory.h"

void main() {
    auto* reader = IOIMPL::LCFactory::getInstance()->createLCReader();
    reader->open("out.slcio");
    auto* read_evt = reader->readNextEvent();
    std::cout<<read_evt<<std::endl;
}

Full Reproducer

Note that you need more than 22 GB of RAM to run it.

source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh -r 2026-01-23 -d
root test.cpp

//test.cpp
#include "IO/LCWriter.h"
#include "IOIMPL/LCFactory.h"
#include "IMPL/LCEventImpl.h"
#include "IMPL/LCCollectionVec.h"
#include "IMPL/MCParticleImpl.h"
#include "IMPL/LCRunHeaderImpl.h"

int parseLine(char* line){
    // This assumes that a digit will be found and the line ends in " Kb".
    int i = strlen(line);
    const char* p = line;
    while (*p <'0' || *p > '9') p++;
    line[i-3] = '\0';
    i = atoi(p);
    return i;
}

int getPhysicalMemoryUsage(){ //Note: this value is in KB!
    FILE* file = fopen("/proc/self/status", "r");
    int result = -1;
    char line[128];

    while (fgets(line, 128, file) != NULL){
        if (strncmp(line, "VmRSS:", 6) == 0){
            result = parseLine(line);
            break;
        }
    }
    fclose(file);
    return result;
}

void test() {
    auto* writer = IOIMPL::LCFactory::getInstance()->createLCWriter();
    writer->setCompressionLevel(0);
    writer->open("out.slcio", EVENT::LCIO::WRITE_NEW );
    
    //1 Gb vs 4 Gb
    auto obj_size = sizeof(EVENT::MCParticle);
    // auto num_particles = uint64_t(1.0 * 1000 * 1000 * 1000 / obj_size); // works
    int num_particles = int(4.0 * 1000 * 1000 * 1000 / obj_size);

    auto* col = new IMPL::LCCollectionVec(EVENT::LCIO::MCPARTICLE);
    for (int i = 0; i < num_particles; i++) {        
        if (i % (num_particles/20) == 0) std::cout<<getPhysicalMemoryUsage()/1000.<<" MB \n";
        auto* p = new IMPL::MCParticleImpl();
        col->addElement(p);
    }
    
    auto evt = IMPL::LCEventImpl();
    evt.addCollection(col, "MyCollection");    
    writer->writeEvent(&evt);
    writer->close();

    auto* reader = IOIMPL::LCFactory::getInstance()->createLCReader();
    reader->open("out.slcio");
    auto* read_evt = reader->readNextEvent();
    std::cout<<read_evt<<std::endl;
}

[dudarboh]

From a closer look, it looks like unsigned int overflow.

From the working example:
For 23,214,285 (MCParticles) * 88 bytes (per-particle) + 52 bytes = 2,042,857,132 buffer is allocated - math works

From the not-working example:
For 71,428,571 (MCParticles) * 88 bytes (per-particle) + 52 bytes = 1,990,747,004 (actually allocated)
For 71,428,571 (MCParticles) * 88 bytes (per-particle) + 52 bytes = 6,285,714,300 (must-have been allocated)

And this difference is exactly the unsigned int overflow

unsigned int a = 6285714300  // 1990747004

Looking at the code I think this variable is to blame:

LCIO/src/cpp/include/MT/LCWriter.h

Line 107 in d45d898

std::atomic<unsigned int> _maxBufferSize {2*1204*1024} ;

If I read the code correctly, this buffer length is written in the run header here:

LCIO/src/cpp/src/MT/LCWriter.cc

Lines 93 to 97 in d45d898

	auto buflen = _maxBufferSize.load() ;
	sio::buffer rawBuffer( buflen ) ;
	// write the record in the buffer
	sio::record_info recinfo {} ;
	SIO::SIORunHeaderRecord::writeRecord( rawBuffer, hdr, recinfo, 0 ) ;

Which is then read from the file here:

https://github.com/iLCSoft/SIO/blob/9368e91bb3f4a99e8286d26dc7a425f216ff97d7/source/src/api.cc#L105C27-L105C39

So it seems like a bug in writing wrong max_buffer_length in the r̶e̶a̶d̶e̶r̶ writer (reader was a typo).

I assume

std::atomic<std::size_t>                   _maxBufferSize {2*1204*1024} ; 

should be enough to fix this, but I didn't test it

[dudarboh]

Maybe something like this could be useful in the future:
https://stackoverflow.com/questions/76840686/how-can-i-get-a-warning-when-comparing-unsigned-integers-of-different-sizes-in-c

[tmadlener]

Thanks for debugging this to this level @dudarboh. I will try to see if fixing sio indeed makes this work. I will also check if there is a warning that could be enabled to catch this earlier the next time.

[tmadlener]

The simple fix of @dudarboh doesn't seem to do the trick at least for the original issue. The problem is also present with "pure" sio, so I suppose also the fix needs to go there, e.g.

$ ~/work/iLCSoft/SIO/install/bin/sio-dump -d neutrinoGun_digi_4200MB.slcio
---------------------------------------------------------------------------------------------------------------------
Record name                    | Start           | End             | Options      | Header len | Data len       
LCEventHeader                  | 0               | 1168            | 0            | 40         | 1128 (1128)    
---------------------------------------------------------------------------------------------------------------------
Block name                     | Start           | End             | Version      | Header len | Data len       
---------------------------------------------------------------------------------------------------------------------
EventHeader                    | 0               | 1128            | 2.22         | 28         | 1100           

---------------------------------------------------------------------------------------------------------------------
Record name                    | Start           | End             | Options      | Header len | Data len       
LCEvent                        | 1168            | 153041592       | 0            | 32         | 153040392 (153040392)
---------------------------------------------------------------------------------------------------------------------
Block name                     | Start           | End             | Version      | Header len | Data len       
---------------------------------------------------------------------------------------------------------------------
ERROR: <prefix>/SIO/source/src/io_device.cc (l.39) in seek: Can't seek device cursor: out of range! [out_of_range]

[dudarboh]

@tmadlener, my assumption was that the "Data len" (153040392 in this example) is wrongly calculated by LCWriter on file creation step.

Fixing the overflow in the LCWriter will not fix the wrong values that have already been written to the files.
The question is, whether the new file produced with this fix becomes readable or not.

I guess one could implement recovery of the proper Data len in the corrupted files, but that sounds like a quite of work already... But the simpliest solution would be:

catch (Exception& buffer_overflow_exception){
// try again with
data_len = += std::numeric_limits<unsigned int>::max();
// until it works...
}

or, in principle, one knows the N elements in all collections on read. So one can make an educated guess by how much the buffer was overflown, by comparing the buffer in the header with the manually calculated one:

std::size_t expected_buffer_size{}; // approximately
for (auto col : cols){
    expected_buffer_size += sizeof( col.getType() ) * col.size()
}

if (data_len + some_margin < expected_buffer_size) // fix corrupted data_len

EDIT: This makes me even think that writing the buffer to the file header is, in principle redundant, as the buffer can be estimated from the other information that is also present in the headers (n_elements, and class types).
But maybe I am missing something, why it is necessary.