Skip to content

Add archiver - #1043

Merged
wangyinz merged 4 commits into
masterfrom
add_archiver
Oct 2, 2026
Merged

wangyinz merged 4 commits into
masterfrom
add_archiver

Conversation

@pavlis

@pavlis pavlis commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

This adds a new feature to allow more portable saves of data sets to files. A key idea is to create data files that have some longevity in an archive and which make it easier to transfer a data set between systems. This new module handles this in several distinct ways:

  1. The lowest level functions are a writer that saves an ensemble in memory to a pair of data files (one for the sample data and a json file for metadata). It has an alter ego in a reader to read such file pairs to construct a TimeSeriesEnsemble or SeismogramEnsemble object.
  2. There is an indexing function that takes data stored in the database as "binary" with {"storage_mode" : "files"} and writes json files for each sample data file (dfile attribute of wf documents) using wf documents as a base and adding a couple of other necessary attributes. That allows creating of an archive in place without having to copy the more massive sample data. It only duplicates metadata in the json files it produces.
  3. There is a special function to handle gridfs data. Approach 2 cannot work with gridfs data as there are no external files. This function has some complexities to handle potentially very large data sets stored in gridfs. See the docstring and the code for how that is handled.

This is the initial push of this new feature. Two things are completely lacking and need to be done before this branch should be merge:

  1. There is no pytest suite yet for this module. I developed a simple test for 1 and 2 above with some data I had on my system, but those tests are not sufficient and would have required adding some more data files to the repository. A thorough pytest suite needs to be built that doesn't need real data to run. Note the gridfs function (item 3) is completely untested. It does pass spyder's lint filters but needs some real tests to be suitable for release.
  2. There is no documentation in docs for this feature. This new module has multiple potential uses and need a good documentation page to describe what all it can do and how to use it. We should not release this until that is in the user manual.

@codecov

codecov Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.58996% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.99%. Comparing base (c53d735) to head (84f5ac5).

Files with missing lines Patch % Lines
python/mspasspy/io/archive.py 81.58% 44 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1043      +/-   ##
==========================================
+ Coverage   74.94%   74.99%   +0.04%     
==========================================
  Files         184      185       +1     
  Lines       37795    38034     +239     
  Branches     2565     2565              
==========================================
+ Hits        28324    28522     +198     
- Misses       8548     8589      +41     
  Partials      923      923              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wangyinz
wangyinz merged commit 65ebb1f into master Oct 2, 2026
23 checks passed
@wangyinz
wangyinz deleted the add_archiver branch October 2, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants