[wangyinz]

Summary

• Write ProcessingHistory, Inventory/Response, and PowerSpectrum payloads as versioned BSON-native documents by default.
• Auto-detect and read legacy pickle bytes in every normal reader, including mixed collections where BSON and pickle records coexist.
• Keep SpectrumDatabase.save_data(format="pickle") as an explicit compatibility option; its default is now "bson".
• Preserve PR Fix inventory location grouping and lookup #962's location grouping: each site payload contains only its location's channels, while channel documents retain their exact metadata.

Compatibility

New MsPASS releases read both old pickle records and new BSON records. Existing databases therefore require no migration and writes are not duplicated. Older MsPASS releases are not expected to read newly written BSON records.

Validation

• Local legacy history, Inventory, and Response decoding tests pass.
• Local explicit pickle-write, invalid-format, and location-grouping tests pass.
• Python compilation, Black formatting, and diff checks pass.
• Full PowerSpectrum and MongoDB integration coverage runs in GitHub CI, which rebuilds the updated C++ bindings.

Closes #841

[wangyinz]

Draft decision required (renewed review)

A maintainer must approve the pickle-to-BSON version/deprecation/migration/rollback policy, integrate accepted per-location inventory grouping, and use payload-level compare-and-set migration. Normal reads and writes must not break existing databases; keep this PR draft until that plan is complete.

[wangyinz]

Human decision required: persistence and migration policy. Recommendation: normal readers never execute pickle fallback; new writes use versioned BSON only, without dual writes; legacy conversion is an explicit trusted migration; and Inventory is migrated losslessly only when a complete trusted source exists. Any lossy synthetic fallback should be a separate explicit tool/PR. This Draft also still needs #962 integration, payload-level CAS updates, mixed-document handling, strict native decoder reuse, rollout documentation, and full CI.

[wangyinz]

Maintainer decision (supersedes the previous review note): legacy pickle persistence remains supported. Revised contract: readers auto-detect a versioned BSON document versus pickle bytes and decode either; new writes default to BSON; the established explicit SpectrumDatabase.save_data(format="pickle") option remains supported; no dual write is required; and in-place migration is optional because old records remain directly readable. Mixed documents must resolve each payload independently. This PR remains Draft until #962 is integrated and the compatibility tests and full CI pass.

[wangyinz]

Final review decision: backward compatibility is required. Normal readers now auto-detect versioned BSON documents and legacy pickle bytes per payload; new writes default to BSON, while SpectrumDatabase retains an explicit pickle output option. The previous reject-old-data and explicit migration design was removed because it added risk and was unnecessary once legacy records remain readable. No human policy decision remains; this PR stays draft only until the rebuilt C++/Python CI matrix passes.

[codecov]

Codecov Report

❌ Patch coverage is 74.83871% with 78 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.44%. Comparing base (92ca6cf) to head (d434e81).

Files with missing lines	Patch %	Lines
python/mspasspy/db/serialization.py	76.95%	56 Missing ⚠️
cxx/test/history/test_history.cc	36.36%	8 Missing and 6 partials ⚠️
cxx/python/utility/utility_py.cc	53.84%	4 Missing and 2 partials ⚠️
python/mspasspy/algorithms/calib.py	66.66%	1 Missing ⚠️
python/mspasspy/db/spectrumdb.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master     #989    +/-   ##
========================================
  Coverage   73.44%   73.44%            
========================================
  Files         182      183     +1     
  Lines       36160    36448   +288     
  Branches     2532     2540     +8     
========================================
+ Hits        26559    26771   +212     
- Misses       8696     8764    +68     
- Partials      905      913     +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.