Skip to content

Add new s3 modules - #1052

Open
pavlis wants to merge 14 commits into
masterfrom
add_new_s3_modules
Open

pavlis wants to merge 14 commits into
masterfrom
add_new_s3_modules

Conversation

@pavlis

@pavlis pavlis commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

This branch creates a new module I've put in mspasspy.io.FDSN. As the name suggests it is aimed to standardize interaction with FDSN networks. In the initial push it contains two major new features:

  1. A new data object called ChannelPreferences. It is based on something I had in my older C++ code base that I then called a "StationChannelMap". The main purpose of this object is to sort out ambiguous channel codes that are ubiquitous in FDSN data. seed uses channel codes and loc codes together to distinguish sample rate, orientation, and sensor. This provides a way to define a hierarchy of which channels to select when there are many to choose from. This comes up quickly in dealing with Earthscope data as they bundle data they serve through S3 in station bundles. Those can be huge and you almost never want to pull all the extra channels along as baggage. This object provides a general way to do that at the cost of being more than a little complicated.
  2. A new data object called S3SeismicProvider standardizes the API for accessing FDSN data store as Amazon S3 objects.
  3. It adds two yaml files that are defaults for the two classes noted.

The initial push is not complete and is not remotely ready to merge. I'm pushing it mainly because I had make a change to a low level class in the mspasspy.db.normalize module called DataFrameCacheMatcher. I created a special matcher class in the FDSN module that uses a DataFrame index. It turns out the current implementation breaks if you use an input DataFrame that has an index defined. This initial push may break the tests for the normalize module.

@pavlis

pavlis commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

PS: We should consider extending the DataFrameCacheMatcher to have a method to add an index to the DataFrame. Implementations of the find method that require the subset method implementation are theoretically orders of magnitude faster for large DataFrames.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant