Repository navigation
Add new s3 modules - #1052
Open
pavlis wants to merge 14 commits into
Open
Add new s3 modules#1052pavlis wants to merge 14 commits into
pavlis wants to merge 14 commits into
Conversation
Collaborator
Author
|
PS: We should consider extending the DataFrameCacheMatcher to have a method to add an index to the DataFrame. Implementations of the find method that require the subset method implementation are theoretically orders of magnitude faster for large DataFrames. |
…llel features need testing
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch creates a new module I've put in mspasspy.io.FDSN. As the name suggests it is aimed to standardize interaction with FDSN networks. In the initial push it contains two major new features:
ChannelPreferences. It is based on something I had in my older C++ code base that I then called a "StationChannelMap". The main purpose of this object is to sort out ambiguous channel codes that are ubiquitous in FDSN data. seed uses channel codes and loc codes together to distinguish sample rate, orientation, and sensor. This provides a way to define a hierarchy of which channels to select when there are many to choose from. This comes up quickly in dealing with Earthscope data as they bundle data they serve through S3 in station bundles. Those can be huge and you almost never want to pull all the extra channels along as baggage. This object provides a general way to do that at the cost of being more than a little complicated.S3SeismicProviderstandardizes the API for accessing FDSN data store as Amazon S3 objects.The initial push is not complete and is not remotely ready to merge. I'm pushing it mainly because I had make a change to a low level class in the
mspasspy.db.normalizemodule calledDataFrameCacheMatcher. I created a special matcher class in the FDSN module that uses a DataFrame index. It turns out the current implementation breaks if you use an input DataFrame that has an index defined. This initial push may break the tests for the normalize module.