Add EUVD mirror pipeline #1
Conversation
Signed-off-by: Samk <[email protected]>
Signed-off-by: Samk <[email protected]>
| if path.exists(): | ||
| self.log(f"skip existing file: {path}") | ||
| return |
There was a problem hiding this comment.
Why do we skip updating the existing file?
There was a problem hiding this comment.
We skip existing files to make the sync idempotent and resumable. If the script is interrupted or re-run, it won't overwrite data that was already successfully saved. Since the data is fetched using the dateUpdated param and updates are forward-only, which makes skipping previously written files safe.
sync_catalog.py
Outdated
| if not isinstance(data, dict): | ||
| return {} |
There was a problem hiding this comment.
Do we have any examples of this invalid output from the endpoint?
There was a problem hiding this comment.
This is a safety check to handle unexpected API responses. The EUVD API could potentially return non-dict responses (e.g empty response, error strings, or malformed objects). Just a safety measure for now — don’t have a concrete example yet :)
Signed-off-by: Samk <[email protected]>
|
I have made the changes as per suggestions. If any other changes are required, do let me know. Looking forward! |
2 participants
EUVD Mirror Pipeline
The script has two modes:
As of now the PR focuses on the code, once reviewed and approved I can add the locally collected backfill data.