Skip to content

fix: prevent storing entries with Unix epoch timestamp (fixes #97)#98

Merged
digizeph merged 2 commits intomainfrom
dev/fix-issue-97-epoch-entries
Apr 8, 2026
Merged

fix: prevent storing entries with Unix epoch timestamp (fixes #97)#98
digizeph merged 2 commits intomainfrom
dev/fix-issue-97-epoch-entries

Conversation

@digizeph
Copy link
Copy Markdown
Member

@digizeph digizeph commented Apr 8, 2026

Summary

Fixes #97 by preventing erroneous broker entries with Unix epoch timestamp (1970-01-01) from being stored and returned in search results.

Changes

Bug Fixes

  • Added MIN_VALID_TIMESTAMP constant (1998-01-01) in crawler/common.rs

    • The earliest collector (rrc00) started on 1999-10-01, so 1998 provides a safe buffer
    • Crawlers now filter out entries with timestamps before 1998-01-01 during parsing
  • Added default 30-day time window for queries without timestamp filters

    • Prevents slow full table scans (12.9s → 1.0s, 13x faster)
    • Returns recent data by default instead of starting from 1970
    • Avoids returning phantom 1970 entries

Testing

  • Added unit test test_min_valid_timestamp to verify the threshold
  • All 60 unit tests pass
  • All 62 doc tests pass
  • Clippy checks clean

Production Impact

  • Production database already cleaned: 3 phantom entries with timestamp=0 deleted from deployment (rrc00, rrc01, route-views2)
  • The code changes prevent new erroneous entries from being inserted

Performance Improvement

Query Type Response Time Speedup
No timestamp filter ~12.9 seconds Baseline
30-day window ~1.0 second 13x faster

digizeph added 2 commits April 7, 2026 20:53
- Added MIN_VALID_TIMESTAMP constant (1998-01-01) in crawler/common.rs
- Crawlers now filter out entries with timestamps before 1998-01-01
- Added default 30-day time window for queries without timestamp filters
  - Prevents slow full table scans (12.9s → 1.0s, 13x faster)
  - Returns recent data instead of starting from 1970
- Added unit test for MIN_VALID_TIMESTAMP validation
- Created CLEANUP_PLAN.md for production database cleanup

Fixes #97
@digizeph digizeph merged commit 6cb9041 into main Apr 8, 2026
1 check passed
@digizeph digizeph deleted the dev/fix-issue-97-epoch-entries branch April 8, 2026 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Few erroneous broker items

1 participant