Skip to content

GPS time correlation on log file parsing#71

Merged
LelsersLasers merged 21 commits into
masterfrom
millan/parse_timestamp_correlation
May 27, 2026
Merged

GPS time correlation on log file parsing#71
LelsersLasers merged 21 commits into
masterfrom
millan/parse_timestamp_correlation

Conversation

@LelsersLasers
Copy link
Copy Markdown
Member

  • After splitting into chunks, look for all GPS messages
  • Linear regression to find correlation between the DAQ timestamp and the time contained in the GPS message
  • Added column to the parsed CSV with the "real" (GPS correlated time)
  • Log files with valid correlated times are named out_###_TimeOfFirstMessage

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds GPS-based time correlation to DAQ log parsing so parsed CSV outputs can include a “real” timestamp derived from GPS time messages, and correlated chunks can be named with the time of the first message.

Changes:

  • Introduces a daq_log_parse::correlate module that extracts gps_time messages per chunk and fits a linear regression between DAQ ms and Unix ms.
  • Updates the log parsing UI flow to run correlation after chunking and pass correlated chunks into CSV generation.
  • Extends CSV output to include a correlated time column and optionally suffixes output filenames with the first correlated timestamp.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
src/ui/log_parser.rs Runs chunk time-correlation before writing CSV tables.
src/daq_log_parse/table.rs Accepts correlated chunks, adds correlated time column, and names output files with correlated first time when available.
src/daq_log_parse/mod.rs Exposes the new correlate module.
src/daq_log_parse/correlate.rs Implements GPS timestamp extraction and linear regression correlation per chunk.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/daq_log_parse/correlate.rs Outdated
Comment thread src/daq_log_parse/correlate.rs
Comment thread src/daq_log_parse/correlate.rs Outdated
Comment thread src/daq_log_parse/table.rs Outdated
Comment thread src/daq_log_parse/table.rs
@LelsersLasers
Copy link
Copy Markdown
Member Author

Currently gives weird results:
GPS correlation successful: slope=1.565244263, intercept_ms=1778437407030.705, rms_error_ms=771.89, points=144

DAQ ticks are 1 ms so the slope should be exactly 1 and the RMS is very high, but from debug printing the algorithm is working correctly, it is the data itself that is noisy

Noisy/inccorect GPS:
Figure_1
Figure_2

@LelsersLasers
Copy link
Copy Markdown
Member Author

Okay updates: so it turns out the clock rate was set wrong on DAQ + some of the other boards (ex: Dash) so that's why the slope was not 1.... Seems to work fine....

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

@LelsersLasers LelsersLasers marked this pull request as ready for review May 27, 2026 19:59
@LelsersLasers LelsersLasers force-pushed the millan/parse_timestamp_correlation branch from 2530a76 to efbc62c Compare May 27, 2026 20:02
@LelsersLasers LelsersLasers requested a review from irvingywang May 27, 2026 20:02
@LelsersLasers LelsersLasers merged commit bf66d19 into master May 27, 2026
1 check passed
@LelsersLasers LelsersLasers deleted the millan/parse_timestamp_correlation branch May 27, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants