Skip to content

collect: filter logs by time range - #496

Open
xhebox wants to merge 1 commit into
pingcap:masterfrom
xhebox:fix-collect-time-filter
Open

collect: filter logs by time range#496
xhebox wants to merge 1 commit into
pingcap:masterfrom
xhebox:fix-collect-time-filter

Conversation

@xhebox

@xhebox xhebox commented Jun 15, 2026

Copy link
Copy Markdown

What problem does this PR solve?

close #436

tiup diag collect passes -f/-t time filters, but logs collected by the remote scraper were still copied as full files. For large log files, this makes collection slower and returns data outside the requested time range.

What is changed and how it works?

  • Filter parsable standard and slow logs by the requested time range on the remote host before downloading them.
  • Store filtered logs in an internal scraper temporary directory and return the filtered file path in scraper output.
  • Preserve the original log file path in local collection results while downloading the filtered file content.
  • Keep unparseable logs unchanged instead of dropping them.
  • Add tests for scraper time-range filtering and log iterator start-boundary inclusion.

Check List

Tests

  • Unit test

Ran:

  • go test ./...

Code changes

  • No exported function/method change
  • No exported variable/fields change
  • No interface methods change
  • No persistent data change

Side effects

  • No breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch

@pingcap-cla-assistant

pingcap-cla-assistant Bot commented Jun 15, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codecov-commenter

codecov-commenter commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 47.28682% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 9.98%. Comparing base (82f9593) to head (c8907ce).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
scraper/log.go 52.17% 32 Missing and 23 partials ⚠️
collector/log.go 0.00% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##           master    #496      +/-   ##
=========================================
- Coverage   10.70%   9.98%   -0.72%     
=========================================
  Files         124     124              
  Lines       12591   10569    -2022     
=========================================
- Hits         1347    1055     -292     
+ Misses      11073    9314    -1759     
- Partials      171     200      +29     
Flag Coverage Δ
unittests 9.98% <47.29%> (-0.72%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@xhebox
xhebox force-pushed the fix-collect-time-filter branch from d9c5d9b to c8907ce Compare June 15, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support collect logs by precise time range

2 participants