Skip to content

fix: include extraction counters in sync progress logs (#4469) - #4471

Open
Jan-Kazlouski-elastic wants to merge 2 commits into
mainfrom
fix/4469-sync-progress-log-noise
Open

fix: include extraction counters in sync progress logs (#4469)#4471
Jan-Kazlouski-elastic wants to merge 2 commits into
mainfrom
fix/4469-sync-progress-log-noise

Conversation

@Jan-Kazlouski-elastic

Copy link
Copy Markdown
Contributor

Part of #4469

Sync progress logs were emitted every display_every documents read, but only reported queued create/update/delete totals. During runs that skip unchanged or filtered documents, this produced bursts of byte-identical log lines that looked like duplication or a stalled sync.

This change enriches progress messages with extracted, filtered, and skipped counters, and aligns counter tracking across all extractor code paths (get_docs, get_docs_incrementally, get_access_control_docs).

Checklists

Pre-Review Checklist

  • this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
  • this PR has a meaningful title
  • this PR links to all relevant github issues that it fixes or partially addresses
  • if there is no GH issue, please create it. Each PR should have a link to an issue
  • this PR has a thorough description
  • Covered the changes with automated tests
  • Tested the changes locally
  • Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
  • For bugfixes: backport safely to all minor branches still receiving patch releases
  • Considered corresponding documentation changes
  • Contributed any configuration settings changes to the configuration reference
  • if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

Changes Requiring Extra Attention

  • Security-related changes (encryption, TLS, SSRF, etc)
  • New external service dependencies added.

Related Pull Requests

  • N/A

Release Note

Sync progress logs now include extracted, filtered, and skipped document counts so agentless connector logs no longer show misleading duplicate progress lines during fast-skipping syncs.

@Jan-Kazlouski-elastic

Copy link
Copy Markdown
Contributor Author

Validated locally with real connector ftest (make ftest NAME=redis DATA_SIZE=small). Two full syncs, 1000 docs — new progress lines are distinct and readable:

Sync progress -- extracted: 101 | filtered: 0 | skipped: 0 | created: 100 | updated: 0 | deleted: 0
Sync progress -- extracted: 201 | filtered: 0 | skipped: 0 | created: 200 | updated: 0 | deleted: 0

Second sync (re-index after small delete): extracted rises, created stays 0, updated climbs — no more byte-identical bursts.

Also ran under Perf8 (PERF8=yes); same log format, report green.

Unit tests: 2445 passed.


@artem-shelkovnikov artem-shelkovnikov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's okay, but I'm a little on the fence - judgement is up to you.

This log like is pretty spammy, so increasing the size of the line 2x increases total logs produced by a sync 2x.

It should not be a big problem, since we don't really count/declare what's the expected log throughput though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants