Feat(Platform): add token count to llm-mode and docs#5200
Draft
Guaris wants to merge 2 commits into
Draft
Conversation
✅ Deploy Preview for kongdeveloper ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds approximate token counts to LLM-focused UX surfaces (llm dropdown + llms.txt) so agents can budget context when fetching docs.
Changes:
- Compute per-page token estimates during page data generation and inject tokens into rendered
.mdfrontmatter. - Refresh token counts post-write based on rendered Markdown and expose per-group + total tokens to
llms.txt. - Display token counts beside the LLM dropdown and in
llms.txtlistings.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| app/_plugins/hooks/site_post_write.rb | Recomputes token counts after write; adds group and total token aggregates to llms.txt payload. |
| app/_plugins/generators/page_data.rb | Runs new Data::TokenEstimate processor for pages/docs. |
| app/_plugins/generators/markdown_pages_generator.rb | Injects tokens: into rendered Markdown frontmatter. |
| app/_plugins/generators/data/token_estimate.rb | New estimator to set initial page.data['tokens']. |
| app/_llms.txt | Displays per-page, per-group, and total token counts. |
| app/_includes/layouts/main.html | Passes page.tokens to the LLM dropdown include. |
| app/_includes/landing_pages/header.md | Passes page.tokens to the LLM dropdown include (landing pages). |
| app/_includes/components/llm_dropdown.html | Emits data-tokens for the Vue mount. |
| app/_assets/javascripts/llm_dropdown.js | Passes parsed token count into Vue app as prop. |
| app/_assets/javascripts/apps/LLMDropdown.vue | Renders a token-count label when provided. |
| # the rendered Markdown files that MarkdownPagesWriter just emitted to disk. | ||
| def refresh_token_counts | ||
| pages.each do |page| | ||
| md_path = File.join(@site.dest, page.url, "#{File.basename(page.url)}.md") |
Comment on lines
+94
to
+98
| # fetching the .md can budget context. The count is chars / 4 of the | ||
| # rendered output, computed before injection (the line itself adds a | ||
| # handful of bytes, negligible at any meaningful page size). | ||
| def inject_token_count!(content) | ||
| tokens = (content.length / 4.0).round |
|
|
||
| class LlmsTxtWriter # rubocop:disable Style/Documentation | ||
| TEMPLATE_PATH = File.expand_path('../../_llms.txt', __dir__) | ||
| CHARS_PER_TOKEN = 4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exposes an estimated token count beside the copy mode and in the text version of the docs.
I got the idea from this article, where it says: