Skip to content

Feat(Platform): add token count to llm-mode and docs#5200

Draft
Guaris wants to merge 2 commits into
mainfrom
llm-token-count
Draft

Feat(Platform): add token count to llm-mode and docs#5200
Guaris wants to merge 2 commits into
mainfrom
llm-token-count

Conversation

@Guaris
Copy link
Copy Markdown
Contributor

@Guaris Guaris commented May 12, 2026

Exposes an estimated token count beside the copy mode and in the text version of the docs.

I got the idea from this article, where it says:

This one is simpler than it sounds but surprisingly high-leverage: surface token counts on your documentation pages. Ideally in both the llms.txt index and on the pages themselves (as metadata or a page header).

This gives agents the information they need to make smart decisions:

“This page is 8K tokens - I can include it fully in context”
“This page is 150K tokens - I should fetch only the relevant section”
“This page exceeds my context window - I’ll use the summary from llms.txt instead
image image

Copilot AI review requested due to automatic review settings May 12, 2026 15:54
@Guaris Guaris requested a review from a team as a code owner May 12, 2026 15:54
@netlify
Copy link
Copy Markdown

netlify Bot commented May 12, 2026

Deploy Preview for kongdeveloper ready!

Name Link
🔨 Latest commit 26679c7
🔍 Latest deploy log https://app.netlify.com/projects/kongdeveloper/deploys/6a034e8179ea760008905cc6
😎 Deploy Preview https://deploy-preview-5200--kongdeveloper.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@Guaris Guaris marked this pull request as draft May 12, 2026 15:55
@Guaris Guaris added the do not merge Issues/ PRs whose changes should not be merged at this time label May 12, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds approximate token counts to LLM-focused UX surfaces (llm dropdown + llms.txt) so agents can budget context when fetching docs.

Changes:

  • Compute per-page token estimates during page data generation and inject tokens into rendered .md frontmatter.
  • Refresh token counts post-write based on rendered Markdown and expose per-group + total tokens to llms.txt.
  • Display token counts beside the LLM dropdown and in llms.txt listings.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
app/_plugins/hooks/site_post_write.rb Recomputes token counts after write; adds group and total token aggregates to llms.txt payload.
app/_plugins/generators/page_data.rb Runs new Data::TokenEstimate processor for pages/docs.
app/_plugins/generators/markdown_pages_generator.rb Injects tokens: into rendered Markdown frontmatter.
app/_plugins/generators/data/token_estimate.rb New estimator to set initial page.data['tokens'].
app/_llms.txt Displays per-page, per-group, and total token counts.
app/_includes/layouts/main.html Passes page.tokens to the LLM dropdown include.
app/_includes/landing_pages/header.md Passes page.tokens to the LLM dropdown include (landing pages).
app/_includes/components/llm_dropdown.html Emits data-tokens for the Vue mount.
app/_assets/javascripts/llm_dropdown.js Passes parsed token count into Vue app as prop.
app/_assets/javascripts/apps/LLMDropdown.vue Renders a token-count label when provided.

# the rendered Markdown files that MarkdownPagesWriter just emitted to disk.
def refresh_token_counts
pages.each do |page|
md_path = File.join(@site.dest, page.url, "#{File.basename(page.url)}.md")
Comment on lines +94 to +98
# fetching the .md can budget context. The count is chars / 4 of the
# rendered output, computed before injection (the line itself adds a
# handful of bytes, negligible at any meaningful page size).
def inject_token_count!(content)
tokens = (content.length / 4.0).round

class LlmsTxtWriter # rubocop:disable Style/Documentation
TEMPLATE_PATH = File.expand_path('../../_llms.txt', __dir__)
CHARS_PER_TOKEN = 4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Issues/ PRs whose changes should not be merged at this time

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants