feat: update robots txt - #1698
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The critical crawler configuration issue must be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates crawler rules to exclude legacy documentation while preserving selected v3.7 API pages.
Changes:
- Adds version- and page-specific crawl exclusions.
- Allows selected v3.7 API documentation.
File summaries
| File | Description | Review |
|---|---|---|
public/robots.txt |
Adds documentation crawler directives. | Critical: The new User-agent hostname does not match search crawlers, so the added rules are ignored. Move them into the wildcard group or target valid crawler tokens. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
bstopyra
approved these changes
Sep 10, 2026
|
🎉 This PR is included in version 2.105.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release requirements
PR information
What does this PR do? (optional)
Type of change
Deploy preview
The deploy preview link will appear in the checks section below once the build is complete.
Versioning
Pull request titles should comply with the Conventional Commits specification. This is required to update the project's version.