codec, checker: fix enable-cross-table-merge for keyspace keys (#10992)

[ti-chi-bot]

This is an automated cherry-pick of #10992

What problem does this PR solve?

Issue Number: Close #10991

On NextGen / keyspace clusters (e.g. TiDB Cloud Premium), enable-cross-table-merge=false did not prevent cross-table region merges. codec.Key.TableID() only recognized classic TiDB keys starting with 't', so keyspace-prefixed keys (x|r + keyspace id + table key) always returned table ID 0. Merge checker then treated every pair of regions as the same table and merged empty tables together after split-merge-interval.

What is changed and how does it work?

Add Key.TableIdentity returning a comparable (keyspace, table) identity and
make MetaOrTable strip the API v2 txn-mode keyspace prefix when the remainder
is a TiDB table/meta key, so enable-cross-table-merge=false can distinguish
different tables under a keyspace. Raw-mode keyspace keys are never treated
as table keys. Remove the keyspace-blind Key.TableID, which has no remaining
callers. Classic keys are unchanged.
Consolidate all keyspace key-format knowledge in pkg/codec: the mode-byte
constants and MakeKeyspacePrefix/ParseKeyspacePrefix move from pkg/keyspace
into pkg/codec, and unwrapKeyspace reuses the shared parser.
Add codec unit tests and AllowMerge coverage for classic and keyspace cases.

Check List

Tests

• Unit test
• Integration test (tests/server/cluster/TestCrossTableMergeWithKeyspace: real pd-server, keyspace-encoded regions; also fails on unfixed master, reproducing the bug)
• Manual test (tiup playground: patched pd-server + TiKV v8.5.6 api-version=2 + TiDB v8.5.6 keyspace-name; 20 empty tables kept their own regions for 150s with enable-cross-table-merge=false, then merged 83→4 regions within 90s after flipping it to true)

Related changes

• Need to cherry-pick to the release branch

Release note

Fix enable-cross-table-merge=false being ineffective on keyspace-encoded (NextGen) clusters.

Summary by CodeRabbit

• New Features

  • Added keyspace-prefixed key handling with shared prefix construction/parsing and a new table identity abstraction.
  • Added a configuration setter/toggle to enable or disable cross-table merges.

• Bug Fixes

  • Correctly determines table vs meta keys when a keyspace prefix is present.
  • Improves cross-table merge identity checks by comparing keyspace-aware table identity (not just table ID).

• Tests

  • Expanded unit and integration coverage for keyspace-prefixed decoding and cross-table merge behavior.

[ti-chi-bot]

@JmPotato This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cc73e8d7-46a8-451c-b395-84519a0e4fd4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

[JmPotato]

/hold cancel

[codecov]

Codecov Report

❌ Patch coverage is 97.82609% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 79.09%. Comparing base (b7828f1) to head (b88b6fd).

Additional details and impacted files

@@                    Coverage Diff                     @@
##           release-nextgen-202603   #10995      +/-   ##
==========================================================
+ Coverage                   79.07%   79.09%   +0.02%     
==========================================================
  Files                         531      531              
  Lines                       72298    72323      +25     
==========================================================
+ Hits                        57168    57206      +38     
+ Misses                      11093    11080      -13     
  Partials                     4037     4037              

Flag	Coverage Δ
unittests	79.09% <97.82%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lhy1024, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [lhy1024,rleungx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-07-13 03:02:22.644801412 +0000 UTC m=+596328.680896488: ☑️ agreed by rleungx.
• 2026-07-13 04:58:59.963210547 +0000 UTC m=+603325.999305603: ☑️ agreed by lhy1024.

[JmPotato]

/retest

[JmPotato]

/hold

[ti-chi-bot]

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

[JmPotato]

/test pull-unit-test-next-gen-3

[JmPotato]

/test pull-unit-test-next-gen-3