Keyspace related HTTP API may encounter error or performance issue when there are too many keyspaces

[MyonKeminta]

Bug Report

What did you do?

• Create 100K keyspaces by posting to /pd/api/v2/keyspaces HTTP API
• Get keyspace meta by getting from /pd/api/v2/keyspaces HTTP API
• Iterate all keyspaces in PD's code

What did you expect to see?

Everything works fine

What did you see instead?

• Create 100K keyspaces by posting to /pd/api/v2/keyspaces HTTP API
  • With 10 concurrency in the client, the performace slows down as the progress proceeds, and it takes about 7-8 hours to finish creating all the keyspaces.
    • The reason might be the UpdateKeyspaceForGroup step.
• Get keyspace meta by getting from /pd/api/v2/keyspaces HTTP API

  • $ curl 192.168.178.11:3250/pd/api/v2/keyspaces
    "rpc error: code = ResourceExhausted desc = trying to send message larger than max (11983608 vs. 2097152)"
    

• Iterate all keyspaces in PD's code
  • It takes about 90s to finish iterating 100K keyspaces.

What version of PD are you using (pd-server -V)?

master, with unrelated modifications in GCStateManager (see: #10498)

[rleungx]

The actual number of keyspaces could be much larger than 100K.