feat(job): implement Redis Set storage for SyncPeersJob to avoid big key issues #4654#4668
feat(job): implement Redis Set storage for SyncPeersJob to avoid big key issues #4654#4668liyi1013 wants to merge 1 commit into
Conversation
…is_bigkey_s Signed-off-by: LiYi <liyi1013@foxmail.com> Solve redis big key problem
|
|
||
| // save results to redis set | ||
| for _, val := range data { | ||
| if err := t.rdb.SAdd(ctx, key, val).Err(); err != nil { |
There was a problem hiding this comment.
When the number of peers becomes excessively large, wouldn't this approach actually place a greater strain on redis?
With 100,000 Peers, this implies that the redis.sAdd() method must be invoked 100,000 times. Although this eliminates the issue of a single Key holding an excessively large volume of data, it introduces a new problem: executing redis.sAdd() 100,000 times incurs significant overhead—potentially resulting in a situation even worse than the original one.
Regardless of how redis.SAdd() is invoked, it alters only the write method, not the resulting data structure. Whether you write 100,000 entries in a single operation or perform 100,000 separate writes of one entry each, the outcome remains the same: a single Set containing 100,000 members.
In a separate PR(#4654), the implementation utilizes sharding to limit the number of Peers associated with any single Key. This method resolves the issue of excessive data storage under a single Key without incurring the overhead of repeated redis.sAdd() calls. I believe this implementation approach is superior; what are your thoughts?
To summarize, I think the correct approach is to attempt to shard the data using different keys, rather than writing data to the same key in small, frequent increments.
|
This PR is stale because it has been open 60 days with no activity. |
|
This PR was closed because it has been stalled for 7 days with no activity. |
5 participants
Description
The Dragonfly syncPeer job has the potential to cause a Redis Big Key Problem （value > 100k）when the number of peer nodes is excessive, which poses a risk of degrading Redis performance.
This PR updates the logic to store SyncPeersJob results in a Redis set instead of storing them as a single string.
Related Issue
#4379
Motivation and Context
Instead of serializing the entire result into a large JSON string (which could lead to Redis big key issues), this PR serializes individual host objects and stores them into a Redis Set. The task then returns the key of this Set. During retrieval, the results are fetched directly from Redis via this key.
Screenshots (if appropriate)
Types of changes
Checklist