fix(populate): split into separate populate per document if resulting $in has more than 50k elements to reduce risk of BSON errors#16333
Merged
Conversation
… $in has more than 50k elements to reduce risk of BSON errors Fix #5890
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #5890
Summary
The crux of the issue in #5890 is that
populate()can lead to BSON size limit errors in the case of multiple documents. About 800k ObjectIds fit in the BSON size limit - however,populate()on multiple documents can try to stuff more ids than that into a query. For example, if we have a relationshipUser->TVShow->View, thepopulate()call forViewcan include theviewsfrom multipleTVShowdocuments, and lead to BSON overflow.The practical solution I came up with is to switch to
perDocumentLimitbehavior (one query per document vs per model) if the resulting query has more than 50k elements. That limit is intentionally not very rigorous, because calculating the BSON serialized size for a large array can be performance intensive; but the 50k limit is enough to avoid BSON overflows for ObjectIds, uuids, dates, and numbers.The key insight behind splitting by document is that documents themselves must fit within the BSON size limit, so an individual document's
populate()cannot trigger a BSON size limit presuming that document was already saved in MongoDB (of course possible to create a document in memory with an ObjectId array of length 1M and try to populate it but not very realistic because you could never save this document).Examples