Skip to content

fix(populate): split into separate populate per document if resulting $in has more than 50k elements to reduce risk of BSON errors#16333

Merged
vkarpov15 merged 1 commit into
masterfrom
vkarpov15/gh-5890-2
Jun 17, 2026
Merged

fix(populate): split into separate populate per document if resulting $in has more than 50k elements to reduce risk of BSON errors#16333
vkarpov15 merged 1 commit into
masterfrom
vkarpov15/gh-5890-2

Conversation

@vkarpov15

Copy link
Copy Markdown
Collaborator

Fix #5890

Summary

The crux of the issue in #5890 is that populate() can lead to BSON size limit errors in the case of multiple documents. About 800k ObjectIds fit in the BSON size limit - however, populate() on multiple documents can try to stuff more ids than that into a query. For example, if we have a relationship User -> TVShow -> View, the populate() call for View can include the views from multiple TVShow documents, and lead to BSON overflow.

The practical solution I came up with is to switch to perDocumentLimit behavior (one query per document vs per model) if the resulting query has more than 50k elements. That limit is intentionally not very rigorous, because calculating the BSON serialized size for a large array can be performance intensive; but the 50k limit is enough to avoid BSON overflows for ObjectIds, uuids, dates, and numbers.

The key insight behind splitting by document is that documents themselves must fit within the BSON size limit, so an individual document's populate() cannot trigger a BSON size limit presuming that document was already saved in MongoDB (of course possible to create a document in memory with an ObjectId array of length 1M and try to populate it but not very realistic because you could never save this document).

Examples

… $in has more than 50k elements to reduce risk of BSON errors

Fix #5890
@vkarpov15 vkarpov15 added this to the 9.7.1 milestone Jun 12, 2026

@hasezoey hasezoey left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vkarpov15 vkarpov15 merged commit 1e81fcc into master Jun 17, 2026
50 checks passed
@hasezoey hasezoey deleted the vkarpov15/gh-5890-2 branch June 17, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Running into 16MB BSON limit with Mongoose Populate

2 participants