Skip to content

feat(gta-streaming-five): give each resource its own pgRawStreamer#4072

Open
xalva98 wants to merge 1 commit into
citizenfx:masterfrom
xalva98:pgrawstreamer
Open

feat(gta-streaming-five): give each resource its own pgRawStreamer#4072
xalva98 wants to merge 1 commit into
citizenfx:masterfrom
xalva98:pgrawstreamer

Conversation

@xalva98

@xalva98 xalva98 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Goal of this PR

Raise the raw-streaming file limit so large servers stop hitting
"pgRawStreamer is full".

A pgRawStreamer addresses its entries with a 16-bit file index, so a single
raw streamer fills up at 65535 files. Servers that register a lot of loose
stream assets overflow that shared streamer, after which streaming breaks.

How is this PR achieving the goal

Each resource gets its own pgRawStreamer, allocated lazily on the first
stream file it registers, so the 65535 limit becomes per-resource instead of
shared across the whole server.

  1. Stream files are routed to their owning resource's streamer via the tag
    already available at registration time
    (GetRawStreamerForFileWithTag / RegisterRawStreamingFileWithTag in
    Streaming.cpp). Files that would otherwise land in the shared game (0) or
    ytd (1) streamer are diverted; faux_pack/addons: and untagged files
    keep the existing shared streamers.
  2. The streaming handle already reserves the upper 16 bits for the collection
    index, so per-resource streamers just occupy their own collection slots and
    total capacity scales with the number of streaming resources.
  3. Per-resource streamers take free sm_Collections slots (packfiles already
    claim some by the time resources load, so a free slot is searched for rather
    than assumed). MaxPackfiles is read from the game's packfile
    slot-allocation loop so the bound follows the game across builds.
  4. A reserve of slots is kept for real packfiles, and once the budget is spent
    further resources bucket into streamers already allocated, so packfiles are
    never starved - this should ensure that nothing can cause crashes in the future.
  5. IsRawHandle is now a membership test (per-resource streamers sit at
    non-contiguous collection indices), and the "Loaded pgRawStreamer assets"
    devtools window lists every streamer with its owning resource and fill level.

This PR applies to the following area(s)

FiveM, Streaming

Successfully tested on

Game builds: 3258
Platforms: Windows
Pattern B8 ? ? ? ? 66 3B D0 72 has been found in all game builds up to 3788
Tested locally with a bunch of copied .ytd files
image
image

Checklist

  • Code compiles and has been tested successfully.
  • Code is properly formatted and follows the existing style.
  • Commit message is clear and follows the project conventions.
  • This PR introduces no new compilation warnings.

Fixes issues

Fixes the "pgRawStreamer is full" (65535 raw file) limit on large servers.

PS: This idea was taken from prikolium
Note: You can't have infinite pgRawStreamers, there's an upper limit (which should increase every gamebuild)
For 3258: MaxPackfiles (3928) − reserve (512) ≈ 3,416 ressources, so unless you have 3400 ressources on your server you should be golden.

@github-actions github-actions Bot added triage Needs a preliminary assessment to determine the urgency and required action invalid Requires changes before it's considered valid and can be (re)triaged and removed triage Needs a preliminary assessment to determine the urgency and required action labels Jul 15, 2026

@DaniGP17 DaniGP17 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I've also tested and everything seems to work fine @prikolium-cfx

@prikolium-cfx

Copy link
Copy Markdown
Collaborator

Thank you alva for the contribution and Dani for the review.

@prikolium-cfx

Copy link
Copy Markdown
Collaborator

Btw RedM failed to build with that.

@DaniGP17

Copy link
Copy Markdown
Contributor

Btw RedM failed to build with that.

Sorry for missing that, checking right now

@DaniGP17

DaniGP17 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@xalva98
You can add in code\components\gta-streaming-rdr3\src\Streaming.cpp line 140:

uint32_t* RegisterRawStreamingFileWithTag(uint32_t* fileId, const char* fileName, bool unkTrue, const char* registerAs, bool errorIfFailed, const std::string& tag)
{
	return RegisterRawStreamingFile(fileId, fileName, unkTrue, registerAs, errorIfFailed);
}

STREAMING_EXPORT rage::fiCollection* GetRawStreamerByIndex(uint16_t idx)
{
	return nullptr;
}

STREAMING_EXPORT bool IsRawStreamerCollection(uint16_t idx)
{
	return idx < 2;
}

STREAMING_EXPORT std::vector<RawStreamerInfo> GetRawStreamerInfos()
{
	return {};
}

@xalva98

xalva98 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@xalva98 You can add in code\components\gta-streaming-rdr3\src\Streaming.cpp line 140:

uint32_t* RegisterRawStreamingFileWithTag(uint32_t* fileId, const char* fileName, bool unkTrue, const char* registerAs, bool errorIfFailed, const std::string& tag)
{
	return RegisterRawStreamingFile(fileId, fileName, unkTrue, registerAs, errorIfFailed);
}

STREAMING_EXPORT rage::fiCollection* GetRawStreamerByIndex(uint16_t idx)
{
	return nullptr;
}

STREAMING_EXPORT bool IsRawStreamerCollection(uint16_t idx)
{
	return idx < 2;
}

STREAMING_EXPORT std::vector<RawStreamerInfo> GetRawStreamerInfos()
{
	return {};
}

should be fixed, thank you 👍

@xalva98

xalva98 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Sadly, after putting in some more testing each ressource having a streamer is most likely not possible.
On b3258 after loading all dlcs you roughly have ~150 slots left which is not really enough cause some servers have way more ressources than this.

So I changed the approach to just increase to up to 32 (could be reduced even further technically). The streamers are shared, if they are about to overflow a new "bucket" is created so to speak.
That removed the ERR_FIL_PACK_1 error for me locally. This should still be an upgrade over the current behaviour.

The high-volume raw asset types (ytd, ybn, yft, ydd, ydr, ymap) each get
their own pgRawStreamer, allocated lazily from free sm_Collections slots
and rolled over to a fresh streamer once one fills, so the 65535 entry
limit no longer applies across the whole server. The stock streamers are
untouched: base game content stays in slot 0 and the ytd/user mod
streamers (1, 2) keep their original routing, so only tagged resource
content is diverted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

invalid Requires changes before it's considered valid and can be (re)triaged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants