Skip to content

Commit 3f11cb0

Browse files
fix(service): Lower the body share of one client to 16 MiB
With 32 MiB, 4 IPv4 addresses or 4 /64s could fill the 128 MiB budget of runs without a token. With 16 MiB it takes 8. 16 MiB is still one body of the largest size, or about 25 of the largest model requests. The README said that a client with a /48 holds the budget "for 30 s at a time". A body gets HTTP 408 after 30 s, but the client can start the next one at once, so the hold does not stop.
1 parent 4bd94e5 commit 3f11cb0

3 files changed

Lines changed: 13 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -698,7 +698,7 @@ To see how well the cache works:
698698
run without the token. A control API body over 1 MiB gets HTTP 413
699699
before the service parses it.
700700
- The requests of all runs without the token can hold 128 MiB of bodies
701-
at one time, and the runs of one client 32 MiB. The client of a run is
701+
at one time, and the runs of one client 16 MiB. The client of a run is
702702
the client that started it, grouped as for the limit of runs. A request
703703
holds its body until it finishes, also while its key waits for the key
704704
worker. A request whose body does not fit gets HTTP 503 with
@@ -725,7 +725,8 @@ To see how well the cache works:
725725
- A gap: a client with an IPv6 /48 has 65,536 /64s, so it can open runs
726726
until the service has 100 runs without a token. Then other clients get
727727
HTTP 429 until its runs end. Its runs can also hold all of the budget of
728-
bodies, for 30 s at a time.
728+
bodies. A body gets HTTP 408 after 30 s, but the client can start the
729+
next one at once, so it holds the budget for as long as it sends.
729730
- GCS deletes each recording 30 days after it was written, also when runs
730731
still replay it. Then `auto` mode records it again, and `replay` mode
731732
fails the request with HTTP 412.

‎packages/service/src/service.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -173,10 +173,10 @@ const PUBLIC_BODY_BUDGET_BYTES = 128 * 1024 * 1024;
173173
/**
174174
* The part of `PUBLIC_BODY_BUDGET_BYTES` that the runs of one client
175175
* (`clientGroup()` of the client that started the run) can hold, so that
176-
* one client cannot take all of it. It is 2 bodies of 16 MiB, or about 50
177-
* of the largest model requests.
176+
* one client cannot take all of it. It takes 8 clients to fill the budget.
177+
* It is 1 body of 16 MiB, or about 25 of the largest model requests.
178178
*/
179-
const PUBLIC_BODY_CLIENT_BYTES = 32 * 1024 * 1024;
179+
const PUBLIC_BODY_CLIENT_BYTES = 16 * 1024 * 1024;
180180
/**
181181
* A body of a run without a token that does not come in this time gets
182182
* HTTP 408, and gives its bytes back. So a client that sends a byte now

‎packages/service/tests/service.test.ts‎

Lines changed: 7 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -806,11 +806,11 @@ describe("limits of the service", () => {
806806
},
807807
],
808808
});
809-
// 7 bodies of 4 MiB fit in the 32 MiB that the runs of one client can
809+
// 3 bodies of 4 MiB fit in the 16 MiB that the runs of one client can
810810
// hold.
811811
const prompt = "a".repeat(4 * 1024 * 1024);
812812
const sent = Promise.allSettled(
813-
Array.from({ length: 7 }, () => send(run.url, { prompt })),
813+
Array.from({ length: 3 }, () => send(run.url, { prompt })),
814814
);
815815
const waits = await caWaitsUntil(sent);
816816
// The keys are made in a worker, so the service answers at once.
@@ -861,7 +861,7 @@ describe("limits of the service", () => {
861861
const results = await Promise.all(
862862
Array.from({ length: 12 }, () => send(reader.url, { prompt })),
863863
);
864-
// The runs of one client can hold 32 MiB, 2 of these bodies. The bytes
864+
// The runs of one client can hold 16 MiB, 1 of these bodies. The bytes
865865
// count when they come, so parts of more bodies can fill it.
866866
const busy = results.filter((result) => result.status === 503);
867867
expect(busy.length).toBeGreaterThanOrEqual(4);
@@ -872,10 +872,7 @@ describe("limits of the service", () => {
872872
}
873873
// The run goes on, and the bodies gave their bytes back.
874874
await expect(reader.stats()).resolves.toBeDefined();
875-
const again = await Promise.all(
876-
Array.from({ length: 2 }, () => send(reader.url, { prompt })),
877-
);
878-
expect(again.map((result) => result.status)).toEqual([412, 412]);
875+
expect((await send(reader.url, { prompt })).status).toBe(412);
879876

880877
// A client that declares large bodies and then stops holds only the
881878
// bytes that it sent.
@@ -942,12 +939,12 @@ describe("limits of the service", () => {
942939
await realWait(500);
943940
const small = 700 * 1024;
944941
expect(await answer(local.url, other, small)).toBe(412);
945-
// The runs of the attacker can hold only 32 MiB, 2 of the bodies.
942+
// The runs of the attacker can hold only 16 MiB, 1 of the bodies.
946943
await until(
947-
() => stalls.filter(({ status }) => status() === 503).length >= 6,
944+
() => stalls.filter(({ status }) => status() === 503).length >= 7,
948945
);
949946
const holders = stalls.filter(({ status }) => status() === 0);
950-
expect(holders.length).toBeLessThanOrEqual(2);
947+
expect(holders.length).toBeLessThanOrEqual(1);
951948

952949
// A body that does not come in 30 s gets HTTP 408, and its tunnel
953950
// closes. Then the attacker has its part again.

0 commit comments

Comments
 (0)