* fix(upload): stop buffering a chunk body before anything checks its size
The chunk route drained the whole request into an array and concatenated it
before calling uploadChunk — which is where every check lives. So a 300MB body
sent against an unknown upload id was read in full, cost ~300MB of heap, and
was only then answered with an error. The per-file cap was real but applied
after the damage, and nothing looked at Content-Length at all.
uploadChunk now takes the request stream itself and consumes nothing until the
upload id, the chunk index and the declared Content-Length have all been
checked against the remaining allowance. A body that clears those is streamed
straight to the chunk file under a hard byte cap, so a sender that lies about
its length — or sends none, the Transfer-Encoding: chunked case — is cut off
at the allowance instead of being read to the end.
A Buffer is still accepted, so the existing callers and chunkedUploadSizeCap
tests are untouched. Worth noting the old code silently did nothing when
handed a stream: fs.promises.writeFile accepts an async iterable, so the body
was written while `chunkData.length` was undefined and the cap comparison was
NaN > max, i.e. always false.
Reachable only for an authenticated admin holding photos.upload on an event
they own, and only once the CSRF gate lets application/octet-stream through.
Relates to issue 1403
* fix(upload): harden the chunk stream against abort, retry and cap failures
Three failure paths the streaming rewrite introduced, all found by external
review. None existed in the buffered version: the async iterator it replaced
rejected a dead request on its own, and never opened the chunk file until it
already held the whole body.
- An already-destroyed request hung the call forever. If the client hangs up
while auth and ownership are awaiting the database, pipe() emits neither
`end` nor `error`, so the promise never settled and the write descriptor
stayed open. Checked up front now, alongside `aborted` and a `close` without
`readableEnded` for a body cut short mid-flight.
- A failed re-send destroyed the chunk it was replacing. createWriteStream
truncates on open, so re-sending an index and then failing left
receivedChunks and chunkSizes still claiming the old copy: status reported
100% and completeUpload died on ENOENT. Chunks are staged through a sibling
.part file and renamed only on success.
- Tripping the cap stopped the 413 from reaching the client. `source` is the
IncomingMessage, so destroying it destroyed the socket under the response
and the client saw a connection reset instead of the size-limit JSON. The
read is paused instead, which is all the cap needs.
Relates to issue 1403
* fix(upload): isolate chunk staging per attempt and close before cleanup
Two races found by a second review round, both in the staging logic added by
the previous commit.
- Two in-flight sends of the same chunk index shared one `.part` path, so
whichever renamed first published bytes the other had already truncated. An
acknowledged 10-byte chunk could end up 2 bytes. The staging suffix is now
per-attempt rather than per-index.
- Unlinking the partial file raced the write stream's pending open(). destroy()
does not await it, so the unlink failed with ENOENT and the open then
recreated the `.part` file after cleanup had supposedly finished — reported
reproducible in 121 of 300 immediately-failing streams. Cleanup now waits for
the stream to close.
Relates to issue 1403
* fix(upload): revalidate the per-file cap before publishing a chunk
`allowance` is computed before the body arrives, so a chunk that completed
while this one was still streaming was not counted in it. Two overlapping
0.75MB chunks under a 1MB cap were therefore both accepted, leaving 1.5MB on
disk; enough concurrent streams could go well past the cap before anyone asked
to complete the upload.
The buffered version got this right for free, because it only ever checked
after reading the whole body. The streaming version keeps its pre-read check —
that is what makes a too-large request cheap — and asks again with the current
aggregate before renaming the staging file into place.
Relates to issue 1403
* fix(upload): answer client-caused chunk states with their own status code
An unknown, finished or expired upload id, and a complete call with chunks
missing, were plain Errors with no statusCode, so both routes fell through to
the blanket 500 and logged at error level. All four are the client's mistake:
they read as a backend fault in monitoring and invite a retry that can never
succeed.
They now carry 404, 409, 410 and 400 respectively, and both routes pass a
tagged status through instead of matching on the two they happened to know
about. Only genuinely unexpected errors reach the 500 and the error log.
No client is affected: uploadLargeFile, the only caller of this endpoint
family, still has no callers of its own.
Relates to issue 1403
* fix(upload): retire the connection after an early refusal, clean up a failed publish
Two more from review.
Refusing a body before reading it is the point of the streaming cap, but the
unread bytes are still in flight on a connection the response advertises as
keep-alive. Node does not drain them, so the NEXT request on that socket hangs
until it times out — reproducible with an 8MB body against a 1MB cap. The
error response now sets Connection: close whenever the request was not read to
the end.
A failed rename — ENOSPC, a vanished directory — left the fully written
staging file behind. Staging names are per-attempt, so a client that retries
instead of aborting accumulates one per try until the upload expires. The
partial is removed on that path too.
Relates to issue 1403
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
Paul Nothaft
4622478e44