Stable backport of #1049 (main: 3600231d). Change content is byte-identical
to the main twin.
S3StorageAdapter built its S3Client with no requestHandler timeouts, and the
SDK's defaults wait indefinitely. When a connection is dropped silently (no
FIN/RST — NAT/LB idle reaps, transient faults), the in-flight request hangs
forever and every subsequent storage operation queues behind it process-wide;
only a restart recovers. _retryOperation never ran because the promise it
wraps never settled.
Configure connectionTimeout (120s) and socketTimeout (60s) on the request
handler, overridable via STORAGE_S3_CONNECTION_TIMEOUT /
STORAGE_S3_SOCKET_TIMEOUT, and add TimeoutError to the retryable list so the
existing backoff engages.
socketTimeout rather than requestTimeout: the latter is a total-duration cap
that would kill legitimate large uploads and only warns without
throwOnRequestTimeout. Both values are deliberately generous — connectionTimeout
covers time queuing for a socket from the agent pool (maxSockets 50), so a
short value expires while merely waiting in line and breaks reads.
Reported against v3.45.16 on Cloudflare R2: wedged roughly every 40 minutes
with serial uploads, every 15-20 with 4 parallel uploaders.
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
Paul Nothaft
88fa3c5297