File uploads

What it is and why it exists

A file upload is the process by which a client gives a server some bytes and descriptive metadata. HTTP transports the message, but the application must decide who may upload, what it accepts, how it stores the result, when the result becomes readable, and which intermediate state to clean after failure. You face these decisions anywhere an API accepts avatars, attachments, data imports, or media.

An upload endpoint differs from an ordinary JSON endpoint because its payload can be large and its bytes may enter a parser, transcoder, malware scanner, or browser. An attacker can spoof extensions and media types or exploit paths, compression ratios, parser flaws, and storage quotas. The security boundary therefore cannot end when form parsing succeeds.

A robust upload flow separates “received” from “available.” New bytes first enter an upload quarantine ; the database record becomes available only after identity, size, type, content, and business rules pass. If scanning times out or processing fails, the file stays invisible and the system can retry or clean it instead of treating an unknown result as success.

Uploads commonly use one of two data paths. A smaller form file can pass through the application server, while a large file can use a short-lived capability to go directly to object storage. Both paths must share the same control plane: the server creates an upload intent, binds ownership and constraints, checks the observed object, and decides whether to publish it.

How it works

Two meanings of multipart

multipart/form-data is an HTTP form encoding. The boundary parameter in the request’s Content-Type separates the body into parts. Each part uses Content-Disposition to carry a field name, and a file part commonly carries a client filename and Content-Type. One request can contain text fields and one or more files.

An object store’s multipart upload instead divides one object into independently uploaded parts and assembles them during a completion operation. It supports retries, concurrent transfer, and recovery for large objects; it is not another way to parse multipart/form-data. An API design should say whether “part” means a form part or an object part.

From intent to publication

The control plane first authenticates the caller, checks resource-level authorization and quotas, and creates a pending upload record. The record holds a server-generated object key , expected size, allowed type, optional checksum, expiration, and owner. The original filename is only length-bounded display metadata.

The data plane then receives the bytes. A multipart parser produces a file stream on the application-server path; an object store receives a PUT or parts on the direct path. Both must count actual bytes and reject excess data instead of trusting the size declared before the request began.

After receipt, the server reads observed properties from a temporary file or object that it controls. It verifies the key, length, and checksum, identifies the type, performs format-specific decoding or rewriting, and scans according to policy. Only after every check succeeds does it move the record from pending to available.

A trusted service must own the state transition; a client must not submit status: "available" for itself. The completion request needs an ownership check, and a conditional update or transaction must ensure that only the expected pending record can be published. A repeated completion should return the stored result or a clear conflict, not create another file record.

Limits at every layer

The outer proxy or gateway should bound request size and read time. The application parser must also bound file count, field count, field length, and per-file size. Storage adds tenant quotas and lifecycle rules, while processors bound decompressed size, image dimensions, page count, or media duration. Each layer protects a different resource, so one Content-Length check cannot replace the others.

Content-Length can be absent, and for multipart input it describes the whole message rather than one file. Even when its value is acceptable, the reader must count actual bytes and, on overflow, abort the source, close the destination, and remove the partial file. Archive handling also needs limits on entry count, nesting, and total decompressed output.

Names, types, and content

A client filename is suitable for display, not for a path or object key. The server should generate a new, unpredictable, non-reused key and keep ownership in a database or protected metadata. On download, it can place a separately encoded display name in Content-Disposition without feeding the original name back into path resolution.

The client’s Content-Type is only useful for cheaply rejecting an obvious mistake. A file signature says more about the bytes than an extension, but a correct prefix still does not prove that the complete structure is valid or harmless. If the product accepts PNG, a maintained PNG decoder should fully parse and, where needed, re-encode it. PDF and archive support require controls specific to those formats.

Detection failure should reject the upload or leave it quarantined. Treating an unavailable scanner as “clean” converts a dependency outage into a security bypass. A product may reject synchronously, retry asynchronously, or send the item for manual review, but its state and user visibility must be explicit.

Streams and backpressure

Streaming lets an application avoid collecting the complete file in one Buffer. A source emits data, transforms count and hash each chunk, and a destination writes to quarantine; pipeline() propagates errors and waits for every stage. Memory use still depends on buffers and concurrency at each layer, so “uses streams” does not automatically mean “capacity bounded.”

Backpressure pauses reads when writes cannot keep up. Code that listens for data and pushes chunks into an unbounded array, or ignores a writable stream’s pressure signal, breaks that feedback chain. Concurrent upload count needs a separate bound as well, because small buffers per stream can still exhaust aggregate memory and file descriptors.

Direct object-store uploads

A presigned URL gives a client time-limited storage permission for a particular method and object key. It saves the application server from relaying the bytes, but it does not delegate authorization, quotas, validation, or publication to the storage service. The application still authenticates the user and generates a new key under a quarantine prefix before signing.

After the client reports success, the application must not trust the submitted size, type, or checksum. A completion endpoint reads the properties observed by storage and, when policy requires, verifies a checksum covered by the signature. A presigned capability can be reused before it expires, and a reused key can overwrite an object, so both the key and the state transition need replay protection.

Examples

A first byte-level gate

This pure Node.js example maps a declared type to an expected signature and also checks for empty or oversized input. The result is deliberately named headerGate: passing it means only that the prefix matches the admission policy, not that a complete PNG was decoded or scanned.

Both samples claim the name avatar.png, but the name does not affect the decision. The second sample fails the cheap gate because its bytes do not match the PNG signature. The first is still only a fixture; production code must give it to a real PNG decoder before accepting it.

Streaming into quarantine

The next example uses pipeline() to write two input chunks into a staging file with mode 0600. A transform counts actual bytes and computes SHA-256, failing above 32 bytes. It renames the file in the same directory only after a successful write and removes a partial file on failure.

import { createHash } from 'node:crypto';
import { createWriteStream } from 'node:fs';
import { mkdir, rename, rm } from 'node:fs/promises';
import { Readable, Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';
class Meter extends Transform {
  constructor(limit) {
    super();
    this.limit = limit;
    this.bytes = 0;
    this.hash = createHash('sha256');
  }
  _transform(chunk, encoding, callback) {
    this.bytes += chunk.length;
    if (this.bytes > this.limit) return callback(new Error('upload too large'));
    this.hash.update(chunk);
    callback(null, chunk);
  }
}
async function store(source, root, objectId) {
  const staged = `${root}/${objectId}.part`;
  const finalPath = `${root}/${objectId}`;
  const meter = new Meter(32);
  try {
    await pipeline(source, meter, createWriteStream(staged, { flags: 'wx', mode: 0o600 }));
    await rename(staged, finalPath);
    return { finalPath, bytes: meter.bytes, sha256: meter.hash.digest('hex') };
  } catch (error) {
    await rm(staged, { force: true });
    throw error;
  }
}
const root = '/tmp/codewiki-upload-example';
await rm(root, { recursive: true, force: true });
await mkdir(root, { recursive: true });
const source = Readable.from([Buffer.from('order='), Buffer.from('CW-42\n')]);
const result = await store(source, root, 'upload-7f3a');
console.log(`stored=${result.finalPath} bytes=${result.bytes}`);
console.log(`sha256=${result.sha256}`);
await rm(root, { recursive: true, force: true });

stored=/tmp/codewiki-upload-example/upload-7f3a bytes=12
sha256=fe2c1abd0da3359bd47aa3d712d29f040a30252a8ce116879b035d47b639de47

The fixed directory and object ID exist only to make the output reproducible. A real service generates a new key for each authorized upload and keeps the staging directory out of public reach. This rename() commits only a local staged file; parsing, malware scanning, and database publication remain later gates.

Checking a direct upload

A direct-upload completion endpoint compares the constraints saved with the upload intent against properties observed by storage. This pure function also checks the caller and state. A mismatched object becomes rejected, a failed scan stays quarantined, and only a complete match becomes available.

function finalizeUpload(record, observed, actorId) {
  if (record.ownerId !== actorId) throw new Error('forbidden');
  if (record.status !== 'pending') throw new Error('invalid upload state');

  const matches = observed.key === record.key
    && observed.size === record.expectedSize
    && observed.contentType === record.expectedType
    && observed.sha256 === record.expectedSha256;

  if (!matches) return { ...record, status: 'rejected' };
  if (observed.scanStatus !== 'clean') {
    return { ...record, status: 'quarantined' };
  }
  return { ...record, status: 'available' };
}

const pending = {
  id: 'up-42',
  ownerId: 'user-7',
  key: 'quarantine/user-7/up-42',
  expectedSize: 12,
  expectedType: 'image/png',
  expectedSha256: 'abc123',
  status: 'pending',
};
const uploaded = {
  key: 'quarantine/user-7/up-42',
  size: 12,
  contentType: 'image/png',
  sha256: 'abc123',
  scanStatus: 'clean',
};

console.log(finalizeUpload(pending, uploaded, 'user-7').status);
console.log(finalizeUpload(pending, { ...uploaded, size: 13 }, 'user-7').status);
console.log(finalizeUpload(pending, { ...uploaded, scanStatus: 'infected' }, 'user-7').status);

available
rejected
quarantined

The pure function shows a publication decision, not the whole transaction. A real implementation reads observed through a trusted storage API and commits the result with a conditional update. Its scan result must also be bound to the same object version or checksum so the object cannot be replaced after scanning.

Pitfalls

Fix: Use the extension and declared type as cheap filters, then inspect the signature and parse the complete content with a maintained format decoder. Re-encode images into new files when appropriate instead of preserving unverified bytes. Parser failure, timeout, or resource exhaustion must leave the upload rejected.

Fix: Generate an object key independently of the original name and map it to ownership and display metadata in the database. If path resolution is unavoidable, use platform normalization and containment checks. Deleting ../ substrings does not cover absolute paths, separator variants, or symbolic links.

Fix: Add separate limits at the proxy, parser, streaming counter, tenant quota, and processor while preserving end-to-end backpressure. Test absent Content-Length, interrupted reads, slow consumers, and concurrent uploads near the admission limit. Verify that failures release both connections and partial files.

Fix: Model explicit states such as pending, quarantined, available, and rejected, and restrict downloads and downstream work to available. Publication checks the object version and conditionally updates the record. Scanner failure must fail closed instead of marking the object clean.

Fix: Bind an upload session to its owner, expiration, part positions, checksums, and allowed transitions. A repeated part must match its expected length and digest, and completion must be idempotent. Lifecycle work removes expired staging files and unfinished object-store multipart uploads.

Further reading

• RFC 7578: multipart/form-data
• OWASP File Upload Cheat Sheet
• Node.js 24 stream documentation
• Amazon S3: Uploading objects with presigned URLs
• Amazon S3: Multipart upload overview