File Upload — Presigned URL

An Experienced Engineer’s Walkthrough for Backend Engineers

When you first need to add “user uploads a file” to your system, the natural approach is: the client sends the file to your server, and your server saves it to disk or forwards it to object storage (S3, OSS, etc.). That design is correct and easy to reason about. Many internal tools and small products work exactly that way.

The problem starts when you stop thinking only about correctness and start thinking about scale, cost, and reliability. This article is written as if I’m sitting next to you: we’ll go through why the “proxy” approach hurts, what presigned URLs actually do, how to design them safely, and when to use them — and when not to. No prior experience with object storage or presigned URLs is assumed.

Lesson 1: Why “Upload Through the App” Becomes a Bottleneck

The First Design Most People Build

In the naive flow, the client (browser or app) does something like:

User selects a file.
Client sends the file in the request body (e.g. multipart form-data) to your API.
Your server receives the bytes, checks the user is logged in, maybe checks file type and size, then streams the file to object storage (or writes to disk).
After the upload succeeds, your server might write a record to the database (e.g. “user X uploaded file Y at path Z”).

Functionally, this is correct. The user uploads a file; you store it. So why would we want something else?

What Happens When Traffic Grows

As an experienced engineer, I care about where the bytes flow and where the CPU is spent.

Every byte of the file passes through your application server. If 10,000 users upload a 5 MB file each per day, that’s 50 GB per day in to your server and 50 GB out to storage. Your server’s network and CPU (parsing, buffering, streaming) are in the middle. You pay for bandwidth twice (client → you, you → storage), and your server becomes the bottleneck.
Slow or flaky clients (e.g. mobile on a bad network) hold a connection to your server for a long time. That ties up threads or connections. If the client disconnects mid-upload, you may have to retry the whole thing through your server again.
Peak traffic (e.g. many users uploading at once) can saturate your server’s network or CPU before you hit any limit on the storage side. So you scale the app server for file throughput, not just for “number of API requests.”

So the “proxy” design has a real cost: your app server is in the hot path of every byte. Presigned URLs are about moving that hot path off your server.

What a Presigned URL Actually Changes

The idea is simple to state:

Your app server never receives the file. It only does two things: (1) decide whether this user is allowed to upload (and optionally enforce quota), and (2) ask object storage for a special, time-limited URL that allows one upload to a specific path. Your server returns that URL to the client.
The client then uploads the file directly to object storage (e.g. HTTP PUT to that URL). The bytes go client → storage; your server is not in the middle.

So your server handles authorization and URL generation; the storage service handles storing the bytes. Bandwidth and streaming load move off your machines. As a newcomer, the first time you see this you might think “but how does the storage know the user is allowed to upload?” — the answer is: your server already decided that when it issued the URL. The URL itself is the capability. Whoever has the URL can upload once, to that path, until it expires. So the security is: only your server (with the storage secret key) can create such URLs, and only after checking the user.

Lesson 1 Takeaway

Presigned URLs are not about “fancy crypto” for its own sake. They are about moving the hot path (file transfer) off the application layer so the app stays small and fast, and so you don’t pay for or bottleneck on bandwidth and CPU for every byte. Think of the app as “issuing a capability” and the storage as “accepting bytes from whoever holds that capability.”

Lesson 2: The End-to-End Flow — Step by Step

Before we talk about security or implementation details, it helps to walk through the flow once, as a user and as a developer, so you know what happens at each step.

What the User and Client Experience

From the user’s point of view: they choose a file and click “Upload.” The client might show a progress bar. When it’s done, they see “Upload complete.” They don’t care whether the bytes went through your server or straight to S3/OSS.

From the client (browser or app) point of view:

It asks your API: “I want to upload a file; here’s the filename, type, and size (and maybe a hash).” It does not send the file content yet.
Your API returns a URL and possibly some form fields (if using POST with form). The URL is valid for a short time (e.g. 5–15 minutes).
The client then uploads the file to that URL (PUT or POST), directly to the storage domain (e.g. bucket.s3.amazonaws.com). So the second request does not go to your server at all.
If the upload succeeds, the client might call your API again to say “I’m done” (or your server finds out via a callback from storage — we’ll get to that).

So there are at least two logical steps: (1) get a URL from your server, (2) upload to that URL. As a beginner, a common mistake is to try to do both in one request; the whole point of presigned URL is to separate “authorization” (your server) from “transfer” (client ↔ storage).

Sequence Diagram — Who Talks to Whom

In words, step by step:

Client → App: “I want to upload.”
The client sends metadata only: filename, content type, size (and optionally a hash). Your server can reject here if the user is not authenticated, or if they’re over quota, or if the size/type is not allowed by your business rules. This is the only place where you enforce “who can upload what.”
App → Storage: “Give me a URL.”
Your server (using the storage provider’s SDK and secret key) asks for a presigned URL. You specify: path (e.g. user/123/uploads/abc-uuid), HTTP method (e.g. PUT), expiry time, and optionally a policy (e.g. max size, allowed content type). The storage returns a URL that includes a signature. Only your server can create this, because only your server has the secret.
App → Client: “Here’s your URL.”
You return the URL (and any extra form fields if needed) to the client. The client does not get your secret key; it only gets this one-time, limited URL.
Client → Storage: PUT the file.
The client uploads the file directly to the storage URL. Your server is not in the middle. Storage validates the signature and policy; if everything matches (path, method, expiry, size, type), it accepts the upload. If the client is slow or the network is bad, only the client and storage are involved; your server is not holding a long connection.
Optional: Storage → App: “Upload finished.”
Many storage systems support a callback (webhook): when the upload completes, storage sends an HTTP POST to your server with the object key, size, etag, etc. Then your server can persist a record (“user X uploaded file at key Y”) and trigger downstream work (e.g. thumbnail generation, virus scan). This way you don’t have to rely on the client to tell you “I’m done” — which could be lost or spoofed if not designed carefully.

1Request Presigned URL
Endpoint
POST /api/upload/request (or similar)
Request (metadata only)
Client sends filename, content type, size (and optionally hash). No file bytes.
Server checks auth and quota; rejects if not allowed.
Server logic
Validate user and business rules (size, type).
Call storage SDK with secret key to generate presigned URL (path, method PUT, expiry, optional policy).
Return the presigned URL (and any form fields) to the client.
Response
Presigned URL (valid 5–15 min typically).
Optional: form fields for POST upload.
Notes
App never sees file content; it only issues a capability (the URL).

2Upload File to Storage
Endpoint
PUT <presigned URL> (direct to storage domain)
Client
Upload file directly to the presigned URL (e.g. bucket.s3.amazonaws.com).
Request does not go to your app server.
Storage
Validates signature and policy; accepts upload if path, method, expiry, size/type match.
Returns 200 on success.
Optional callback
Storage can POST to your app when upload completes (webhook).
App then persists record (user, key, size) and triggers downstream (e.g. thumbnail).
Notes
Bandwidth and streaming load are off your server; only client and storage are in the path.

High-Level Architecture

So: App = auth, quota, URL generation, and (optionally) callback handling. Storage = store bytes, validate signature, optional callback. Client = get URL, then talk to storage. As a newcomer, keep this picture in your head: the app never sees the file content; it only issues a capability and optionally learns when the upload finished.

Lesson 2 Takeaway

The app never sees the file bytes. It only issues a capability (the presigned URL) and optionally learns when the upload finished via a callback. The client gets the URL in one request and performs the actual upload in a second request, directly to storage. Separating these two steps is what lets you move the heavy transfer off your server.

Lesson 3: Security — What Could Go Wrong, and How We Prevent It

A presigned URL is a “ticket” that allows one upload to a specific path. If that URL leaks (e.g. in logs, in a shared chat, or because the client sends it somewhere you didn’t intend), someone else could use it to upload a file to your bucket. So we design around: short lifetime, signed parameters, and server-enforced policy.

Why Expiry Matters

If the URL were valid forever, a single leak would allow anyone to upload to that path at any time. So we make the URL short-lived (e.g. 5–15 minutes). If the URL leaks, it becomes useless soon. If the client’s upload fails (e.g. network error), the client can ask your server for a new URL; there’s no need to make the first URL long-lived. As a beginner, it’s easy to set expiry too long “to be safe” for slow clients; the better approach is short expiry plus “retry by requesting a new URL.”

Why the Signature Matters

The URL contains a signature computed over: method (e.g. PUT), path, expiry, and (if you use a policy) the policy body. The signature is produced with a secret key that only your server (and the storage service) has. So:

The client cannot change the path (e.g. upload to another user’s folder) or extend the expiry, because that would invalidate the signature.
Only your server can create valid URLs, because only your server has the secret. So “who can get a URL?” is entirely under your control (your auth and quota logic).

If you didn’t have a signature, a client could tamper with the URL (e.g. change the path) and storage would have no way to know. So the signature is what makes the URL both tamper-proof and issuable only by you.

Why Policy Matters (Optional but Recommended)

Even with expiry and signature, the URL might allow “upload anything to this path.” A policy (e.g. a JSON document that you sign together with the URL) can restrict:

Content-Type (e.g. only image/jpeg, or image/*).
Max file size (e.g. 10 MB).
Path prefix (so the client can’t escape the folder you intended).

The policy is signed, so the client cannot relax it. Storage checks the actual upload against the policy and rejects if it doesn’t match. So even if the client is buggy or malicious, it cannot upload a 1 GB file or a .exe if your policy says “only images, max 10 MB.” As a newcomer, don’t skip the policy: “we’ll check on our server” is not enough, because the upload goes directly to storage; your server only sees metadata or the callback. The policy is how you enforce limits at the door (storage).

Summary Table

Point	Description
Expiry	Short-lived (e.g. 5–15 min); re-issue on failure; limits damage if URL leaks
Auth	App checks identity and quota before issuing URL; only app has secret to sign
Callback	On complete, OSS calls app; persist record, trigger follow-up; verify callback origin/signature
Policy	Restrict Content-Type, size, path; signed with secret; enforced by storage

Lesson 3 Takeaway

Security comes from short expiry (limit damage from leaks), signed parameters (tamper-proof, only your server can issue), and server-side policy (enforced by storage so the client can’t bypass). Never rely on the client alone to enforce limits; the upload doesn’t go through your server, so policy at storage is your enforcement point.

Lesson 4: Path Design and Callback — Details That Matter in Production

Path Isolation — Why It Matters

The presigned URL includes a path (object key) in the bucket. If you use a path like uploads/file1, and the client can influence the filename, one user might overwrite another’s file or escape to a different path. So we design paths so that:

They are isolated by user (or tenant). For example: user/{userId}/uploads/{uuid}.jpg. The userId comes from your auth; the uuid is generated by your server (so the client can’t guess or enumerate other users’ paths). That way one user cannot overwrite another’s file by guessing the path.
You can enforce quota and cleanup per user by prefix (e.g. “total size under user/123/”). Avoid paths that are predictable or enumerable (e.g. sequential IDs without a user prefix).

As a newcomer, a common mistake is to let the client choose the full path; then you must be very careful about path traversal and overwrite. Safer: server chooses path; client only gets a URL that writes to that path.

Callback — Why Verify and How to Stay Idempotent

When storage supports a callback (webhook) on upload completion, your server gets a POST with object key, size, etag, etc. You can then persist a record and trigger downstream work. Two things that matter:

Verify the callback. The request comes from the internet; anyone could POST to your URL and claim “upload complete.” So you must verify that the request really came from your storage provider (e.g. check a signature or shared secret in the request). If you don’t, an attacker could forge callbacks and create fake “upload complete” records in your DB.
Idempotency. The same upload might trigger the callback more than once (e.g. retries). So when you process the callback, use the object key (or a request id from the callback) as an idempotency key: if you’ve already processed this key, skip or return success without duplicating the record. That way duplicate callbacks don’t create duplicate DB rows or duplicate downstream jobs.

Lesson 4 Takeaway

Path design prevents privilege escalation (one user can’t overwrite another) and simplifies quota and cleanup. Callback verification prevents forgery; idempotent processing prevents duplicate records when the provider retries the callback.

Lesson 5: When to Use Presigned URL — and When Not To

When Presigned URL Is a Good Fit

You want to offload bandwidth and CPU from the app server, and you’re okay with the client talking directly to storage (which is the normal case for web and mobile).
You can accept short-lived URLs and “request a new URL if upload fails.”
You’re okay with eventual consistency for “upload complete” (e.g. you learn via callback a few seconds after the client finishes). If you need “return to the client only after we’ve written the DB record,” you can still do that by having the client poll or wait for a second API that checks the record created by the callback.

When to Prefer Upload Through the App

You need strict validation before accepting the file — e.g. virus scan or transcoding — and you want to do that before the file is stored. With presigned URL, the file lands in storage first; you’d run virus scan or transcoding in a separate step after callback. So “validate then store” is simpler when the file goes through your server; “store then validate” is the presigned-URL pattern.
Policy or compliance requires that no client talks directly to storage (e.g. all traffic must go through a gateway). Then you have to proxy; presigned URL is not an option for that path.

Download

Presigned URLs also work for private download: your server checks permission, then generates a short-lived download URL; the client accesses storage directly to download. Same idea: app issues capability, storage serves bytes; your server doesn’t stream the file.

Lesson 5 Takeaway

Use presigned URL when you want to move the hot path off your server and can accept short-lived URLs and eventual consistency for completion. Use upload-through-app when you need validate-then-store or when direct client–storage access is not allowed. For private download, the same “issue a URL, client talks to storage” pattern applies.

Key Rules (Summary)

Expiry: Keep it short (e.g. 5–15 min); client can request a new URL if the upload fails.
Path: Isolate by user/tenant; server chooses path; avoid guessable or shared paths.
Callback: Verify origin and signature; process idempotently (by object key or request id).
Policy: Use signed policy to enforce type, size, and path on the storage side; don’t rely on client-only checks.

What's Next

See API Gateway for auth. See your object storage docs (S3, OSS, etc.) for presigned URL APIs and policy syntax.

File Upload - Presigned URL