Skip to content

19 · Safe Retries and Idempotent Browser Writes

The previous lesson assembled our complete document-review workflow:

Browser POST
→ FastAPI
→ service
→ SQLite commits a durable job
→ 202 Accepted
→ worker performs review
→ SSE reports progress

But one difficult situation remained:

SQLite commits the job
→ FastAPI sends 202
→ network connection breaks
→ browser never receives the response

The browser sees an error.

The database contains a valid queued job.

If JavaScript blindly retries the POST, it may create a second job.

Today we will make that retry safe using an idempotency key.

The improved lifecycle is:

Browser creates one operation ID
↓
POST includes Idempotency-Key
↓
Service stores the key with the job
↓
Response is lost
↓
Browser retries with the same key
↓
Service finds the existing job
↓
No duplicate job is created
↓
Browser resumes progress tracking

This lesson sits directly at the boundary between browser uncertainty and database certainty.


An operation is idempotent when repeating the same intended operation produces the same effective result.

For our application:

Start document review with operation key ABC

may be submitted twice, but it should create only one review job.

The first request may produce:

Job 410 created

The retry should produce:

Job 410 already exists

It should not produce:

Job 411 created

An HTTP POST is not inherently idempotent. We are adding application behavior that makes this particular POST safely repeatable.


When the user deliberately starts a new review, JavaScript creates a UUID:

const operationKey = crypto.randomUUID();

The result resembles:

8c043429-e90e-4351-ad69-1aa0dd938a4d

This value does not identify the user or document by itself.

It identifies one intended operation:

User 27
+
Idempotency key 8c043429-...
→ one review-start operation

The key is not an authentication credential. Knowing it does not grant access to anything.

Authentication, authorization and CSRF verification still occur normally.


Our ReviewJob model keeps the depth and timestamp fields from lesson 18 and gains an idempotency key plus a uniqueness constraint:

app/models/review_jobs.py
from datetime import datetime, timezone
from sqlalchemy import (
ForeignKey,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.orm import Mapped, mapped_column
from ..database import Base
class ReviewJob(Base):
__tablename__ = "review_jobs"
__table_args__ = (
UniqueConstraint(
"requested_by_user_id",
"idempotency_key",
name="uq_review_jobs_user_idempotency_key",
),
)
id: Mapped[int] = mapped_column(
primary_key=True,
)
document_id: Mapped[int] = mapped_column(
ForeignKey("documents.id"),
index=True,
)
requested_by_user_id: Mapped[int] = mapped_column(
ForeignKey("users.id"),
)
idempotency_key: Mapped[str] = mapped_column(
String(36),
)
depth: Mapped[str] = mapped_column(
String(20),
)
status: Mapped[str] = mapped_column(
String(30),
index=True,
)
progress: Mapped[int] = mapped_column(
default=0,
)
result_summary: Mapped[str | None] = mapped_column(
Text,
nullable=True,
)
error_message: Mapped[str | None] = mapped_column(
Text,
nullable=True,
)
created_at: Mapped[datetime] = mapped_column(
default=lambda: datetime.now(timezone.utc),
)
started_at: Mapped[datetime | None] = mapped_column(
nullable=True,
)
finished_at: Mapped[datetime | None] = mapped_column(
nullable=True,
)

The important database rule is:

requested_by_user_id + idempotency_key
must be unique

Different users may coincidentally generate the same key:

Alice + ABC
Max + ABC

Those are separate operations.

But Alice cannot create two jobs with:

Alice + ABC

The database is the final authority, even if two matching requests arrive almost simultaneously.

An Alembic migration establishes these columns and the unique constraint before the new application code is deployed.

Changing the SQLAlchemy model alone would not modify the existing SQLite table.


Why the key is stored with the business record

Section titled “Why the key is stored with the business record”

We could store recently seen keys in a Python dictionary:

seen_keys = {}

That would fail as soon as:

  • FastAPI restarts
  • Another Uvicorn worker handles the retry
  • The application runs on another server
  • The dictionary is cleared
  • The original request and retry reach different processes

The review job is durable, so its operation identity should be durable too.

Review job
→ SQLite row
Idempotency key
→ part of that row

The browser and web process may disappear. The database can still recognize the operation later.


The response may describe either a newly created job or an existing one:

from typing import Literal
from pydantic import BaseModel
ReviewStatus = Literal[
"queued",
"processing",
"completed",
"failed",
]
class ReviewAccepted(BaseModel):
job_id: int
document_id: int
status: ReviewStatus
progress: int
replayed: bool
events_url: str
result_summary: str | None = None
error_message: str | None = None

A new operation might return:

{
"job_id": 410,
"document_id": 101,
"status": "queued",
"progress": 0,
"replayed": false,
"events_url": "http://localhost:8000/documents/101/review-events?job_id=410",
"result_summary": null,
"error_message": null
}

A retry might return:

{
"job_id": 410,
"document_id": 101,
"status": "processing",
"progress": 40,
"replayed": true,
"events_url": "http://localhost:8000/documents/101/review-events?job_id=410",
"result_summary": null,
"error_message": null
}

The status may have advanced between the original request and its retry.

The replay returns the current durable state rather than pretending the job is still newly queued.


The route receives the key through an HTTP header:

Idempotency-Key: 8c043429-e90e-4351-ad69-1aa0dd938a4d

FastAPI can parse it as a Python UUID:

from typing import Annotated
from uuid import UUID
from fastapi import Header
IdempotencyKey = Annotated[
UUID,
Header(alias="Idempotency-Key"),
]

The route declares:

idempotency_key: IdempotencyKey

FastAPI now prepares another validated object before calling the route.

If the header is missing or is not a valid UUID, the route does not proceed normally.

The key travels through these representations:

Browser JavaScript
→ UUID string
HTTP
→ Idempotency-Key header text
FastAPI
→ Python UUID object
Service
→ canonical string
SQLite
→ VARCHAR value

The route becomes:

@router.post(
"/{document_id}/reviews",
response_model=ReviewAccepted,
status_code=status.HTTP_202_ACCEPTED,
name="start_document_review",
)
def start_document_review(
request: Request,
response: Response,
payload: ReviewStart,
idempotency_key: IdempotencyKey,
document: AuthorizedDocument,
operator: ReviewOperator,
db: DbSession,
):
verify_csrf_token(request)
try:
job, replayed = queue_document_review(
db=db,
document_id=document.id,
firm_id=document.firm_id,
requested_by_user_id=operator["id"],
idempotency_key=idempotency_key,
depth=payload.depth,
)
except IdempotencyConflictError as exc:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=str(exc),
) from exc
except ReviewAlreadyRunningError as exc:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=str(exc),
) from exc
if replayed:
response.status_code = status.HTTP_200_OK
return ReviewAccepted(
job_id=job.id,
document_id=job.document_id,
status=job.status,
progress=job.progress,
replayed=replayed,
events_url=str(
request.url_for(
"document_review_events",
document_id=job.document_id,
).include_query_params(job_id=job.id)
),
result_summary=job.result_summary,
error_message=job.error_message,
)

The route can return two successful HTTP statuses:

Status Meaning
202 Accepted A new durable job was created
200 OK This operation was already accepted; here is its existing job

Both responses contain the same Pydantic shape.

The role requirement from lessons 14 and 18 still applies through ReviewOperator. The service receives the already authorized document’s firm, preserving cross-firm administrator access. The events URL includes this job’s ID, and a completed replay includes its saved result so the browser can display it without opening a new stream.

The browser does not need separate parsing logic for each success status.


The service looks for a previous operation

Section titled “The service looks for a previous operation”

The service begins by converting the UUID to its standard text representation:

key_text = str(idempotency_key)

Then it searches for a matching job:

existing_job = db.scalar(
select(ReviewJob).where(
ReviewJob.requested_by_user_id
== requested_by_user_id,
ReviewJob.idempotency_key
== key_text,
)
)

If it finds one, it does not immediately create anything.

It first checks whether the key is being reused for the same request:

if existing_job is not None:
if (
existing_job.document_id != document_id
or existing_job.depth != depth
):
raise IdempotencyConflictError(
"This operation key was already used "
"for a different request."
)
return existing_job, True

This comparison is essential.

Suppose Alice first sends:

{
"document_id": 101,
"depth": "standard"
}

using key ABC.

Later, some buggy code sends:

{
"document_id": 202,
"depth": "thorough"
}

using the same key.

The server must not silently return document 101’s job as if the requests were equivalent.

The rule is:

Same key + same operation
→ safe replay
Same key + different operation
→ conflict

Here is the full service:

from uuid import UUID
from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from ..models.documents import Document
from ..models.review_jobs import ReviewJob
class IdempotencyConflictError(Exception):
pass
class ReviewAlreadyRunningError(Exception):
pass
STARTABLE_STATUSES = {
"not_started",
"completed",
"failed",
}
def queue_document_review(
*,
db: Session,
document_id: int,
firm_id: int,
requested_by_user_id: int,
idempotency_key: UUID,
depth: str,
) -> tuple[ReviewJob, bool]:
key_text = str(idempotency_key)
existing_job = find_idempotent_job(
db=db,
user_id=requested_by_user_id,
key=key_text,
)
if existing_job is not None:
verify_same_operation(
job=existing_job,
document_id=document_id,
depth=depth,
)
return existing_job, True
transition = db.execute(
update(Document)
.where(
Document.id == document_id,
Document.firm_id == firm_id,
Document.review_status.in_(
STARTABLE_STATUSES
),
)
.values(
review_status="queued",
)
)
if transition.rowcount != 1:
db.rollback()
existing_job = find_idempotent_job(
db=db,
user_id=requested_by_user_id,
key=key_text,
)
if existing_job is not None:
verify_same_operation(
job=existing_job,
document_id=document_id,
depth=depth,
)
return existing_job, True
raise ReviewAlreadyRunningError(
"A review is already active."
)
job = ReviewJob(
document_id=document_id,
requested_by_user_id=requested_by_user_id,
idempotency_key=key_text,
depth=depth,
status="queued",
progress=0,
)
db.add(job)
try:
db.commit()
except IntegrityError:
db.rollback()
existing_job = find_idempotent_job(
db=db,
user_id=requested_by_user_id,
key=key_text,
)
if existing_job is None:
raise
verify_same_operation(
job=existing_job,
document_id=document_id,
depth=depth,
)
return existing_job, True
db.refresh(job)
return job, False

The returned Boolean means:

False
→ The service created this job now
True
→ The service found a prior job for this operation

The lookup helper is intentionally small:

def find_idempotent_job(
*,
db: Session,
user_id: int,
key: str,
) -> ReviewJob | None:
return db.scalar(
select(ReviewJob).where(
ReviewJob.requested_by_user_id
== user_id,
ReviewJob.idempotency_key
== key,
)
)

The comparison helper protects against key reuse:

def verify_same_operation(
*,
job: ReviewJob,
document_id: int,
depth: str,
) -> None:
if (
job.document_id != document_id
or job.depth != depth
):
raise IdempotencyConflictError(
"This operation key was already used "
"for different review parameters."
)

These are service-layer concerns.

The router should not contain database race-handling logic. The browser should not be expected to enforce uniqueness.


The service searches for an existing job before attempting the insert.

Why also catch:

IntegrityError

afterward?

Because two requests can race:

Request A checks for key ABC
→ none exists
Request B checks for key ABC
→ none exists
Request A inserts ABC
Request B inserts ABC

The initial Python query is not enough.

The database unique constraint resolves the race:

First INSERT
→ succeeds
Second INSERT
→ rejected

The losing request rolls back and loads the row created by the winner.

The translation is:

SQLite unique-constraint failure
→ SQLAlchemy IntegrityError
→ service reloads existing job
→ route returns successful replay response

A database integrity error does not always mean the user’s operation failed. In this case it can mean another copy of the same request already succeeded.


The browser must retain the key across retries

Section titled “The browser must retain the key across retries”

The most important browser rule is:

Generate one key per intended operation, not one key per HTTP attempt.

This would be wrong:

async function submitReview() {
const key = crypto.randomUUID();
await fetch(url, {
headers: {
"Idempotency-Key": key,
},
});
}

If submitReview() is called again after a network failure, it generates a new key.

The server reasonably interprets that as a new operation.

Instead, we store the pending operation:

let pendingOperation = null;

When the user initiates a review:

function prepareOperation() {
if (pendingOperation !== null) {
return pendingOperation;
}
pendingOperation = {
key: crypto.randomUUID(),
depth: depthElement.value,
};
return pendingOperation;
}

The operation contains both:

key
→ identity of this operation
depth
→ original input belonging to the operation

If the request must be retried, both remain unchanged.


The submit function becomes:

async function startReview(event) {
event.preventDefault();
const operation = prepareOperation();
startButton.disabled = true;
depthElement.disabled = true;
resultSection.hidden = true;
summaryElement.textContent = "";
messageElement.textContent =
"Submitting review request…";
try {
const response = await fetch(
panel.dataset.startUrl,
{
method: "POST",
headers: {
"Content-Type": "application/json",
"Accept": "application/json",
"X-CSRF-Token":
panel.dataset.csrfToken,
"Idempotency-Key":
operation.key,
},
body: JSON.stringify({
depth: operation.depth,
}),
},
);
if (response.status === 409) {
pendingOperation = null;
throw new Error(
"The request conflicts with the current review state. Reload the page."
);
}
if (response.status >= 400 && response.status < 500) {
pendingOperation = null;
throw new Error(
"The review request was rejected."
);
}
if (!response.ok) {
throw new Error(
"The server could not confirm this operation."
);
}
const data = await response.json();
pendingOperation = null;
startButton.textContent = "Start review";
panel.dataset.eventsUrl = data.events_url;
statusElement.textContent = data.status;
progressElement.textContent =
`${data.progress}%`;
messageElement.textContent =
data.replayed
? "Existing review recovered."
: "Review queued.";
if (
data.status === "queued"
|| data.status === "processing"
) {
openReviewEvents(data.events_url);
} else {
updateReviewDisplay(data);
}
} catch (error) {
if (pendingOperation !== null) {
messageElement.textContent =
"The outcome is uncertain. Retry will safely check the same operation.";
startButton.disabled = false;
startButton.textContent =
"Retry status";
depthElement.disabled = true;
return;
}
messageElement.textContent =
error.message;
startButton.textContent = "Start review";
setReviewControls(false);
}
}

A network failure commonly appears as a TypeError, but it is not the only uncertain outcome. A server error can occur after a commit, and an accepted response can arrive with an unreadable body. We retain the pending key for all of those cases. A definite client rejection clears it; a readable acknowledgment clears it after parsing.

In that branch, we deliberately do not clear:

pendingOperation

The next form submission reuses:

operation.key
operation.depth

The browser is retrying the same logical operation.

This example retains the pending key in page memory. Reloading discards it and reads the stored job through Jinja. Retaining the same key across a full browser restart would require persistent client storage or a server-side operation lookup; it is not a guarantee supplied by this variable.


After the first uncertain attempt, we retain:

pendingOperation.depth

Suppose the original request used:

standard

The user then changes the select field to:

thorough

and retries using the same key.

That would violate the operation’s meaning:

Same key
Different payload

Therefore, while an operation is uncertain:

depthElement.disabled = true;

The browser preserves the original input.

If the user wants to initiate a genuinely different review later, the application creates a new idempotency key.


Now let’s trace the exact failure.

The browser creates:

Key: 8c043429-...
Depth: standard

It sends:

POST /documents/101/reviews HTTP/1.1
Cookie: app_session=...
X-CSRF-Token: ...
Idempotency-Key: 8c043429-...
Content-Type: application/json
{"depth":"standard"}

FastAPI:

  1. Authenticates the user.
  2. Authorizes access to document 101.
  3. Verifies CSRF.
  4. Validates the header as a UUID.
  5. Validates the body as ReviewStart.
  6. Calls the service.

The service:

  1. Finds no existing job for this key.
  2. Changes the document to queued.
  3. Inserts job 410.
  4. Stores the key on job 410.
  5. Commits.

SQLite now contains:

job_id: 410
document_id: 101
requested_by: 27
idempotency_key: 8c043429-...
depth: standard
status: queued

FastAPI attempts to return 202.

The network breaks.


The browser knows:

I sent the request.
I did not receive a usable response.

It does not know:

Did the server receive it?
Did validation pass?
Did SQLite commit?
Did the response merely get lost?

The interface displays:

The outcome is uncertain. Retry will safely check the same operation.

This is an honest error state.


The user clicks again.

JavaScript reuses:

Key: 8c043429-...
Depth: standard

FastAPI authenticates, authorizes and validates the retry exactly as it did the first request.

The service queries SQLite and finds job 410.

It verifies:

Same user
Same key
Same document
Same depth

Then it returns:

existing_job, True

FastAPI responds:

HTTP/1.1 200 OK
Content-Type: application/json
{
"job_id": 410,
"document_id": 101,
"status": "processing",
"progress": 40,
"replayed": true,
"events_url": "http://localhost:8000/documents/101/review-events?job_id=410",
"result_summary": null,
"error_message": null
}

No second job is created.

The browser displays:

Existing review recovered.

Then it opens the SSE connection and resumes normal progress updates.


The service lookup includes:

ReviewJob.requested_by_user_id
== requested_by_user_id

But that does not replace document authorization.

The route still receives:

document: AuthorizedDocument

before calling the service.

A user cannot submit someone else’s document ID and a guessed idempotency key to gain access.

Every retry is a new HTTP request:

New middleware execution
→ new authentication check
→ new authorization check
→ new CSRF verification
→ new Pydantic validation

The key answers only:

Have we already accepted this same logical operation from this user?

It does not answer:

Is this user currently permitted to perform it?


The application needs a retention policy.

If review jobs remain permanently in the database, their keys remain available permanently.

If old jobs are archived or deleted, the application must decide how long retry protection should remain.

A typical principle is:

Keep idempotency records longer than the longest realistic client retry window.

For our small application, keeping the key on the durable review job is simple.

Larger systems sometimes maintain a separate table containing:

User or tenant
Idempotency key
Request fingerprint
HTTP status
Response body
Creation time
Expiration time

That lets the server reproduce the original response even when the business object has moved or been archived.


The worker must also handle retries safely

Section titled “The worker must also handle retries safely”

Protecting the POST prevents duplicate job creation.

It does not automatically make every step inside the worker safe.

Suppose a worker:

  1. Sends a document to an external review provider.
  2. Crashes before storing the provider’s result.
  3. Restarts and retries the job.
  4. Sends the same document again.

The external service may perform the work twice or charge twice.

A complete design may propagate a stable identifier downstream:

Our review job ID: 410
→ external operation key: review-job-410

The worker may also store:

external_request_id
attempt_count
last_attempt_at

Idempotency must be considered at every boundary where duplicate effects would matter.

For our current SQLite worker, the database job ID is the natural durable identity.


Our browser already disables the Start button:

startButton.disabled = true;

That prevents ordinary accidental double-clicks.

But it does not protect against:

  • A network retry
  • Two browser tabs
  • A script sending requests directly
  • A mobile browser resubmitting
  • A proxy retry
  • Two nearly simultaneous event handlers
  • An impatient user reloading the page

Interface controls improve usability.

Database constraints protect data integrity.

The two layers have different responsibilities:

Disabled button
→ reduce accidental duplicate interaction
Idempotency key
→ identify repeated logical request
Unique constraint
→ enforce the rule under concurrency

Idempotency versus duplicate-business rules

Section titled “Idempotency versus duplicate-business rules”

Our application now has two related protections.

The first is the document state transition:

A document may have only one active review.

The second is the operation identity:

The same submitted operation may be safely replayed.

These produce different outcomes.

Same user
Same key
Same document
Same depth

Result:

Return existing job successfully

Different operation while review is active

Section titled “Different operation while review is active”
Same document
Different key

Result:

409 Conflict

The first is a retry.

The second is a new business request that conflicts with current state.

That distinction allows the server to be both safe and helpful.


For a successful first attempt:

  1. Jinja renders the document page and CSRF token.
  2. The browser constructs the DOM.
  3. The user submits the form.
  4. JavaScript creates an operation UUID.
  5. JavaScript freezes the selected depth.
  6. JSON.stringify() creates the request body.
  7. fetch() sends the body, cookie, CSRF token and idempotency key.
  8. Middleware reconstructs the session.
  9. FastAPI authenticates the user.
  10. The authorization dependency loads the permitted document.
  11. FastAPI parses the UUID header.
  12. Pydantic validates the JSON body.
  13. The route verifies CSRF.
  14. The service finds no matching prior job.
  15. The service atomically changes the document to queued.
  16. SQLAlchemy inserts the new ReviewJob.
  17. SQLite enforces the unique key constraint.
  18. SQLite commits.
  19. FastAPI returns 202 Accepted.
  20. JavaScript parses the response.
  21. JavaScript clears the pending operation key.
  22. The browser opens the SSE stream.
  23. The worker and progress lifecycle continue normally.

For a lost response and retry:

  1. Steps 1–18 occur normally.
  2. The response disappears in transit.
  3. JavaScript receives a network error.
  4. JavaScript retains the original key and depth.
  5. The interface offers a safe retry.
  6. The browser resends the same logical operation.
  7. Middleware, authentication, authorization and CSRF run again.
  8. The service finds the existing job.
  9. The service verifies that the payload matches.
  10. No INSERT occurs.
  11. FastAPI returns 200 OK with the existing job.
  12. JavaScript displays its current status.
  13. SSE resumes progress reporting.

A network error tells the browser:

I do not know what happened.

It does not necessarily mean:

Nothing happened.

The idempotency key gives one intended operation a stable identity across multiple HTTP attempts:

Logical operation
→ one key
First HTTP attempt
→ may succeed or disappear
Retry
→ same key
Database
→ returns the same durable job

The full safety chain is:

Browser retains operation identity
↓
HTTP carries Idempotency-Key
↓
FastAPI validates the header
↓
Service compares the intended operation
↓
SQLAlchemy executes within a transaction
↓
SQLite enforces uniqueness
↓
Route returns new or existing job
↓
Browser resumes from authoritative state

This is one of the clearest examples of why a production application cannot treat the browser, HTTP response or visible DOM as the final truth.

The database records what actually happened. The idempotency key gives the browser a safe way to ask again.