19 · Safe Retries and Idempotent Browser Writes
The previous lesson assembled our complete document-review workflow:
Browser POST→ FastAPI→ service→ SQLite commits a durable job→ 202 Accepted→ worker performs review→ SSE reports progressBut one difficult situation remained:
SQLite commits the job→ FastAPI sends 202→ network connection breaks→ browser never receives the responseThe browser sees an error.
The database contains a valid queued job.
If JavaScript blindly retries the POST, it may create a second job.
Today we will make that retry safe using an idempotency key.
The improved lifecycle is:
Browser creates one operation ID ↓POST includes Idempotency-Key ↓Service stores the key with the job ↓Response is lost ↓Browser retries with the same key ↓Service finds the existing job ↓No duplicate job is created ↓Browser resumes progress trackingThis lesson sits directly at the boundary between browser uncertainty and database certainty.
What “idempotent” means here
Section titled “What “idempotent” means here”An operation is idempotent when repeating the same intended operation produces the same effective result.
For our application:
Start document review with operation key ABCmay be submitted twice, but it should create only one review job.
The first request may produce:
Job 410 createdThe retry should produce:
Job 410 already existsIt should not produce:
Job 411 createdAn HTTP POST is not inherently idempotent. We are adding application behavior that makes this particular POST safely repeatable.
The browser creates the operation key
Section titled “The browser creates the operation key”When the user deliberately starts a new review, JavaScript creates a UUID:
const operationKey = crypto.randomUUID();The result resembles:
8c043429-e90e-4351-ad69-1aa0dd938a4dThis value does not identify the user or document by itself.
It identifies one intended operation:
User 27+Idempotency key 8c043429-...→ one review-start operationThe key is not an authentication credential. Knowing it does not grant access to anything.
Authentication, authorization and CSRF verification still occur normally.
The database records the key
Section titled “The database records the key”Our ReviewJob model keeps the depth and timestamp fields from lesson 18 and gains an idempotency key plus a uniqueness constraint:
from datetime import datetime, timezone
from sqlalchemy import ( ForeignKey, String, Text, UniqueConstraint,)from sqlalchemy.orm import Mapped, mapped_column
from ..database import Base
class ReviewJob(Base): __tablename__ = "review_jobs"
__table_args__ = ( UniqueConstraint( "requested_by_user_id", "idempotency_key", name="uq_review_jobs_user_idempotency_key", ), )
id: Mapped[int] = mapped_column( primary_key=True, )
document_id: Mapped[int] = mapped_column( ForeignKey("documents.id"), index=True, )
requested_by_user_id: Mapped[int] = mapped_column( ForeignKey("users.id"), )
idempotency_key: Mapped[str] = mapped_column( String(36), )
depth: Mapped[str] = mapped_column( String(20), )
status: Mapped[str] = mapped_column( String(30), index=True, )
progress: Mapped[int] = mapped_column( default=0, )
result_summary: Mapped[str | None] = mapped_column( Text, nullable=True, )
error_message: Mapped[str | None] = mapped_column( Text, nullable=True, )
created_at: Mapped[datetime] = mapped_column( default=lambda: datetime.now(timezone.utc), )
started_at: Mapped[datetime | None] = mapped_column( nullable=True, )
finished_at: Mapped[datetime | None] = mapped_column( nullable=True, )The important database rule is:
requested_by_user_id + idempotency_keymust be uniqueDifferent users may coincidentally generate the same key:
Alice + ABCMax + ABCThose are separate operations.
But Alice cannot create two jobs with:
Alice + ABCThe database is the final authority, even if two matching requests arrive almost simultaneously.
An Alembic migration establishes these columns and the unique constraint before the new application code is deployed.
Changing the SQLAlchemy model alone would not modify the existing SQLite table.
Why the key is stored with the business record
Section titled “Why the key is stored with the business record”We could store recently seen keys in a Python dictionary:
seen_keys = {}That would fail as soon as:
- FastAPI restarts
- Another Uvicorn worker handles the retry
- The application runs on another server
- The dictionary is cleared
- The original request and retry reach different processes
The review job is durable, so its operation identity should be durable too.
Review job→ SQLite row
Idempotency key→ part of that rowThe browser and web process may disappear. The database can still recognize the operation later.
The response model now supports replays
Section titled “The response model now supports replays”The response may describe either a newly created job or an existing one:
from typing import Literal
from pydantic import BaseModel
ReviewStatus = Literal[ "queued", "processing", "completed", "failed",]
class ReviewAccepted(BaseModel): job_id: int document_id: int status: ReviewStatus progress: int replayed: bool events_url: str result_summary: str | None = None error_message: str | None = NoneA new operation might return:
{ "job_id": 410, "document_id": 101, "status": "queued", "progress": 0, "replayed": false, "events_url": "http://localhost:8000/documents/101/review-events?job_id=410", "result_summary": null, "error_message": null}A retry might return:
{ "job_id": 410, "document_id": 101, "status": "processing", "progress": 40, "replayed": true, "events_url": "http://localhost:8000/documents/101/review-events?job_id=410", "result_summary": null, "error_message": null}The status may have advanced between the original request and its retry.
The replay returns the current durable state rather than pretending the job is still newly queued.
Reading the header in FastAPI
Section titled “Reading the header in FastAPI”The route receives the key through an HTTP header:
Idempotency-Key: 8c043429-e90e-4351-ad69-1aa0dd938a4dFastAPI can parse it as a Python UUID:
from typing import Annotatedfrom uuid import UUID
from fastapi import Header
IdempotencyKey = Annotated[ UUID, Header(alias="Idempotency-Key"),]The route declares:
idempotency_key: IdempotencyKeyFastAPI now prepares another validated object before calling the route.
If the header is missing or is not a valid UUID, the route does not proceed normally.
The key travels through these representations:
Browser JavaScript→ UUID string
HTTP→ Idempotency-Key header text
FastAPI→ Python UUID object
Service→ canonical string
SQLite→ VARCHAR valueThe updated route
Section titled “The updated route”The route becomes:
@router.post( "/{document_id}/reviews", response_model=ReviewAccepted, status_code=status.HTTP_202_ACCEPTED, name="start_document_review",)def start_document_review( request: Request, response: Response, payload: ReviewStart, idempotency_key: IdempotencyKey, document: AuthorizedDocument, operator: ReviewOperator, db: DbSession,): verify_csrf_token(request)
try: job, replayed = queue_document_review( db=db, document_id=document.id, firm_id=document.firm_id, requested_by_user_id=operator["id"], idempotency_key=idempotency_key, depth=payload.depth, )
except IdempotencyConflictError as exc: raise HTTPException( status_code=status.HTTP_409_CONFLICT, detail=str(exc), ) from exc
except ReviewAlreadyRunningError as exc: raise HTTPException( status_code=status.HTTP_409_CONFLICT, detail=str(exc), ) from exc
if replayed: response.status_code = status.HTTP_200_OK
return ReviewAccepted( job_id=job.id, document_id=job.document_id, status=job.status, progress=job.progress, replayed=replayed, events_url=str( request.url_for( "document_review_events", document_id=job.document_id, ).include_query_params(job_id=job.id) ), result_summary=job.result_summary, error_message=job.error_message, )The route can return two successful HTTP statuses:
| Status | Meaning |
|---|---|
202 Accepted |
A new durable job was created |
200 OK |
This operation was already accepted; here is its existing job |
Both responses contain the same Pydantic shape.
The role requirement from lessons 14 and 18 still applies through ReviewOperator. The service receives the already authorized document’s firm, preserving cross-firm administrator access. The events URL includes this job’s ID, and a completed replay includes its saved result so the browser can display it without opening a new stream.
The browser does not need separate parsing logic for each success status.
The service looks for a previous operation
Section titled “The service looks for a previous operation”The service begins by converting the UUID to its standard text representation:
key_text = str(idempotency_key)Then it searches for a matching job:
existing_job = db.scalar( select(ReviewJob).where( ReviewJob.requested_by_user_id == requested_by_user_id,
ReviewJob.idempotency_key == key_text, ))If it finds one, it does not immediately create anything.
It first checks whether the key is being reused for the same request:
if existing_job is not None: if ( existing_job.document_id != document_id or existing_job.depth != depth ): raise IdempotencyConflictError( "This operation key was already used " "for a different request." )
return existing_job, TrueThis comparison is essential.
Suppose Alice first sends:
{ "document_id": 101, "depth": "standard"}using key ABC.
Later, some buggy code sends:
{ "document_id": 202, "depth": "thorough"}using the same key.
The server must not silently return document 101’s job as if the requests were equivalent.
The rule is:
Same key + same operation→ safe replay
Same key + different operation→ conflictThe complete service
Section titled “The complete service”Here is the full service:
from uuid import UUID
from sqlalchemy import select, updatefrom sqlalchemy.exc import IntegrityErrorfrom sqlalchemy.orm import Session
from ..models.documents import Documentfrom ..models.review_jobs import ReviewJob
class IdempotencyConflictError(Exception): pass
class ReviewAlreadyRunningError(Exception): pass
STARTABLE_STATUSES = { "not_started", "completed", "failed",}
def queue_document_review( *, db: Session, document_id: int, firm_id: int, requested_by_user_id: int, idempotency_key: UUID, depth: str,) -> tuple[ReviewJob, bool]: key_text = str(idempotency_key)
existing_job = find_idempotent_job( db=db, user_id=requested_by_user_id, key=key_text, )
if existing_job is not None: verify_same_operation( job=existing_job, document_id=document_id, depth=depth, )
return existing_job, True
transition = db.execute( update(Document) .where( Document.id == document_id, Document.firm_id == firm_id, Document.review_status.in_( STARTABLE_STATUSES ), ) .values( review_status="queued", ) )
if transition.rowcount != 1: db.rollback()
existing_job = find_idempotent_job( db=db, user_id=requested_by_user_id, key=key_text, )
if existing_job is not None: verify_same_operation( job=existing_job, document_id=document_id, depth=depth, )
return existing_job, True
raise ReviewAlreadyRunningError( "A review is already active." )
job = ReviewJob( document_id=document_id, requested_by_user_id=requested_by_user_id, idempotency_key=key_text, depth=depth, status="queued", progress=0, )
db.add(job)
try: db.commit()
except IntegrityError: db.rollback()
existing_job = find_idempotent_job( db=db, user_id=requested_by_user_id, key=key_text, )
if existing_job is None: raise
verify_same_operation( job=existing_job, document_id=document_id, depth=depth, )
return existing_job, True
db.refresh(job)
return job, FalseThe returned Boolean means:
False→ The service created this job now
True→ The service found a prior job for this operationThe helper functions
Section titled “The helper functions”The lookup helper is intentionally small:
def find_idempotent_job( *, db: Session, user_id: int, key: str,) -> ReviewJob | None: return db.scalar( select(ReviewJob).where( ReviewJob.requested_by_user_id == user_id,
ReviewJob.idempotency_key == key, ) )The comparison helper protects against key reuse:
def verify_same_operation( *, job: ReviewJob, document_id: int, depth: str,) -> None: if ( job.document_id != document_id or job.depth != depth ): raise IdempotencyConflictError( "This operation key was already used " "for different review parameters." )These are service-layer concerns.
The router should not contain database race-handling logic. The browser should not be expected to enforce uniqueness.
Why we check twice
Section titled “Why we check twice”The service searches for an existing job before attempting the insert.
Why also catch:
IntegrityErrorafterward?
Because two requests can race:
Request A checks for key ABC→ none exists
Request B checks for key ABC→ none exists
Request A inserts ABCRequest B inserts ABCThe initial Python query is not enough.
The database unique constraint resolves the race:
First INSERT→ succeeds
Second INSERT→ rejectedThe losing request rolls back and loads the row created by the winner.
The translation is:
SQLite unique-constraint failure→ SQLAlchemy IntegrityError→ service reloads existing job→ route returns successful replay responseA database integrity error does not always mean the user’s operation failed. In this case it can mean another copy of the same request already succeeded.
The browser must retain the key across retries
Section titled “The browser must retain the key across retries”The most important browser rule is:
Generate one key per intended operation, not one key per HTTP attempt.
This would be wrong:
async function submitReview() { const key = crypto.randomUUID();
await fetch(url, { headers: { "Idempotency-Key": key, }, });}If submitReview() is called again after a network failure, it generates a new key.
The server reasonably interprets that as a new operation.
Instead, we store the pending operation:
let pendingOperation = null;When the user initiates a review:
function prepareOperation() { if (pendingOperation !== null) { return pendingOperation; }
pendingOperation = { key: crypto.randomUUID(), depth: depthElement.value, };
return pendingOperation;}The operation contains both:
key→ identity of this operation
depth→ original input belonging to the operationIf the request must be retried, both remain unchanged.
The updated browser request
Section titled “The updated browser request”The submit function becomes:
async function startReview(event) { event.preventDefault();
const operation = prepareOperation();
startButton.disabled = true; depthElement.disabled = true; resultSection.hidden = true; summaryElement.textContent = "";
messageElement.textContent = "Submitting review request…";
try { const response = await fetch( panel.dataset.startUrl, { method: "POST", headers: { "Content-Type": "application/json", "Accept": "application/json", "X-CSRF-Token": panel.dataset.csrfToken, "Idempotency-Key": operation.key, }, body: JSON.stringify({ depth: operation.depth, }), }, );
if (response.status === 409) { pendingOperation = null;
throw new Error( "The request conflicts with the current review state. Reload the page." ); }
if (response.status >= 400 && response.status < 500) { pendingOperation = null;
throw new Error( "The review request was rejected." ); }
if (!response.ok) { throw new Error( "The server could not confirm this operation." ); }
const data = await response.json();
pendingOperation = null; startButton.textContent = "Start review"; panel.dataset.eventsUrl = data.events_url;
statusElement.textContent = data.status; progressElement.textContent = `${data.progress}%`;
messageElement.textContent = data.replayed ? "Existing review recovered." : "Review queued.";
if ( data.status === "queued" || data.status === "processing" ) { openReviewEvents(data.events_url); } else { updateReviewDisplay(data); }
} catch (error) { if (pendingOperation !== null) { messageElement.textContent = "The outcome is uncertain. Retry will safely check the same operation.";
startButton.disabled = false; startButton.textContent = "Retry status"; depthElement.disabled = true;
return; }
messageElement.textContent = error.message;
startButton.textContent = "Start review"; setReviewControls(false); }}A network failure commonly appears as a TypeError, but it is not the only uncertain outcome. A server error can occur after a commit, and an accepted response can arrive with an unreadable body. We retain the pending key for all of those cases. A definite client rejection clears it; a readable acknowledgment clears it after parsing.
In that branch, we deliberately do not clear:
pendingOperationThe next form submission reuses:
operation.keyoperation.depthThe browser is retrying the same logical operation.
This example retains the pending key in page memory. Reloading discards it and reads the stored job through Jinja. Retaining the same key across a full browser restart would require persistent client storage or a server-side operation lookup; it is not a guarantee supplied by this variable.
Why the depth control remains disabled
Section titled “Why the depth control remains disabled”After the first uncertain attempt, we retain:
pendingOperation.depthSuppose the original request used:
standardThe user then changes the select field to:
thoroughand retries using the same key.
That would violate the operation’s meaning:
Same keyDifferent payloadTherefore, while an operation is uncertain:
depthElement.disabled = true;The browser preserves the original input.
If the user wants to initiate a genuinely different review later, the application creates a new idempotency key.
Following the lost-response path
Section titled “Following the lost-response path”Now let’s trace the exact failure.
First attempt
Section titled “First attempt”The browser creates:
Key: 8c043429-...Depth: standardIt sends:
POST /documents/101/reviews HTTP/1.1Cookie: app_session=...X-CSRF-Token: ...Idempotency-Key: 8c043429-...Content-Type: application/json
{"depth":"standard"}FastAPI:
- Authenticates the user.
- Authorizes access to document
101. - Verifies CSRF.
- Validates the header as a UUID.
- Validates the body as
ReviewStart. - Calls the service.
The service:
- Finds no existing job for this key.
- Changes the document to
queued. - Inserts job
410. - Stores the key on job
410. - Commits.
SQLite now contains:
job_id: 410document_id: 101requested_by: 27idempotency_key: 8c043429-...depth: standardstatus: queuedFastAPI attempts to return 202.
The network breaks.
What the browser knows
Section titled “What the browser knows”The browser knows:
I sent the request.I did not receive a usable response.It does not know:
Did the server receive it?Did validation pass?Did SQLite commit?Did the response merely get lost?The interface displays:
The outcome is uncertain. Retry will safely check the same operation.This is an honest error state.
The retry
Section titled “The retry”The user clicks again.
JavaScript reuses:
Key: 8c043429-...Depth: standardFastAPI authenticates, authorizes and validates the retry exactly as it did the first request.
The service queries SQLite and finds job 410.
It verifies:
Same userSame keySame documentSame depthThen it returns:
existing_job, TrueFastAPI responds:
HTTP/1.1 200 OKContent-Type: application/json
{ "job_id": 410, "document_id": 101, "status": "processing", "progress": 40, "replayed": true, "events_url": "http://localhost:8000/documents/101/review-events?job_id=410", "result_summary": null, "error_message": null}No second job is created.
The browser displays:
Existing review recovered.Then it opens the SSE connection and resumes normal progress updates.
Idempotency does not bypass authorization
Section titled “Idempotency does not bypass authorization”The service lookup includes:
ReviewJob.requested_by_user_id== requested_by_user_idBut that does not replace document authorization.
The route still receives:
document: AuthorizedDocumentbefore calling the service.
A user cannot submit someone else’s document ID and a guessed idempotency key to gain access.
Every retry is a new HTTP request:
New middleware execution→ new authentication check→ new authorization check→ new CSRF verification→ new Pydantic validationThe key answers only:
Have we already accepted this same logical operation from this user?
It does not answer:
Is this user currently permitted to perform it?
Idempotency does not mean caching forever
Section titled “Idempotency does not mean caching forever”The application needs a retention policy.
If review jobs remain permanently in the database, their keys remain available permanently.
If old jobs are archived or deleted, the application must decide how long retry protection should remain.
A typical principle is:
Keep idempotency records longer than the longest realistic client retry window.For our small application, keeping the key on the durable review job is simple.
Larger systems sometimes maintain a separate table containing:
User or tenantIdempotency keyRequest fingerprintHTTP statusResponse bodyCreation timeExpiration timeThat lets the server reproduce the original response even when the business object has moved or been archived.
The worker must also handle retries safely
Section titled “The worker must also handle retries safely”Protecting the POST prevents duplicate job creation.
It does not automatically make every step inside the worker safe.
Suppose a worker:
- Sends a document to an external review provider.
- Crashes before storing the provider’s result.
- Restarts and retries the job.
- Sends the same document again.
The external service may perform the work twice or charge twice.
A complete design may propagate a stable identifier downstream:
Our review job ID: 410→ external operation key: review-job-410The worker may also store:
external_request_idattempt_countlast_attempt_atIdempotency must be considered at every boundary where duplicate effects would matter.
For our current SQLite worker, the database job ID is the natural durable identity.
Disabled buttons are not enough
Section titled “Disabled buttons are not enough”Our browser already disables the Start button:
startButton.disabled = true;That prevents ordinary accidental double-clicks.
But it does not protect against:
- A network retry
- Two browser tabs
- A script sending requests directly
- A mobile browser resubmitting
- A proxy retry
- Two nearly simultaneous event handlers
- An impatient user reloading the page
Interface controls improve usability.
Database constraints protect data integrity.
The two layers have different responsibilities:
Disabled button→ reduce accidental duplicate interaction
Idempotency key→ identify repeated logical request
Unique constraint→ enforce the rule under concurrencyIdempotency versus duplicate-business rules
Section titled “Idempotency versus duplicate-business rules”Our application now has two related protections.
The first is the document state transition:
A document may have only one active review.The second is the operation identity:
The same submitted operation may be safely replayed.These produce different outcomes.
Same operation repeated
Section titled “Same operation repeated”Same userSame keySame documentSame depthResult:
Return existing job successfullyDifferent operation while review is active
Section titled “Different operation while review is active”Same documentDifferent keyResult:
409 ConflictThe first is a retry.
The second is a new business request that conflicts with current state.
That distinction allows the server to be both safe and helpful.
The complete lifecycle
Section titled “The complete lifecycle”For a successful first attempt:
- Jinja renders the document page and CSRF token.
- The browser constructs the DOM.
- The user submits the form.
- JavaScript creates an operation UUID.
- JavaScript freezes the selected depth.
JSON.stringify()creates the request body.fetch()sends the body, cookie, CSRF token and idempotency key.- Middleware reconstructs the session.
- FastAPI authenticates the user.
- The authorization dependency loads the permitted document.
- FastAPI parses the UUID header.
- Pydantic validates the JSON body.
- The route verifies CSRF.
- The service finds no matching prior job.
- The service atomically changes the document to
queued. - SQLAlchemy inserts the new
ReviewJob. - SQLite enforces the unique key constraint.
- SQLite commits.
- FastAPI returns
202 Accepted. - JavaScript parses the response.
- JavaScript clears the pending operation key.
- The browser opens the SSE stream.
- The worker and progress lifecycle continue normally.
For a lost response and retry:
- Steps 1–18 occur normally.
- The response disappears in transit.
- JavaScript receives a network error.
- JavaScript retains the original key and depth.
- The interface offers a safe retry.
- The browser resends the same logical operation.
- Middleware, authentication, authorization and CSRF run again.
- The service finds the existing job.
- The service verifies that the payload matches.
- No INSERT occurs.
- FastAPI returns
200 OKwith the existing job. - JavaScript displays its current status.
- SSE resumes progress reporting.
The central idea
Section titled “The central idea”A network error tells the browser:
I do not know what happened.It does not necessarily mean:
Nothing happened.The idempotency key gives one intended operation a stable identity across multiple HTTP attempts:
Logical operation→ one key
First HTTP attempt→ may succeed or disappear
Retry→ same key
Database→ returns the same durable jobThe full safety chain is:
Browser retains operation identity ↓HTTP carries Idempotency-Key ↓FastAPI validates the header ↓Service compares the intended operation ↓SQLAlchemy executes within a transaction ↓SQLite enforces uniqueness ↓Route returns new or existing job ↓Browser resumes from authoritative stateThis is one of the clearest examples of why a production application cannot treat the browser, HTTP response or visible DOM as the final truth.
The database records what actually happened. The idempotency key gives the browser a safe way to ask again.