Celery and FastAPI: answering before the work is done

The need was ordinary: receive photographs of receipts, extract the lines, return something usable. The first prototype did it all inside the request. It held while the images were small and the traffic light, which is to say until the day it went live. An extraction takes several seconds; an HTTP request that lasts several seconds holds a worker, and a hundred simultaneous uploads is enough for the server to stop answering: including the requests that had asked for nothing heavy.

FastAPI offers `BackgroundTasks`, and it is the most tempting trap. The task does start after the response, but in the same process: it competes for the processor with live requests, and if the process stops (a deployment, a restart, memory pressure) the work is lost without trace. That is acceptable for sending an email, never for processing you have promised a user.

The server accepts, drops a message, and hands back control. A separate worker consumes the queue. The essential part is a rule everybody forgets the first time: **the image does not travel through the queue**. A broker carries instructions, not megabytes: write the image to object storage and pass only its key.

from celery import Celery
from fastapi import FastAPI, UploadFile
from fastapi.responses import JSONResponse

celery = Celery("receipts", broker="redis://redis:6379/0", backend="redis://redis:6379/1")
api = FastAPI()

@api.post("/receipts", status_code=202)
async def submit(upload: UploadFile) -> JSONResponse:
    # The image goes to object storage; the queue only ever gets a key.
    key = await store(upload)
    task = extract.delay(key)
    # 202 and not 200: the request is accepted, it is not fulfilled.
    return JSONResponse({"id": task.id}, status_code=202)

@api.get("/receipts/{task_id}")
def status(task_id: str) -> dict:
    r = celery.AsyncResult(task_id)
    return {"state": r.state, "result": r.result if r.successful() else None}

Celery’s defaults are calibrated for tasks that are short and numerous. Image processing is the opposite: long and expensive. Three of them turn against you, and they are better changed before the incident than during it.

celery.conf.update(
    # Defaults to 4: a worker reserves four messages ahead and holds them while
    # processing one. On long tasks, three images wait behind the current one
    # while another worker sits idle.
    worker_prefetch_multiplier=1,
    # Defaults to False: the message is acknowledged on receipt, so it is lost
    # if the worker dies midway. At True it is only acknowledged at the end —
    # at the cost of a possibly repeated execution.
    task_acks_late=True,
    task_reject_on_worker_lost=True,
    # With no limit, one pathological image holds a worker indefinitely. The
    # soft limit raises a catchable exception; the hard one kills.
    task_soft_time_limit=110,
    task_time_limit=120,
)

The third setting has a consequence to be owned rather than discovered: from the moment a message can be replayed, the task must be replayable too. Writing the result under a key derived from the input rather than by appending, and checking before processing, costs three lines and avoids a duplicate in a client report.

I say nothing about the choice of broker. Redis holds the role perfectly well as long as its delivery promise is accepted, weaker as it is than a transactional broker’s; past a certain stake the question reopens, and it reopens with arguments I did not have to settle here. I also leave out progress reporting: the percentage moving while the user waits. It is feasible and it is nearly always wasted work: what somebody who uploaded an image wants is to know when to come back, not to watch a bar.

• Celery · configuration
• FastAPI · background tasks