Feasibility solve flow — async job + poll

In simple terms

The feasibility stage solves a whole year of hourly operation as one big optimisation. That takes about 13 seconds — far too long to make the browser wait on a single request. So the app runs it as a background job: the “solve” click returns immediately, the room polls every couple of seconds until the job is done, and then reads the results. The heavy result (8,760 hours × ~20 numbers) is saved once to disk; every chart is derived from that file on demand.

Two modes, one path. This async flow describes the Solver (LP) mode (~13 s). The Rule-based mode runs through the same POST /feasibility/solve route and job, but the myopic pass finishes in a fraction of a second — so the first status poll already returns done. The result is persisted to the same single dispatch file, stamped with meta.engine so every room knows which mode produced it. See Feasibility suite for the mode difference.

Why it’s asynchronous

A 13-second synchronous request would block the UI, risk gateway timeouts, and lose the work if the page reloaded. A background thread-job decouples “start the work” from “read the result”, so the room can show progress, survive a reload (re-attaching to the running job), and surface solver errors instead of hanging.

The flow

sequenceDiagram
  participant R as Feasibility room
  participant C as client.ts
  participant A as app.py
  participant P as PtxApi
  participant Q as solve queue
  participant T as worker thread
  participant D as Parquet + JSON sidecar

  R->>C: startFeasibilitySolve(projectId)
  C->>A: POST /feasibility/solve
  A->>P: start_feasibility_solve
  P->>Q: submit the job
  Q->>T: a free worker starts it (status = running)
  P-->>R: { status: "running" } or { status: "queued", position, ahead }  (returns at once)
  loop every ~2s
    R->>C: getFeasibilityStatus(projectId)
    C->>A: GET /feasibility
    A->>P: feasibility_status
    P-->>R: { status: queued | running | done | error }
  end
  T->>T: build PyPSA network → HiGHS solve
  T->>D: save_dispatch(frame + sidecar)  [SOLVE, once]
  T->>P: status = done
  R->>C: getFeasibilityDispatch(res=day) etc.
  C->>A: GET /feasibility/dispatch, /duration, /series …
  A->>P: derive on read  [POST-PROCESS]
  P-->>R: envelopes / curves / heatmap

Step by step

Start (POST /feasibility/solve). start_feasibility_solve records a job and hands it to the solve queue (core/solve_queue.py), which starts it at once in a worker thread when one of the server’s workers is free (PTX_SOLVE_WORKERS, default 2) and otherwise keeps it waiting its turn — reported queued with position and ahead — and returns the job state immediately. The thread acts as the user who asked. A second start while one runs restarts it: a new token supersedes the running thread, whose result is discarded; while it still waits, the restart keeps its place in line. Who waits for what, and the rule of fairness between workspaces: the feasibility suite, “When the server is busy”.
The worker (thread). solve_feasibility rebuilds the config, resolves the hourly frame (_feasibility_frame: dataset curve › override-rescaled › flat loose-CF › refuse with a 422 naming the carrier), resolves the book into params, and calls run_feasibility_from_config — which builds the PyPSA network and solves it with HiGHS (IPM + crossover). Errors are captured onto the job (status = error, message kept), never lost in the thread.
SOLVE persists once. save_dispatch writes the hourly frame to data/results/<project_id>/dispatch.parquet (float32, ~20 series + a binding-constraint column) plus a JSON sidecar (annual aggregates, capacity duals, solver metadata). The solve also stamps the sizing it consumed (meta.sizing) so the rooms can later warn “this result is stale — you changed a size since.” The controller returns only annual, duals, meta — never the raw hours.
Poll (GET /feasibility). The room polls feasibility_status every ~2 s. While the job is queued or running, the status route deliberately does not read the store (no racing the writer, no re-reading the frame for nothing). On done it returns the persisted summary; on error it returns the message.
Reload safety. Because the job lives on PtxApi and the result lives on disk, reloading the page mid-solve re-attaches to the running job (or reads the finished frame) instead of starting over.
POST-PROCESS on read. Once done, the charts fetch derived views, each a cheap post-process of the one stored frame:
- GET /feasibility/dispatch?res=day → 365 min/mean/max envelopes per series (or res=hour&start=&end= for a full-res window ≤ 21 days).
- GET /feasibility/duration?series=&points= → descending quantile samples.
- GET /feasibility/series?name= → one full-res series (the calendar heatmap).
- /reliability, /monthly, /sweep, curtailment-by-cause — all derived from the same frame. WACC or other post-process-only changes never re-solve.

The SOLVE / POST-PROCESS line (why this is fast to explore)

The expensive optimisation runs once and is frozen to disk. Everything the Feasibility suite shows — dispatch, duration curve, SoC traces, heatmap, reliability episodes, monthly table, storage-value duals — is a pure function of that frame, computed on read. That’s why panning charts and switching sub-rooms is instant even though the solve took 13 seconds. See System architecture and Data model.

Failure & edge cases

Missing hourly inputs: a sized carrier with neither a linked dataset nor a CF raises ValueError → 422 naming the carrier (“link a Site dataset or set a capacity factor”).
Solver error: captured on the job and surfaced via status — the room shows it instead of spinning forever; the worker is freed for the next job.
Every worker busy: the job waits, queued with its place, and starts on its own; Stop takes it out of the line before it starts.
Stale result: if you change a size after solving, the stamped meta.sizing differs from the live sizing fingerprint and the room shows a staleness banner (see the storage lab / dispatch rooms).
Snapshots: GET /feasibility?snapshot_id= serves the frozen summary for a captured scenario (job state is irrelevant there).

How to modify this safely

Keep the SOLVE/POST-PROCESS line intact: persist the frame once; derive new views on read rather than storing more aggregates.
New derived chart → a new post-process read route + an api.ts type; don’t widen the solve response with raw hours.
Preserve the “don’t read the store while running” rule in the poller to avoid racing the writer.

From the app’s documentation, docs/01-architecture/feasibility-solve-flow.md · Hyprisma 1.0.5