Stream live worker progress via async build dispatch

The agent's /upload and /checkout_git were synchronous, blocking until a
build finished, so the controller could not relay live per-step progress.
Add ?async=1 support: the agent saves/clones the source, starts the build in
a background thread (holding build_lock), and returns 202 + {"id": ...}
immediately. The controller dispatches async, then polls /progress/<job_id>,
appending each step/log line to the job log and SSE stream, and treats the
build as done when progress clears (404) and the worker reports ready.

DESIGN.md

@@ -44,8 +44,8 @@ trusted-LAN posture as today). The worker endpoint contract used by the controll
 | Worker endpoint | Method | Purpose |
 |---|---|---|
 | `/status` | GET | readiness, os, cpu, agent_version, hostname, ip |
-| `/upload` | POST | multipart `file` zip → starts build, returns job JSON |
-| `/checkout_git` | POST | JSON `{"repo_url": ...}` → starts build |
+| `/upload` | POST | multipart `file` zip → starts build, returns job JSON. `?async=1` returns `202` + `{"id": ...}` immediately, building in background |
+| `/checkout_git` | POST | JSON `{"repo_url": ...}` → starts build. `?async=1` behaves like `/upload?async=1` |
 | `/download/<job_id>` | GET | zip of `dist/` for a built job |
 | `/delete_cache` | GET | clear cached jobs |
 | `/healthz` | GET | liveness (already added) |
@@ -108,17 +108,18 @@ A single background thread owns all dispatch work.
    - exclude the local/controller LXC if present, to avoid self-builds
 3. Choose first match (optionally: prefer workers with more free disk — v1: first match).
 4. Mark job `dispatching`; set `worker_host`.
-5. Send to worker:
-   - upload → `POST /upload` with the stored file
-   - git → `POST /checkout_git` with `{repo_url}`
-   - on success, read the worker job JSON → store `worker_job_id`, `worker_url`, set `building`
-   - on hard failure, mark `failed` (§ failure handling below)
-6. **Progress relay** — while the dispatch call is in flight (the worker is building), poll
-   `GET /progress/<worker_job_id>` on an interval and append each snapshot to the job log +
-   push it to connected SSE clients. This gives live per-step progress in the UI, enabled by
-   the small agent addition (§1).
-7. On success (dispatch call returns): compute download URL `worker_url + /download/<worker_job_id>`,
-   fetch the zip, save to `builds/<job_id>/`, record artifact in the job row, set `done`.
+ 5. Send to worker (**async**): `POST /upload?async=1` (with the stored file) or
+   `POST /checkout_git?async=1` (with `{repo_url}`). The agent clones/saves the source, then
+   starts the build in a background thread (holding its `build_lock`) and returns `202`
+   immediately with `{"id": <worker_job_id>}`. On hard failure, mark `failed` (§ failure handling
+   below). Store `worker_job_id`, `worker_url`, set `building`.
+ 6. **Progress relay** — while the build runs, poll `GET /progress/<worker_job_id>` on an interval,
+   appending each `last_log_line`/step snapshot to the job log and pushing to connected SSE
+   clients → live per-step progress in the UI. The build is complete when `/progress` returns
+   `404` (progress cleared) and the worker's `/status` reports `ready`. (This relies on the small
+   agent additions in §1.)
+ 7. On completion: compute download URL `worker_url + /download/<worker_job_id>`, fetch the zip,
+   save to `builds/<job_id>/`, record artifact in the job row, set `done`.
 
 **Concurrency/limits:** scheduler processes one job at a time (workers can already build only one
 at a time — `409`). Multiple queued jobs simply wait. This makes NIC/disk behavior predictable