cross-py-builder/DESIGN.md

# Cross-Py-Builder — Controller + Web UI Design

Target architecture: keep the existing worker (`build_agent.py`) HTTP API as-is, and add a
**controller** LXC that runs a web app. The controller stores every submitted request as a
**job row**, polls workers to pick a target, dispatches the build, records download artifacts,
and exposes a browser UI to submit, track progress, and download results.

The workers (`build_agent.py`) keep their HTTP API; only one small read-only endpoint
(`/progress/<job_id>`) is added for live build progress. Everything else is new controller code.

---

## 1. Components

```
  Browser
     │  (HTTP, port 8080)
     ▼
  ┌───────────────────────────────── ─┐      ┌──────────────────────┐
  │  CONTROLLER LXC                   │      │  WORKER A (ubuntu x64)│
  │                                   │      │                      │
  │  Flask app (waitress)             │      │  build_agent.py      │
  │  ├─ web UI (submit/track/download)│      │  /upload /status     │
  │  ├─ job API                       │ ───► │  /download /checkout  │
  │  ├─ scheduler (string executor)   │      └──────────────────────┘
  │  └─ SQLite (jobs.db)              │      ┌──────────────────────┐
  │                                   │      │  WORKER B (win arm64)│
  └───────────────────────────────── ─┘      │  build_agent.py      │
                                             └──────────────────────┘
  ctrl/  controller package
  ctrl/static/  frontend (HTML/JS/CSS)
  ctrl/jobs.db   SQLite database (runtime)
  builds/        downloaded artifacts (runtime)
```

**Worker discovery:** configurable via the web UI and persisted in the controller's SQLite
`workers` table (host + port per agent). On each job the controller queries every worker
`/status`, filters to `status=="ready"` and matching CPU/OS, and picks one. This replaces
Zeroconf on the controller; workers can stop advertising, and the Zeroconf code path can be
removed with the CLI retirement (§7).

**Worker management API** (adds/removes agents without touching config files):

| Endpoint | Method | Purpose |
|---|---|---|
| `GET /api/workers` | GET | live `/status` for each configured worker |
| `POST /api/workers` | POST | add a worker: JSON `{"spec": "host:port"}` (or `host`+`port`) |
| `DELETE /api/workers/<port>/<host>` | DELETE | remove a configured worker |

The workers table is seeded on demand from the UI; there is no `CROSS_PY_WORKERS`
environment bootstrap — the DB is the single source of truth.

We deliberately do **not** rewrite the agent's HTTP API, and do **not** add auth in v1 (same
trusted-LAN posture as today). The worker endpoint contract used by the controller:

| Worker endpoint | Method | Purpose |
|---|---|---|
| `/status` | GET | readiness, os, cpu, agent_version, hostname, ip |
| `/upload` | POST | multipart `file` zip → starts build, returns job JSON. `?async=1` returns `202` + `{"id": ...}` immediately, building in background |
| `/checkout_git` | POST | JSON `{"repo_url": ...}` → starts build. `?async=1` behaves like `/upload?async=1` |
| `/download/<job_id>` | GET | zip of `dist/` for a built job |
| `/delete_cache` | GET | clear cached jobs |
| `/healthz` | GET | liveness (already added) |
| `/progress/<job_id>` | GET | **NEW (small agent change):** live build snapshot (status, current step, elapsed, last log line) for SSE relay |

---

## 2. Data model

**SQLite table `jobs`** (one row per submitted build; single table keeps v1 simple):

| column | type | notes |
|---|---|---|
| `id` | text PK | controller job id (e.g. `JOB-<8 hex>`) — distinct from worker job id |
| `status` | text | `queued → dispatching → building → done \| failed \| cancelled` |
| `source_type` | text | `upload` \| `git` |
| `source` | text | uploaded filename or repo URL |
| `os_req` | text? | requested OS filter (nullable) |
| `cpu_req` | text? | requested CPU filter (nullable) |
| `worker_host` | text | worker chosen (host:port) |
| `worker_job_id` | text? | job id returned by the worker |
| `worker_url` | text? | base URL of worker (for download link) |
| `created_at` | text | ISO timestamp |
| `started_at` | text? | |
| `finished_at` | text? | |
| `log` | text | aggregated progress lines (appended, newline-delimited) |
| `error` | text? | last error detail |
| `artifacts` | text | JSON list of downloaded artifact paths/names |

Artifacts are downloaded by the controller into `builds/<job_id>/` (`<app>-<version>-<os>-<cpu>.zip`)
so results survive worker container restarts and remain available after the worker cleans up.

---

## 3. REST API (controller)

| Method | Path | Purpose |
|---|---|---|
| GET | `/` | web UI (single page) |
| GET | `/api/workers` | live `/status` for each configured worker |
| POST | `/api/workers` | add a worker: JSON `{"spec": "host:port"}` |
| DELETE | `/api/workers/<port>/<host>` | remove a configured worker |
| GET | `/api/capabilities` | distinct (os, cpu) across ready workers, for dropdowns |
| POST | `/api/jobs` | create job: multipart upload `file` OR JSON `{repo_url, os, cpu}` |
| GET | `/api/jobs` | list jobs (newest first, with status) |
| GET | `/api/jobs/<id>` | job detail incl. live `log` |
| POST | `/api/jobs/<id>/cancel` | set status → `cancelled` (best-effort) |
| GET | `/api/jobs/<id>/artifacts/<name>` | download a built artifact |
| GET | `/api/jobs/<id>/stream` | SSE log stream for live tail |

---

## 4. Scheduler (dispatch loop)

A single background thread owns all dispatch work.

1. Pop the oldest job with `status == "queued"` (FIFO).
2. From worker config, query `/status` (short timeout). Filter:
   - `status` contains `ready` (not building/updating)
   - if `os_req` set, worker `os` contains `os_req` (case-insensitive)
   - if `cpu_req` set, worker `cpu` contains `cpu_req`
   - exclude the local/controller LXC if present, to avoid self-builds
3. Choose first match (optionally: prefer workers with more free disk — v1: first match).
4. Mark job `dispatching`; set `worker_host`.
 5. Send to worker (**async**): `POST /upload?async=1` (with the stored file) or
   `POST /checkout_git?async=1` (with `{repo_url}`). The agent clones/saves the source, then
   starts the build in a background thread (holding its `build_lock`) and returns `202`
   immediately with `{"id": <worker_job_id>}`. On hard failure, mark `failed` (§ failure handling
   below). Store `worker_job_id`, `worker_url`, set `building`.
 6. **Progress relay** — while the build runs, poll `GET /progress/<worker_job_id>` on an interval,
   appending each `last_log_line`/step snapshot to the job log and pushing to connected SSE
   clients → live per-step progress in the UI. The build is complete when `/progress` returns
   `404` (progress cleared) and the worker's `/status` reports `ready`. (This relies on the small
   agent additions in §1.)
 7. On completion: compute download URL `worker_url + /download/<worker_job_id>`, fetch the zip,
   save to `builds/<job_id>/`, record artifact in the job row, set `done`.

**Concurrency/limits:** scheduler processes one job at a time (workers can already build only one
at a time — `409`). Multiple queued jobs simply wait. This makes NIC/disk behavior predictable
and avoids the current CLI dumping N builds in parallel.

**Failure handling:** if dispatch fails (worker down/409/500), mark job `failed` and record
`error` + log tail. No auto-retry in v1 (confirmed decision) — surface the error and let the
user resubmit to an eligible worker.

---

## 5. Frontend (single-page, no build-step)

- Ask for `/api/capabilities` → render OS + CPU dropdowns (or "any").
- Upload a zip **or** paste a git URL; submit.
- Job list refreshes via periodic `/api/jobs`.
- Detail view: status badge + log (poll, or SSE stream). Download button appears when `done`.
- No npm/JS toolchain — vanilla JS + `<link>`ed CSS; keeps the controller LXC dependency-light.

---

## 6. Project layout (new/changed files)

```
setup.py                      (add ctrl package + entry point `cross-py-controller`)
requirements.txt              (add waitress; keep Flask/requests)
agent/                    (worker agent: build_agent.py, zeroconf_server.py)
ctrl/
  __init__.py
  app.py           (Flask factory, routes, waitress runner)
  db.py            (SQLite init + helpers; jobs + workers tables)
  scheduler.py     (dispatch + progress-poll loop, worker client)
  workers.py       (worker config load + /status probe)
  settings.py      (env-driven config: DB path, builds dir, worker list)
  static/
    index.html
    app.js
    style.css
```

Controller config example (env or JSON file):
```
CROSS_PY_DATA=/var/lib/cross-py-controller   # holds jobs.db (jobs + workers) + builds/
CROSS_PY_PORT=8080
```

Workers are added/removed in the web UI and stored in the SQLite `workers` table (no
`CROSS_PY_WORKERS` env var).

---

## 7. What we reuse vs. retire

- **Reuse on workers:** agent HTTP API, plus the new `/progress/<job_id>` endpoint.
- **Retire the CLI entirely (confirmed decision):** `agent_manager.py` is removed. The
  controller web UI is the sole frontend. This drops the Zeroconf-dependent manager code paths
  (multi-parallel submits, hardcoded `DEFAULT_PORT`, the self-update flow) and leaves a single
  interface to maintain.

## 8. Build environment model (per-build venv)

Workers keep creating a fresh venv and installing requirements per job. This is the isolated
model — no cross-job env contamination, and a bad or malicious `requirements.txt` can't poison
a shared environment. Tradeoff: each job pays ~2-5 min of pip installs (PyInstaller + deps)
and repeats downloads/disk churn.

Mitigation ladder, if wall-clock ever hurts (preserve isolation in all cases):
1. Persistent pip-cache volume on the worker (`~/.cache/pip`) — fast, no isolation loss.
2. Local wheel mirror / `--find-links` — same, plus offline-friendly.
3. Prebuilt PyInstaller baked into the worker LXC snapshot (skip its reinstall per job).

Do **not** move to a shared persistent venv until a concrete cross-job dependency problem
appears.

## 9. Deferred / next-phase (explicitly out of scope for v1)

- Auth / tokens on controller and worker endpoints
- Worker add/remove/update/restart/shutdown via the UI
- Auto-retry on a different worker; retry-with-backoff for queued jobs
- Persisting/streaming full-build logs from worker (`build-<name>.log` is on the worker today)
- Storing source zip centrally for re-runs
- Auto-scaling/provisioning of worker LXCs