cross-py-builder/DESIGN.md

Cross-Py-Builder — Controller + Web UI Design

Target architecture: keep the existing worker (build_agent.py) HTTP API as-is, and add a controller LXC that runs a web app. The controller stores every submitted request as a job row, polls workers to pick a target, dispatches the build, records download artifacts, and exposes a browser UI to submit, track progress, and download results.

The workers (build_agent.py) keep their HTTP API; only one small read-only endpoint (/progress/<job_id>) is added for live build progress. Everything else is new controller code.

---

1. Components

  Browser
     │  (HTTP, port 8080)
     ▼
  ┌───────────────────────────────── ─┐      ┌──────────────────────┐
  │  CONTROLLER LXC                   │      │  WORKER A (ubuntu x64)│
  │                                   │      │                      │
  │  Flask app (waitress)             │      │  build_agent.py      │
  │  ├─ web UI (submit/track/download)│      │  /upload /status     │
  │  ├─ job API                       │ ───► │  /download /checkout  │
  │  ├─ scheduler (string executor)   │      └──────────────────────┘
  │  └─ SQLite (jobs.db)              │      ┌──────────────────────┐
  │                                   │      │  WORKER B (win arm64)│
  └───────────────────────────────── ─┘      │  build_agent.py      │
                                             └──────────────────────┘
  ctrl/  controller package
  ctrl/static/  frontend (HTML/JS/CSS)
  ctrl/jobs.db   SQLite database (runtime)
  builds/        downloaded artifacts (runtime)

Worker discovery: static config (list of host:port) in the controller settings. On each job the controller queries every worker /status, filters to status=="ready" and matching CPU/OS, and picks one. This replaces Zeroconf on the controller; workers can stop advertising, and the Zeroconf code path can be removed with the CLI retirement (§7).

We deliberately do not rewrite the agent's HTTP API, and do not add auth in v1 (same trusted-LAN posture as today). The worker endpoint contract used by the controller:

Worker endpoint	Method	Purpose
/status	GET	readiness, os, cpu, agent_version, hostname, ip
/upload	POST	multipart file zip → starts build, returns job JSON
/checkout_git	POST	JSON {"repo_url": ...} → starts build
/download/<job_id>	GET	zip of dist/ for a built job
/delete_cache	GET	clear cached jobs
/healthz	GET	liveness (already added)
/progress/<job_id>	GET	NEW (small agent change): live build snapshot (status, current step, elapsed, last log line) for SSE relay

---

2. Data model

SQLite table jobs (one row per submitted build; single table keeps v1 simple):

column	type	notes
id	text PK	controller job id (e.g. JOB-<8 hex>) — distinct from worker job id
status	text	queued → dispatching → building → done | failed | cancelled
source_type	text	upload | git
source	text	uploaded filename or repo URL
os_req	text?	requested OS filter (nullable)
cpu_req	text?	requested CPU filter (nullable)
worker_host	text	worker chosen (host:port)
worker_job_id	text?	job id returned by the worker
worker_url	text?	base URL of worker (for download link)
created_at	text	ISO timestamp
started_at	text?
finished_at	text?
log	text	aggregated progress lines (appended, newline-delimited)
error	text?	last error detail
artifacts	text	JSON list of downloaded artifact paths/names

Artifacts are downloaded by the controller into builds/<job_id>/ (<app>-<version>-<os>-<cpu>.zip) so results survive worker container restarts and remain available after the worker cleans up.

---

3. REST API (controller)

Method	Path	Purpose
GET	/	web UI (single page)
GET	/api/workers	live /status for each configured worker
GET	/api/capabilities	distinct (os, cpu) across ready workers, for dropdowns
POST	/api/jobs	create job: multipart upload file OR JSON {repo_url, os, cpu}
GET	/api/jobs	list jobs (newest first, with status)
GET	/api/jobs/<id>	job detail incl. live log
POST	/api/jobs/<id>/cancel	set status → cancelled (best-effort)
GET	/api/jobs/<id>/artifacts/<name>	download a built artifact
GET	/api/jobs/<id>/stream	SSE log stream for live tail

---

4. Scheduler (dispatch loop)

A single background thread owns all dispatch work.

1. Pop the oldest job with status == "queued" (FIFO).
2. From worker config, query /status (short timeout). Filter:
   • status contains ready (not building/updating)
   • if os_req set, worker os contains os_req (case-insensitive)
   • if cpu_req set, worker cpu contains cpu_req
   • exclude the local/controller LXC if present, to avoid self-builds
3. Choose first match (optionally: prefer workers with more free disk — v1: first match).
4. Mark job dispatching; set worker_host.
5. Send to worker:
   • upload → POST /upload with the stored file
   • git → POST /checkout_git with {repo_url}
   • on success, read the worker job JSON → store worker_job_id, worker_url, set building
   • on hard failure, mark failed (§ failure handling below)
6. Progress relay — while the dispatch call is in flight (the worker is building), poll GET /progress/<worker_job_id> on an interval and append each snapshot to the job log + push it to connected SSE clients. This gives live per-step progress in the UI, enabled by the small agent addition (§1).
7. On success (dispatch call returns): compute download URL worker_url + /download/<worker_job_id>, fetch the zip, save to builds/<job_id>/, record artifact in the job row, set done.

Concurrency/limits: scheduler processes one job at a time (workers can already build only one at a time — 409). Multiple queued jobs simply wait. This makes NIC/disk behavior predictable and avoids the current CLI dumping N builds in parallel.

Failure handling: if dispatch fails (worker down/409/500), mark job failed and record error + log tail. No auto-retry in v1 (confirmed decision) — surface the error and let the user resubmit to an eligible worker.

---

5. Frontend (single-page, no build-step)

• Ask for /api/capabilities → render OS + CPU dropdowns (or "any").
• Upload a zip or paste a git URL; submit.
• Job list refreshes via periodic /api/jobs.
• Detail view: status badge + log (poll, or SSE stream). Download button appears when done.
• No npm/JS toolchain — vanilla JS + <link>ed CSS; keeps the controller LXC dependency-light.

---

6. Project layout (new/changed files)

setup.py                      (add ctrl package + entry point `cross-py-controller`)
requirements.txt              (add waitress; keep Flask/requests)
cross_py_builder/             (agent unchanged except new /progress endpoint)
ctrl/
  __init__.py
  app.py           (Flask factory, routes, waitress runner)
  db.py            (SQLite init + helpers)
  scheduler.py     (dispatch + progress-poll loop, worker client)
  workers.py       (worker config load + /status probe)
  settings.py      (env-driven config: DB path, builds dir, worker list)
  static/
    index.html
    app.js
    style.css

Worker config example (env or JSON file):

CROSS_PY_WORKERS='["10.0.0.11:9001","10.0.0.12:9001"]'
CROSS_PY_DATA=/var/lib/cross-py-controller   # holds jobs.db + builds/
CROSS_PY_PORT=8080

---

7. What we reuse vs. retire

• Reuse on workers: agent HTTP API, plus the new /progress/<job_id> endpoint.
• Retire the CLI entirely (confirmed decision): agent_manager.py is removed. The controller web UI is the sole frontend. This drops the Zeroconf-dependent manager code paths (multi-parallel submits, hardcoded DEFAULT_PORT, the self-update flow) and leaves a single interface to maintain.

8. Build environment model (per-build venv)

Workers keep creating a fresh venv and installing requirements per job. This is the isolated model — no cross-job env contamination, and a bad or malicious requirements.txt can't poison a shared environment. Tradeoff: each job pays ~2-5 min of pip installs (PyInstaller + deps) and repeats downloads/disk churn.

Mitigation ladder, if wall-clock ever hurts (preserve isolation in all cases):

1. Persistent pip-cache volume on the worker (~/.cache/pip) — fast, no isolation loss.
2. Local wheel mirror / --find-links — same, plus offline-friendly.
3. Prebuilt PyInstaller baked into the worker LXC snapshot (skip its reinstall per job).

Do not move to a shared persistent venv until a concrete cross-job dependency problem appears.

9. Deferred / next-phase (explicitly out of scope for v1)

• Auth / tokens on controller and worker endpoints
• Worker add/remove/update/restart/shutdown via the UI
• Auto-retry on a different worker; retry-with-backoff for queued jobs
• Persisting/streaming full-build logs from worker (build-<name>.log is on the worker today)
• Storing source zip centrally for re-runs
• Auto-scaling/provisioning of worker LXCs