diff --git a/DESIGN.md b/DESIGN.md index 5a1cbb9..d7f2cc9 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -33,10 +33,22 @@ The workers (`build_agent.py`) keep their HTTP API; only one small read-only end builds/ downloaded artifacts (runtime) ``` -**Worker discovery:** static config (list of `host:port`) in the controller settings. On each -job the controller queries every worker `/status`, filters to `status=="ready"` and matching -CPU/OS, and picks one. This replaces Zeroconf on the controller; workers can stop advertising, -and the Zeroconf code path can be removed with the CLI retirement (§7). +**Worker discovery:** configurable via the web UI and persisted in the controller's SQLite +`workers` table (host + port per agent). On each job the controller queries every worker +`/status`, filters to `status=="ready"` and matching CPU/OS, and picks one. This replaces +Zeroconf on the controller; workers can stop advertising, and the Zeroconf code path can be +removed with the CLI retirement (§7). + +**Worker management API** (adds/removes agents without touching config files): + +| Endpoint | Method | Purpose | +|---|---|---| +| `GET /api/workers` | GET | live `/status` for each configured worker | +| `POST /api/workers` | POST | add a worker: JSON `{"spec": "host:port"}` (or `host`+`port`) | +| `DELETE /api/workers//` | DELETE | remove a configured worker | + +The workers table is seeded on demand from the UI; there is no `CROSS_PY_WORKERS` +environment bootstrap — the DB is the single source of truth. We deliberately do **not** rewrite the agent's HTTP API, and do **not** add auth in v1 (same trusted-LAN posture as today). The worker endpoint contract used by the controller: @@ -86,6 +98,8 @@ so results survive worker container restarts and remain available after the work |---|---|---| | GET | `/` | web UI (single page) | | GET | `/api/workers` | live `/status` for each configured worker | +| POST | `/api/workers` | add a worker: JSON `{"spec": "host:port"}` | +| DELETE | `/api/workers//` | remove a configured worker | | GET | `/api/capabilities` | distinct (os, cpu) across ready workers, for dropdowns | | POST | `/api/jobs` | create job: multipart upload `file` OR JSON `{repo_url, os, cpu}` | | GET | `/api/jobs` | list jobs (newest first, with status) | @@ -150,7 +164,7 @@ agent/ (worker agent: build_agent.py, zeroconf_server.py) ctrl/ __init__.py app.py (Flask factory, routes, waitress runner) - db.py (SQLite init + helpers) + db.py (SQLite init + helpers; jobs + workers tables) scheduler.py (dispatch + progress-poll loop, worker client) workers.py (worker config load + /status probe) settings.py (env-driven config: DB path, builds dir, worker list) @@ -160,13 +174,15 @@ ctrl/ style.css ``` -Worker config example (env or JSON file): +Controller config example (env or JSON file): ``` -CROSS_PY_WORKERS='["10.0.0.11:9001","10.0.0.12:9001"]' -CROSS_PY_DATA=/var/lib/cross-py-controller # holds jobs.db + builds/ +CROSS_PY_DATA=/var/lib/cross-py-controller # holds jobs.db (jobs + workers) + builds/ CROSS_PY_PORT=8080 ``` +Workers are added/removed in the web UI and stored in the SQLite `workers` table (no +`CROSS_PY_WORKERS` env var). + --- ## 7. What we reuse vs. retire diff --git a/README.md b/README.md index 13071ab..9ae6e68 100644 --- a/README.md +++ b/README.md @@ -80,24 +80,25 @@ source venv/bin/activate pip install -e . ``` -Set the static worker list (JSON array of `host:port`, matching where your agents are -listening) and start it: +Set the port (defaults to `8080`) and start it: ```sh -CROSS_PY_WORKERS='["192.168.1.40:9001","192.168.1.41:9001"]' \ -cross-py-controller +CROSS_PY_PORT=8080 cross-py-controller ``` -Then open `http://:8080` in a browser. +Then open `http://:8080` in a browser and add your agents from the +**Workers** section of the UI (in `host:port` form, e.g. `192.168.1.40:9001`). Each +row shows live status (ready/down) and the agent's OS/CPU/version. Configured workers +persist in the controller's SQLite DB, so they survive restarts — no environment +variables needed after setup. ### Controller environment variables | Variable | Default | Purpose | |---|---|---| -| `CROSS_PY_WORKERS` | (none) | JSON array of agent `host:port` strings | | `CROSS_PY_PORT` | `8080` | HTTP port the controller UI listens on | | `CROSS_PY_MAX_CONCURRENT` | `1` | Max simultaneous builds (across distinct agents) | -| `CROSS_PY_DATA` | `/data` | Data dir (jobs DB, uploaded sources) | +| `CROSS_PY_DATA` | `/data` | Data dir (SQLite jobs + workers DB, uploaded sources) | | `CROSS_PY_BUILDS` | `/builds` | Dir where finished build artifacts are saved | | `CROSS_PY_DEBUG` | off | Use Flask dev server instead of waitress | @@ -105,12 +106,14 @@ Then open `http://:8080` in a browser. ## Using the UI -1. Open the controller in a browser. -2. Pick the target **OS** and **CPU** (or "any"). -3. Either upload a `.zip` of your project (PyInstaller spec + source) or paste a git +1. **Add your agents** in the Workers section (`host:port`). Each shows live status + (ready/down) and OS/CPU once reachable. +2. Open the controller in a browser. +3. Pick the target **OS** and **CPU** (or "any"). +4. Either upload a `.zip` of your project (PyInstaller spec + source) or paste a git `repo_url`. -4. Submit. The job is queued → dispatched to a matching agent → built → artifacts fetched. -5. Watch live per-step progress, then download the built binaries from the job detail. +5. Submit. The job is queued → dispatched to a matching agent → built → artifacts fetched. +6. Watch live per-step progress, then download the built binaries from the job detail. A submitted project only needs the source plus at least one `*.spec` file; each spec file in the archive produces a build. diff --git a/ctrl/app.py b/ctrl/app.py index d16e1d9..2e8f297 100644 --- a/ctrl/app.py +++ b/ctrl/app.py @@ -46,6 +46,36 @@ def api_workers(): return jsonify(wrk.probe_all()) +@app.post("/api/workers") +def api_add_worker(): + data = request.get_json(silent=True) or {} + spec = str(data.get("spec") or "").strip() + host = str(data.get("host") or "").strip() + port = data.get("port") + try: + if spec: + w = wrk.parse_worker(spec) + elif host and port: + w = wrk.parse_worker(f"{host}:{int(port)}") + else: + return jsonify({"error": "Provide spec ('host:port') or host+port"}), 400 + except (ValueError, TypeError) as e: + return jsonify({"error": str(e)}), 400 + + db.add_worker(w["host"], w["port"]) + db.set_queued_jobs_for_worker(w["url"]) + log.info("Added worker %s:%s", w["host"], w["port"]) + status = wrk.probe_worker(w["url"]) + return jsonify({"host": w["host"], "port": w["port"], "online": status is not None, "status": status}), 201 + + +@app.delete("/api/workers//") +def api_remove_worker(host, port): + db.remove_worker(host, port) + log.info("Removed worker %s:%s", host, port) + return jsonify({"removed": True, "host": host, "port": port}), 200 + + @app.get("/api/capabilities") def api_capabilities(): combos = { @@ -161,7 +191,7 @@ def main(): _ensure_ready() scheduler.start() _configure_logging() - log.info("Controller listening on 0.0.0.0:%s (%d workers configured)", settings.PORT, len(settings.WORKERS)) + log.info("Controller listening on 0.0.0.0:%s (%d workers configured)", settings.PORT, len(db.list_workers())) if settings.DEBUG: app.run(host="0.0.0.0", port=settings.PORT, threaded=True) return diff --git a/ctrl/db.py b/ctrl/db.py index f7f53b3..44d641d 100644 --- a/ctrl/db.py +++ b/ctrl/db.py @@ -41,6 +41,39 @@ def init_db(): ) """ ) + conn.execute( + """ + CREATE TABLE IF NOT EXISTS workers ( + host TEXT NOT NULL, + port INTEGER NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (host, port) + ) + """ + ) + conn.commit() + + +# ---- workers ------------------------------------------------------------- + +def list_workers(): + rows = _conn().execute("SELECT host, port FROM workers ORDER BY host, port").fetchall() + return [{"host": r["host"], "port": r["port"]} for r in rows] + + +def add_worker(host, port): + import datetime + conn = _conn() + conn.execute( + "INSERT OR IGNORE INTO workers (host, port, created_at) VALUES (?, ?, ?)", + (host, int(port), datetime.datetime.now().isoformat()), + ) + conn.commit() + + +def remove_worker(host, port): + conn = _conn() + conn.execute("DELETE FROM workers WHERE host = ? AND port = ?", (host, int(port))) conn.commit() diff --git a/ctrl/settings.py b/ctrl/settings.py index 8854140..560d1fa 100644 --- a/ctrl/settings.py +++ b/ctrl/settings.py @@ -1,4 +1,3 @@ -import json import os @@ -9,19 +8,6 @@ def _get_bool(name, default=False): return val.strip().lower() in ("1", "true", "yes", "on") -def _get_json_list(name, default=None): - raw = os.environ.get(name) - if not raw: - return list(default or []) - try: - value = json.loads(raw) - except json.JSONDecodeError: - raise ValueError(f"{name} must be a JSON array string") - if not isinstance(value, list): - raise ValueError(f"{name} must be a JSON array") - return value - - def _get_int(name, default): raw = os.environ.get(name) if raw is None: @@ -34,9 +20,6 @@ DB_PATH = os.environ.get("CROSS_PY_DB", os.path.join(DATA_DIR, "jobs.db")) BUILDS_DIR = os.environ.get("CROSS_PY_BUILDS", os.path.join(DATA_DIR, "builds")) PORT = _get_int("CROSS_PY_PORT", 8080) -# Static worker list: JSON array of "host:port" strings. -WORKERS = _get_json_list("CROSS_PY_WORKERS") - # One job at a time by default; raise to allow parallel dispatches to distinct workers. MAX_CONCURRENT = _get_int("CROSS_PY_MAX_CONCURRENT", 1) diff --git a/ctrl/static/app.js b/ctrl/static/app.js index d031d0c..dc53493 100644 --- a/ctrl/static/app.js +++ b/ctrl/static/app.js @@ -50,15 +50,71 @@ async function refreshWorkers() { const workers = await api("/api/workers"); const online = workers.filter((w) => w.online).length; const parts = workers.map((w) => - `${w.host || w.url}${w.online ? "" : " (down)"}` + `${w.url}${w.online ? "" : " (down)"}` ); $("#worker-summary").textContent = - `${online}/${workers.length} workers online — ${parts.join(" · ")}`; + `${online}/${workers.length} workers online` + (parts.length ? ` — ${parts.join(" · ")}` : ""); + + const box = $("#workers"); + if (!workers.length) { + box.innerHTML = `

No workers configured. Add one below.

`; + return; + } + box.innerHTML = ` + ` + + workers.map((w) => { + const addr = `${w.host || ""}:${w.port || ""}`; + const status = w.online + ? `ready · ${esc(w.status || "ok")}` + : `down`; + const osCpu = w.online ? `${esc(w.os || "?")} / ${esc(w.cpu || "?")}` : "—"; + return ` + + + + + `; + }).join("") + `
AddressStatusOS / CPU
${esc(addr)}${status}${osCpu}
`; } catch (_) { $("#worker-summary").textContent = "Worker summary unavailable"; } } +$("#worker-form").addEventListener("submit", async (e) => { + e.preventDefault(); + const input = $("#worker-spec"); + const spec = input.value.trim(); + if (!spec) return; + try { + const resp = await fetch("/api/workers", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ spec }), + }); + const data = await resp.json().catch(() => ({})); + if (!resp.ok) throw new Error(data.error || resp.statusText); + input.value = ""; + refreshWorkers(); + refreshCapabilities(); + } catch (err) { + $("#worker-spec").value = ""; + $("#worker-spec").placeholder = `Error: ${err.message}`; + } +}); + +$("#workers").addEventListener("click", async (e) => { + const btn = e.target.closest("button.remove-worker"); + if (!btn) return; + const url = btn.dataset.url; + const host = url.replace(/^https?:\/\//, "").split(":")[0]; + const port = url.replace(/^https?:\/\//, "").split(":")[1]; + try { + await fetch(`/api/workers/${encodeURIComponent(port)}/${encodeURIComponent(host)}`, { method: "DELETE" }); + refreshWorkers(); + refreshCapabilities(); + } catch (_) {} +}); + async function refreshJobs() { let jobs; try { diff --git a/ctrl/static/index.html b/ctrl/static/index.html index 43c1090..0b556fc 100644 --- a/ctrl/static/index.html +++ b/ctrl/static/index.html @@ -13,6 +13,15 @@
+
+

Workers

+
+ + +
+
+
+

New build

diff --git a/ctrl/static/style.css b/ctrl/static/style.css index d4a876e..c28a095 100644 --- a/ctrl/static/style.css +++ b/ctrl/static/style.css @@ -59,6 +59,14 @@ main { } .fields-row { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; } +.inline-form { display: flex; gap: 10px; margin-bottom: 12px; } +.inline-form input { flex: 1; } +button.remove-worker { + padding: 5px 10px; + background: var(--fail); + font-size: 12px; +} + button { padding: 8px 14px; background: var(--accent); diff --git a/ctrl/workers.py b/ctrl/workers.py index 38657a5..004c1ed 100644 --- a/ctrl/workers.py +++ b/ctrl/workers.py @@ -1,3 +1,4 @@ +import ctrl.db as db import ctrl.settings as settings import requests @@ -11,7 +12,14 @@ def parse_worker(spec): def configured_workers(): - return [parse_worker(s) for s in settings.WORKERS] + out = [] + for w in db.list_workers(): + out.append({ + "url": f"http://{w['host']}:{w['port']}", + "host": w["host"], + "port": w["port"], + }) + return out def probe_worker(url, timeout=None): diff --git a/data/jobs.db b/data/jobs.db new file mode 100644 index 0000000..d36133b Binary files /dev/null and b/data/jobs.db differ