Manage workers from the web UI, persisted in SQLite

Replace the CROSS_PY_WORKERS env list with a DB-backed worker store so agents
can be added/removed (and their online status verified) from the browser.
- db.py: add workers table + list_workers/add_worker/remove_worker.
- workers.py: configured_workers() reads the DB instead of settings.
- app.py: POST /api/workers (add, probes reachability) and
  DELETE /api/workers/<port>/<host>; startup log uses DB worker count.
- UI: Workers card with add form, per-worker live status + OS/CPU, remove.
- settings.py: drop CROSS_PY_WORKERS/_get_json_list; DB is single source.
- README/DESIGN updated to describe UI-managed workers.

DESIGN.md

@@ -33,10 +33,22 @@ The workers (`build_agent.py`) keep their HTTP API; only one small read-only end
   builds/        downloaded artifacts (runtime)
 ```
 
-**Worker discovery:** static config (list of `host:port`) in the controller settings. On each
-job the controller queries every worker `/status`, filters to `status=="ready"` and matching
-CPU/OS, and picks one. This replaces Zeroconf on the controller; workers can stop advertising,
-and the Zeroconf code path can be removed with the CLI retirement (§7).
+**Worker discovery:** configurable via the web UI and persisted in the controller's SQLite
+`workers` table (host + port per agent). On each job the controller queries every worker
+`/status`, filters to `status=="ready"` and matching CPU/OS, and picks one. This replaces
+Zeroconf on the controller; workers can stop advertising, and the Zeroconf code path can be
+removed with the CLI retirement (§7).
+
+**Worker management API** (adds/removes agents without touching config files):
+
+| Endpoint | Method | Purpose |
+|---|---|---|
+| `GET /api/workers` | GET | live `/status` for each configured worker |
+| `POST /api/workers` | POST | add a worker: JSON `{"spec": "host:port"}` (or `host`+`port`) |
+| `DELETE /api/workers/<port>/<host>` | DELETE | remove a configured worker |
+
+The workers table is seeded on demand from the UI; there is no `CROSS_PY_WORKERS`
+environment bootstrap — the DB is the single source of truth.
 
 We deliberately do **not** rewrite the agent's HTTP API, and do **not** add auth in v1 (same
 trusted-LAN posture as today). The worker endpoint contract used by the controller:
@@ -86,6 +98,8 @@ so results survive worker container restarts and remain available after the work
 |---|---|---|
 | GET | `/` | web UI (single page) |
 | GET | `/api/workers` | live `/status` for each configured worker |
+| POST | `/api/workers` | add a worker: JSON `{"spec": "host:port"}` |
+| DELETE | `/api/workers/<port>/<host>` | remove a configured worker |
 | GET | `/api/capabilities` | distinct (os, cpu) across ready workers, for dropdowns |
 | POST | `/api/jobs` | create job: multipart upload `file` OR JSON `{repo_url, os, cpu}` |
 | GET | `/api/jobs` | list jobs (newest first, with status) |
@@ -150,7 +164,7 @@ agent/                    (worker agent: build_agent.py, zeroconf_server.py)
 ctrl/
   __init__.py
   app.py           (Flask factory, routes, waitress runner)
-  db.py            (SQLite init + helpers)
+  db.py            (SQLite init + helpers; jobs + workers tables)
   scheduler.py     (dispatch + progress-poll loop, worker client)
   workers.py       (worker config load + /status probe)
   settings.py      (env-driven config: DB path, builds dir, worker list)
@@ -160,13 +174,15 @@ ctrl/
     style.css
 ```
 
-Worker config example (env or JSON file):
+Controller config example (env or JSON file):
 ```
-CROSS_PY_WORKERS='["10.0.0.11:9001","10.0.0.12:9001"]'
-CROSS_PY_DATA=/var/lib/cross-py-controller   # holds jobs.db + builds/
+CROSS_PY_DATA=/var/lib/cross-py-controller   # holds jobs.db (jobs + workers) + builds/
 CROSS_PY_PORT=8080
 ```
 
+Workers are added/removed in the web UI and stored in the SQLite `workers` table (no
+`CROSS_PY_WORKERS` env var).
+
 ---
 
 ## 7. What we reuse vs. retire

README.md

@@ -80,24 +80,25 @@ source venv/bin/activate
 pip install -e .
 ```
 
-Set the static worker list (JSON array of `host:port`, matching where your agents are
-listening) and start it:
+Set the port (defaults to `8080`) and start it:
 
 ```sh
-CROSS_PY_WORKERS='["192.168.1.40:9001","192.168.1.41:9001"]' \
-cross-py-controller
+CROSS_PY_PORT=8080 cross-py-controller
 ```
 
-Then open `http://<controller-ip>:8080` in a browser.
+Then open `http://<controller-ip>:8080` in a browser and add your agents from the
+**Workers** section of the UI (in `host:port` form, e.g. `192.168.1.40:9001`). Each
+row shows live status (ready/down) and the agent's OS/CPU/version. Configured workers
+persist in the controller's SQLite DB, so they survive restarts — no environment
+variables needed after setup.
 
 ### Controller environment variables
 
 | Variable | Default | Purpose |
 |---|---|---|
-| `CROSS_PY_WORKERS` | (none) | JSON array of agent `host:port` strings |
 | `CROSS_PY_PORT` | `8080` | HTTP port the controller UI listens on |
 | `CROSS_PY_MAX_CONCURRENT` | `1` | Max simultaneous builds (across distinct agents) |
-| `CROSS_PY_DATA` | `<repo>/data` | Data dir (jobs DB, uploaded sources) |
+| `CROSS_PY_DATA` | `<repo>/data` | Data dir (SQLite jobs + workers DB, uploaded sources) |
 | `CROSS_PY_BUILDS` | `<data>/builds` | Dir where finished build artifacts are saved |
 | `CROSS_PY_DEBUG` | off | Use Flask dev server instead of waitress |
 
@@ -105,12 +106,14 @@ Then open `http://<controller-ip>:8080` in a browser.
 
 ## Using the UI
 
-1. Open the controller in a browser.
-2. Pick the target **OS** and **CPU** (or "any").
-3. Either upload a `.zip` of your project (PyInstaller spec + source) or paste a git
+1. **Add your agents** in the Workers section (`host:port`). Each shows live status
+   (ready/down) and OS/CPU once reachable.
+2. Open the controller in a browser.
+3. Pick the target **OS** and **CPU** (or "any").
+4. Either upload a `.zip` of your project (PyInstaller spec + source) or paste a git
    `repo_url`.
-4. Submit. The job is queued → dispatched to a matching agent → built → artifacts fetched.
-5. Watch live per-step progress, then download the built binaries from the job detail.
+5. Submit. The job is queued → dispatched to a matching agent → built → artifacts fetched.
+6. Watch live per-step progress, then download the built binaries from the job detail.
 
 A submitted project only needs the source plus at least one `*.spec` file; each spec file in
 the archive produces a build.

ctrl/app.py

@@ -46,6 +46,36 @@ def api_workers():
     return jsonify(wrk.probe_all())
 
 
+@app.post("/api/workers")
+def api_add_worker():
+    data = request.get_json(silent=True) or {}
+    spec = str(data.get("spec") or "").strip()
+    host = str(data.get("host") or "").strip()
+    port = data.get("port")
+    try:
+        if spec:
+            w = wrk.parse_worker(spec)
+        elif host and port:
+            w = wrk.parse_worker(f"{host}:{int(port)}")
+        else:
+            return jsonify({"error": "Provide spec ('host:port') or host+port"}), 400
+    except (ValueError, TypeError) as e:
+        return jsonify({"error": str(e)}), 400
+
+    db.add_worker(w["host"], w["port"])
+    db.set_queued_jobs_for_worker(w["url"])
+    log.info("Added worker %s:%s", w["host"], w["port"])
+    status = wrk.probe_worker(w["url"])
+    return jsonify({"host": w["host"], "port": w["port"], "online": status is not None, "status": status}), 201
+
+
+@app.delete("/api/workers/<int:port>/<path:host>")
+def api_remove_worker(host, port):
+    db.remove_worker(host, port)
+    log.info("Removed worker %s:%s", host, port)
+    return jsonify({"removed": True, "host": host, "port": port}), 200
+
+
 @app.get("/api/capabilities")
 def api_capabilities():
     combos = {
@@ -161,7 +191,7 @@ def main():
     _ensure_ready()
     scheduler.start()
     _configure_logging()
-    log.info("Controller listening on 0.0.0.0:%s (%d workers configured)", settings.PORT, len(settings.WORKERS))
+    log.info("Controller listening on 0.0.0.0:%s (%d workers configured)", settings.PORT, len(db.list_workers()))
     if settings.DEBUG:
         app.run(host="0.0.0.0", port=settings.PORT, threaded=True)
         return

ctrl/db.py

@@ -41,6 +41,39 @@ def init_db():
         )
         """
     )
+    conn.execute(
+        """
+        CREATE TABLE IF NOT EXISTS workers (
+            host       TEXT NOT NULL,
+            port       INTEGER NOT NULL,
+            created_at TEXT NOT NULL,
+            PRIMARY KEY (host, port)
+        )
+        """
+    )
+    conn.commit()
+
+
+# ---- workers -------------------------------------------------------------
+
+def list_workers():
+    rows = _conn().execute("SELECT host, port FROM workers ORDER BY host, port").fetchall()
+    return [{"host": r["host"], "port": r["port"]} for r in rows]
+
+
+def add_worker(host, port):
+    import datetime
+    conn = _conn()
+    conn.execute(
+        "INSERT OR IGNORE INTO workers (host, port, created_at) VALUES (?, ?, ?)",
+        (host, int(port), datetime.datetime.now().isoformat()),
+    )
+    conn.commit()
+
+
+def remove_worker(host, port):
+    conn = _conn()
+    conn.execute("DELETE FROM workers WHERE host = ? AND port = ?", (host, int(port)))
     conn.commit()
 
 

ctrl/settings.py

@@ -1,4 +1,3 @@
-import json
 import os
 
 
@@ -9,19 +8,6 @@ def _get_bool(name, default=False):
     return val.strip().lower() in ("1", "true", "yes", "on")
 
 
-def _get_json_list(name, default=None):
-    raw = os.environ.get(name)
-    if not raw:
-        return list(default or [])
-    try:
-        value = json.loads(raw)
-    except json.JSONDecodeError:
-        raise ValueError(f"{name} must be a JSON array string")
-    if not isinstance(value, list):
-        raise ValueError(f"{name} must be a JSON array")
-    return value
-
-
 def _get_int(name, default):
     raw = os.environ.get(name)
     if raw is None:
@@ -34,9 +20,6 @@ DB_PATH = os.environ.get("CROSS_PY_DB", os.path.join(DATA_DIR, "jobs.db"))
 BUILDS_DIR = os.environ.get("CROSS_PY_BUILDS", os.path.join(DATA_DIR, "builds"))
 PORT = _get_int("CROSS_PY_PORT", 8080)
 
-# Static worker list: JSON array of "host:port" strings.
-WORKERS = _get_json_list("CROSS_PY_WORKERS")
-
 # One job at a time by default; raise to allow parallel dispatches to distinct workers.
 MAX_CONCURRENT = _get_int("CROSS_PY_MAX_CONCURRENT", 1)
 

ctrl/static/app.js

@@ -50,15 +50,71 @@ async function refreshWorkers() {
     const workers = await api("/api/workers");
     const online = workers.filter((w) => w.online).length;
     const parts = workers.map((w) =>
-      `${w.host || w.url}${w.online ? "" : " (down)"}`
+      `${w.url}${w.online ? "" : " (down)"}`
     );
     $("#worker-summary").textContent =
-      `${online}/${workers.length} workers online — ${parts.join(" · ")}`;
+      `${online}/${workers.length} workers online` + (parts.length ? ` — ${parts.join(" · ")}` : "");
+
+    const box = $("#workers");
+    if (!workers.length) {
+      box.innerHTML = `<p class="muted">No workers configured. Add one below.</p>`;
+      return;
+    }
+    box.innerHTML = `<table>
+      <tr><th>Address</th><th>Status</th><th>OS / CPU</th><th></th></tr>` +
+      workers.map((w) => {
+        const addr = `${w.host || ""}:${w.port || ""}`;
+        const status = w.online
+          ? `<span class="badge done">ready · ${esc(w.status || "ok")}</span>`
+          : `<span class="badge failed">down</span>`;
+        const osCpu = w.online ? `${esc(w.os || "?")} / ${esc(w.cpu || "?")}` : "—";
+        return `<tr>
+          <td><code>${esc(addr)}</code></td>
+          <td>${status}</td>
+          <td>${osCpu}</td>
+          <td><button class="remove-worker" data-url="${esc(w.url)}">Remove</button></td>
+        </tr>`;
+      }).join("") + `</table>`;
   } catch (_) {
     $("#worker-summary").textContent = "Worker summary unavailable";
   }
 }
 
+$("#worker-form").addEventListener("submit", async (e) => {
+  e.preventDefault();
+  const input = $("#worker-spec");
+  const spec = input.value.trim();
+  if (!spec) return;
+  try {
+    const resp = await fetch("/api/workers", {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ spec }),
+    });
+    const data = await resp.json().catch(() => ({}));
+    if (!resp.ok) throw new Error(data.error || resp.statusText);
+    input.value = "";
+    refreshWorkers();
+    refreshCapabilities();
+  } catch (err) {
+    $("#worker-spec").value = "";
+    $("#worker-spec").placeholder = `Error: ${err.message}`;
+  }
+});
+
+$("#workers").addEventListener("click", async (e) => {
+  const btn = e.target.closest("button.remove-worker");
+  if (!btn) return;
+  const url = btn.dataset.url;
+  const host = url.replace(/^https?:\/\//, "").split(":")[0];
+  const port = url.replace(/^https?:\/\//, "").split(":")[1];
+  try {
+    await fetch(`/api/workers/${encodeURIComponent(port)}/${encodeURIComponent(host)}`, { method: "DELETE" });
+    refreshWorkers();
+    refreshCapabilities();
+  } catch (_) {}
+});
+
 async function refreshJobs() {
   let jobs;
   try {

ctrl/static/index.html

@@ -13,6 +13,15 @@
   </header>
 
   <main>
+    <section class="card">
+      <h2>Workers</h2>
+      <form id="worker-form" class="inline-form">
+        <input type="text" id="worker-spec" placeholder="host:port (e.g. 192.168.1.40:9001)">
+        <button type="submit">Add worker</button>
+      </form>
+      <div id="workers"></div>
+    </section>
+
     <section class="card" id="submit-card">
       <h2>New build</h2>
       <form id="job-form">

ctrl/static/style.css

@@ -59,6 +59,14 @@ main {
 }
 .fields-row { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; }
 
+.inline-form { display: flex; gap: 10px; margin-bottom: 12px; }
+.inline-form input { flex: 1; }
+button.remove-worker {
+  padding: 5px 10px;
+  background: var(--fail);
+  font-size: 12px;
+}
+
 button {
   padding: 8px 14px;
   background: var(--accent);

ctrl/workers.py

@@ -1,3 +1,4 @@
+import ctrl.db as db
 import ctrl.settings as settings
 import requests
 
@@ -11,7 +12,14 @@ def parse_worker(spec):
 
 
 def configured_workers():
-    return [parse_worker(s) for s in settings.WORKERS]
+    out = []
+    for w in db.list_workers():
+        out.append({
+            "url": f"http://{w['host']}:{w['port']}",
+            "host": w["host"],
+            "port": w["port"],
+        })
+    return out
 
 
 def probe_worker(url, timeout=None):