19 Commits

Author SHA1 Message Date
Brett Williams 44e6b4332f Fix create-executables.yml workflow to upload output to release page 2026-06-06 17:32:16 -05:00
brett c38213fb58 Update and modernize create-executables action (#138)
* Modernize create-executables.yml

* Update version numbers

* Fix API version in test
2026-06-06 17:10:53 -05:00
brett 3486feaaf4 Add timeouts and fix multi-camera render issue (#136)
* Fix issue where add job window always submits to localhost, regardless of selected hostname

* Add lots of timeouts

* Fix issue where multiple cameras would not render
2026-06-06 17:01:37 -05:00
brett 141843c916 Fix issue where jobs were not always deleted (#137)
* Better job deletion logic

* Cleanup UI after deleting runs
2026-06-06 16:08:02 -05:00
brett 472c7968b3 Fix issue where add job window always submits to localhost, regardless of selected hostname (#135) 2026-06-06 15:05:31 -05:00
brett b8b71d1e16 Add Blender plugin (#134)
* Unbind hostname to allow localhost submissions

* Fix issue where multiple cameras were outputting to the same directory

* Add Blender plugin
2026-06-06 14:32:48 -05:00
brett f0be78adcc REST API endpoint streamlining and cleanup (#133)
* Consolidate engine_info api calls

* Change api methods to use POST when possible

* Delete engine API cleanup

* Remove redundant installed_engines endpoint

* Update proxy to use similar named methods to new API calls

* More API cleanup

* Jobs API cleanup

* More jobs API cleanup

* Fix add jobs error due to null queue

* Remove unnecessary full_status and snapshot API endpoints

* Streamline add job POST endpoint

* Fix test after method name change
2026-06-06 12:04:52 -05:00
brett 24eb7b5616 Add Unit Tests (#132)
* Add tests and new github workflow

* Add new unit tests

* Add Github CI workflow

* Workflow fix

* Add pytest install to workflow file

* More CI / test updates

* More test cleanup

* Whitespace cleanup and link complexity override

* More whitespace cleanup

* Make lint less strict

* More lint tweaks
2026-06-06 00:02:01 -05:00
Brett Williams 7bf5fb554e Merge remote-tracking branch 'origin/master'
# Conflicts:
#	server.py
2026-06-05 22:05:42 -05:00
brett fa4a97f6fa Refactor: ApplicationContext DI wiring (#131)
* refactor: wire all services through ApplicationContext

- Created src/application_context.py as DI container with TYPE_CHECKING imports
- server.py now instantiates all services in dependency order via ApplicationContext
- Fixed infinite recursion bug: 48 instance methods renamed with underscore prefix
  to avoid shadowing by same-named @classmethod forwarders
- ZeroconfServer: instantiate Zeroconf() in __init__, add _sync_class() to
  configure forwarder, direct _configure/_start calls during wiring
- Config, EngineManager, PreviewManager: all forwarders and _sync_class() intact
- RenderQueue: load_state and subscribe moved to __init__, threading.Lock retained
- DistributedJobManager: subscribe_to_listener moved to __init__

* fix: greedy regex in get_render_devices swallows BlenderKit log output

Changed regex from greedy [\s\S]* to non-greedy .*? with re.DOTALL
so it stops at the first ] (the end of the GPU data JSON array)
instead of matching through timestamped log lines like
[19:36:22.109, __init__.py:2881] that contain trailing brackets.

* fix: AttributeError on .enabled in update_job_count prevents options from rendering

* refactor: log silent AttributeError catches, add _sync_class to remaining services, drop dead ctx slot
2026-06-05 22:01:20 -05:00
Brett Williams b7ba1201e4 refactor: log silent AttributeError catches, add _sync_class to remaining services, drop dead ctx slot 2026-06-05 21:27:02 -05:00
Brett Williams c592236c98 fix: AttributeError on .enabled in update_job_count prevents options from rendering 2026-06-05 20:12:14 -05:00
Brett Williams 0c62f454a7 fix: greedy regex in get_render_devices swallows BlenderKit log output
Changed regex from greedy [\s\S]* to non-greedy .*? with re.DOTALL
so it stops at the first ] (the end of the GPU data JSON array)
instead of matching through timestamped log lines like
[19:36:22.109, __init__.py:2881] that contain trailing brackets.
2026-06-05 19:37:12 -05:00
Brett Williams 552c791207 refactor: wire all services through ApplicationContext
- Created src/application_context.py as DI container with TYPE_CHECKING imports
- server.py now instantiates all services in dependency order via ApplicationContext
- Fixed infinite recursion bug: 48 instance methods renamed with underscore prefix
  to avoid shadowing by same-named @classmethod forwarders
- ZeroconfServer: instantiate Zeroconf() in __init__, add _sync_class() to
  configure forwarder, direct _configure/_start calls during wiring
- Config, EngineManager, PreviewManager: all forwarders and _sync_class() intact
- RenderQueue: load_state and subscribe moved to __init__, threading.Lock retained
- DistributedJobManager: subscribe_to_listener moved to __init__
2026-06-05 05:34:32 -05:00
brett e8992fc91a Add missing docstrings and pylint improvements (#130) 2026-01-20 23:13:38 -06:00
brett 74dce5cc3d Windows path fixes (#129)
* Change uses of os.path to use Pathlib

* Add return types and type hints

* Add more docstrings

* Add missing import to api_server
2026-01-18 00:18:43 -06:00
brett ee9f44e4c4 Mix bad imports 2026-01-16 01:07:34 -06:00
Brett Williams 0a69c184eb Fix pyinstaller spec files 2026-01-12 09:08:35 -06:00
brett 8b3fdd14b5 Add Job Window Redesign (#128)
* Initial refactor of add_job_window

* Improved project naming and fixed Blender engine issue

* Improve time representation in main window

* Cleanup Blender job creation

* Send resolution / fps data in job submission

* More window improvements

* EngineManager renaming and refactoring

* FFMPEG path fixes for ffprobe

* More backend refactoring / improvements

* Performance improvements / API refactoring

* Show current job count in add window UI before submission

* Move some UI update code out of background thread

* Move some main window UI update code out of background thread
2026-01-12 09:06:53 -06:00
53 changed files with 5913 additions and 1518 deletions
+45
View File
@@ -0,0 +1,45 @@
name: CI
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install system deps
run: sudo apt-get update && sudo apt-get install -y libxcb-cursor0 libxcb-xinerama0 blender
- name: Install Python deps
run: |
pip install -r requirements.txt
pip install pytest flake8 pylint
- name: Test
run: python -m pytest tests/ --ignore=tests/job_creation_tests.py -v
- name: Lint — bugs (flake8)
# Fails on syntax errors only. Ignored: ~100+ pre-existing
# style issues (E/W), star-import false positives (F403/F405),
# unused imports (F401), unused vars (F841), f-string debt (F541).
run: flake8 --select=E9 --exclude=src/engines/aerender,build,dist,.git,__pycache__,venv --max-line-length=127 --max-complexity=35
- name: Lint — style (flake8)
# Reports style issues but never fails the build.
# TODO: resolve ~100+ pre-existing style issues
run: flake8 --exit-zero --exclude=src/engines/aerender,build,dist,.git,__pycache__,venv --max-line-length=127 --max-complexity=35
- name: Lint (pylint)
# Reports all issues but never fails the build.
# TODO: resolve pre-existing debt (current score 7.73/10)
run: pylint src/ --max-line-length=120 --ignore-paths=^src/engines/aerender/ --fail-under=0
+95 -29
View File
@@ -1,38 +1,104 @@
name: Create Executables
on:
workflow_dispatch:
release:
- types: [created]
types: [published]
permissions:
contents: write
jobs:
pyinstaller-build-windows:
runs-on: windows-latest
build:
name: Build executables (${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: windows-latest
artifact_suffix: windows
- os: ubuntu-latest
artifact_suffix: linux
- os: macos-latest
artifact_suffix: macos
steps:
- name: Create Executables (Windows)
uses: sayyid5416/pyinstaller@v1
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python_ver: '3.11'
spec: 'client.spec'
requirements: 'requirements.txt'
upload_exe_with_name: 'Zordon'
pyinstaller-build-linux:
runs-on: ubuntu-latest
steps:
- name: Create Executables (Linux)
uses: sayyid5416/pyinstaller@v1
python-version: '3.11'
- name: Install Linux system dependencies
if: runner.os == 'Linux'
run: sudo apt-get update && sudo apt-get install -y libxcb-cursor0 libxcb-xinerama0
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip wheel setuptools
python -m pip install -r requirements.txt
python -m pip install pyinstaller pyinstaller_versionfile
- name: Build client
run: pyinstaller --clean client.spec
- name: Build server
run: pyinstaller --clean server.spec
- name: Package build outputs
shell: bash
env:
ARTIFACT_SUFFIX: ${{ matrix.artifact_suffix }}
run: |
python - <<'PY'
from pathlib import Path
import os
import zipfile
suffix = os.environ['ARTIFACT_SUFFIX']
dist_dir = Path('dist')
asset_dir = Path('release-assets')
asset_dir.mkdir(exist_ok=True)
for app_name in ('Zordon-client', 'Zordon-server'):
source = next(
(
path for path in (
dist_dir / f'{app_name}.exe',
dist_dir / f'{app_name}.app',
dist_dir / app_name,
)
if path.exists()
),
None,
)
if source is None:
raise FileNotFoundError(f'Could not find PyInstaller output for {app_name}')
zip_path = asset_dir / f'{app_name}-{suffix}.zip'
with zipfile.ZipFile(zip_path, 'w', compression=zipfile.ZIP_DEFLATED) as zip_file:
if source.is_dir():
for file_path in source.rglob('*'):
if file_path.is_file():
zip_file.write(file_path, source.name / file_path.relative_to(source))
else:
zip_file.write(source, source.name)
PY
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
python_ver: '3.11'
spec: 'client.spec'
requirements: 'requirements.txt'
upload_exe_with_name: 'Zordon'
pyinstaller-build-macos:
runs-on: macos-latest
steps:
- name: Create Executables (macOS)
uses: sayyid5416/pyinstaller@v1
with:
python_ver: '3.11'
spec: 'client.spec'
requirements: 'requirements.txt'
upload_exe_with_name: 'Zordon'
name: Zordon-build-${{ matrix.artifact_suffix }}
path: release-assets/*.zip
if-no-files-found: error
- name: Attach assets to release
if: github.event_name == 'release'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: gh release upload "${{ github.event.release.tag_name }}" release-assets/*.zip --clobber
-40
View File
@@ -1,40 +0,0 @@
# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
name: Python application
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.10
uses: actions/setup-python@v3
with:
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
continue-on-error: false
# - name: Test with pytest
# run: |
# pytest
+196 -22
View File
@@ -4,43 +4,217 @@
# Zordon
A lightweight, zero-install, distributed rendering and management tool designed to streamline and optimize rendering workflows across multiple machines
A Python-based distributed rendering management tool that supports Blender, FFmpeg, and other render engines. Zordon efficiently manages render jobs across multiple machines, making it ideal for small render farms in home studios or small businesses.
## What is Zordon?
Zordon is tool designed for small render farms, such as those used in home studios or small businesses, to efficiently manage and run render jobs for Blender, FFMPEG, and other video renderers. It simplifies the process of distributing rendering tasks across multiple available machines, optimizing the rendering workflow for artists, animators, and video professionals.
Zordon is a tool designed for small render farms, such as those used in home studios or small businesses, to efficiently manage and run render jobs for Blender, FFmpeg, and other video renderers. It simplifies the process of distributing rendering tasks across multiple available machines, optimizing the rendering workflow for artists, animators, and video professionals.
The system works by:
- **Server**: Central coordinator that manages job queues and distributes tasks to available workers
- **Clients**: Lightweight workers that run on rendering machines and execute assigned jobs
- **API**: RESTful endpoints for programmatic job submission and monitoring
## Features
- **Distributed Rendering**: Queue and distribute render jobs across multiple machines
- **Multi-Engine Support**: Compatible with Blender, FFmpeg, and extensible to other render engines
- **Desktop UI**: PyQt6 interface for job management and monitoring
- **REST API**: Flask-based API for programmatic access
- **Cross-Platform**: Runs on Windows, macOS, and Linux
- **Zero-Install Clients**: Lightweight client executables for worker machines
## Installation
### Prerequisites
- Python 3.11 or later
- Git
### Setup
1. Clone the repository:
```bash
git clone https://github.com/blw1138/Zordon.git
cd Zordon
```
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. (Optional) Install PyInstaller for building executables:
```bash
pip install pyinstaller pyinstaller_versionfile
```
## Usage
### Quick Start
1. **Start the Server**: Run the central server to coordinate jobs.
```bash
python server.py
```
2. **Launch Clients**: On each rendering machine, run the client to connect to the server.
```bash
python client.py
```
Notice: This should be considered a beta and is meant for casual / hobbiest use. Do not use in mission critical environments!
## Supported Renderers
### Detailed Workflow
Zordon supports or plans to support the following renderers:
#### Setting Up a Render Farm
1. Choose one machine as the server (preferably a dedicated machine with good network connectivity).
2. Build and distribute client executables to worker machines:
```bash
pyinstaller client.spec
```
Copy the generated executable to each worker machine.
3. Ensure all machines can communicate via network (same subnet recommended).
#### Submitting Render Jobs
Jobs can be submitted via the desktop UI or programmatically via the API:
- **Via UI**: Use the desktop interface to upload project files, specify render settings, and queue jobs.
- **Via API**: Send `POST` requests to `/api/jobs` with job configuration in JSON format.
Example API request:
```bash
curl -X POST http://localhost:8080/api/jobs \
-H "Content-Type: application/json" \
-d '{
"name": "example-render",
"engine_name": "blender",
"local_path": "/path/to/project.blend",
"output_path": "example-output",
"start_frame": 1,
"end_frame": 100,
"args": {
"export_format": "PNG",
"resolution": [1920, 1080]
},
"enable_split_jobs": false
}'
```
#### Monitoring and Managing Jobs
- **UI**: View job status, progress, logs, and worker availability in real-time.
- **API Endpoints**:
- `GET /api/jobs`: List all jobs
- `POST /api/jobs`: Submit a new job
- `GET /api/jobs/<job_id>`: Get job details
- `POST /api/jobs/<job_id>/cancel`: Cancel a job
- `POST /api/jobs/<job_id>/delete`: Delete a job
- `GET /api/status`: Get server and queue status
- `GET /api/engines`: List engine information
For the full endpoint reference, see [`docs/api.html`](docs/api.html) or
[`docs/API.md`](docs/API.md).
#### Blender Add-on
Zordon includes a Blender add-on for submitting the current `.blend` file
directly from Blender.
- Source: [`addons/blender/zordon_blender`](addons/blender/zordon_blender)
- Installable zip: [`addons/blender/zordon_blender.zip`](addons/blender/zordon_blender.zip)
After installing the add-on, use `Properties > Render > Zordon` to discover or
choose a server, test the connection, upload the current file, and submit
camera-specific jobs from the active Blender scene.
#### Worker Management
Workers automatically connect to the server when started. You can:
- View worker status and capabilities in the dashboard
- Configure worker priorities and resource limits
- Monitor CPU/GPU usage per worker
### Development Mode
For development and testing:
Run the server:
```bash
python server.py
```
Run a client (can run multiple for testing):
```bash
python client.py
```
### Building Executables
Build server executable:
```bash
pyinstaller server.spec
```
Build client executable:
```bash
pyinstaller client.spec
```
## Configuration
Settings are stored in `src/utilities/config.py`. Supports YAML/JSON for data serialization and environment-specific configurations.
## Architecture
Zordon follows a modular architecture with the following key components:
- **API Server** (`src/api/`): Flask-based REST API
- **Engine System** (`src/engines/`): Pluggable render engines (Blender, FFmpeg, etc.)
- **UI** (`src/ui/`): PyQt6-based interface
- **Job Management** (`src/render_queue.py`): Distributed job queue
Design patterns include Factory Pattern for engine creation, Observer Pattern for status updates, and Strategy Pattern for different worker implementations.
## Contributing
1. Fork the repository
2. Create a feature branch: `git checkout -b feature/your-feature`
3. Follow the code style guidelines in `AGENTS.md`
4. Test the build: `pyinstaller server.spec`
5. Submit a pull request
### Commit Message Format
```
feat: add support for new render engine
fix: resolve crash when engine path is invalid
docs: update API documentation
refactor: simplify job status handling
```
## Supported Render Engines
- **Blender**
- **FFMPEG**
- **Adobe After Effects** ([coming soon](https://github.com/blw1138/Zordon/issues/84))
- **Cinema 4D** ([planned](https://github.com/blw1138/Zordon/issues/105))
- **Autodesk Maya** ([planned](https://github.com/blw1138/Zordon/issues/106))
- **FFmpeg**
- **Adobe After Effects** (planned)
- **Cinema 4D** (planned)
- **Autodesk Maya** (planned)
## System Requirements
- Windows 10 or later
- macOS Ventura (13.0) or later
- Linux (Supported versions TBD)
## Build using Pyinstaller
Zordon is regularly tested with Python 3.11 and later. It's packaged and distributed with pyinstaller. It is supported on Windows, macOS and Linux.
```
git clone https://github.com/blw1138/Zordon.git
pip3 install -r requirements.txt
pip3 install pyinstaller
pip3 install pyinstaller_versionfile
pyinstaller main.spec
```
- Linux (supported versions TBD)
## License
Zordon is licensed under the MIT License. See the [LICENSE](LICENSE.txt) file for more details.
## Notice
This software is in beta and intended for casual/hobbyist use. Not recommended for mission-critical environments.
+3 -3
View File
@@ -95,7 +95,7 @@ def main():
new_job = {"name": job_name, "engine_name": args.engine}
try:
response = found_proxy.post_job_to_server(file_path, new_job)
response = found_proxy.create_job(file_path, new_job)
except Exception as e:
print(f"Error creating job: {e}")
exit(1)
@@ -113,7 +113,7 @@ def main():
while percent_complete < 1.0:
# add checks for errors
time.sleep(1)
running_job_data = found_proxy.get_job_info(job_id)
running_job_data = found_proxy.get_job(job_id)
percent_complete = running_job_data['percent_complete']
sys.stdout.write("\x1b[1A") # Move up 1
sys.stdout.write("\x1b[0J") # Clear from cursor to end of screen (optional)
@@ -128,4 +128,4 @@ def main():
if __name__ == "__main__":
main()
main()
Binary file not shown.
+153
View File
@@ -0,0 +1,153 @@
# Zordon Blender Add-on
Submit the current Blender file to a Zordon render server from Blender's Render
properties panel.
## Features
- Submit the current `.blend` file directly to `POST /api/jobs`.
- Choose a configured Zordon server from Blender.
- Test the server connection before submitting.
- Save the current file before upload.
- Default job and output names to the `.blend` filename.
- Use the current scene frame range, render resolution, FPS, and image format.
- Select one or more cameras for submission.
- Submit multiple selected cameras as camera-specific child jobs.
## Install
1. In Blender, open `Edit > Preferences > Add-ons`.
2. Click `Install...`.
3. Select `addons/blender/zordon_blender.zip`.
4. Enable `Zordon Render Submitter`.
## Configure Servers
Use `Discover Servers` to find Zordon servers advertised with Zeroconf.
Discovered servers are merged into the configured server list.
You can also open the add-on preferences and edit `Servers` manually.
Use a comma-separated list:
```text
localhost:8080, render-node.local:8080
```
The selected server appears in `Properties > Render > Zordon`.
If Blender is running on the same machine as Zordon, `localhost:8080` should
work. If Zordon is running on another machine, use that machine's hostname or IP
address.
Start the Zordon server before testing the connection:
```bash
python server.py
```
Discovery first tries Python's `zeroconf` package if it is available inside
Blender. If it is not available, the add-on falls back to the macOS `dns-sd`
command when present.
## Submit A Job
1. Open or save a `.blend` file.
2. Choose a Zordon server in `Properties > Render > Zordon`.
3. Click `Test Connection`.
4. Set the job name, output name, and notes if needed.
5. Choose one or more cameras. At least one camera is always required.
6. Click `Submit Current File`.
The addon uploads the current `.blend` to `POST /api/jobs` as multipart form
data. It uses the current scene frame range, render resolution, FPS, and image
format.
Job name and output name default to the current `.blend` filename.
If one camera is selected, the job renders that camera. If multiple cameras are
selected, the addon submits camera-specific child jobs so each camera renders as
its own Zordon job.
Use `All` to select every camera or `Active Only` to render just the active
scene camera.
## Camera Behavior
At least one camera must be selected. The add-on prevents unchecking the final
selected camera.
When one camera is selected, the add-on sends a single job with:
```json
{
"args": {
"camera": "Camera"
}
}
```
When multiple cameras are selected, the add-on sends `child_jobs`. Each child job
gets its own camera argument and output name suffix, such as:
```json
{
"name": "scene_Camera-001",
"output_path": "scene_Camera-001",
"args": {
"camera": "Camera.001"
}
}
```
## Troubleshooting
### Could Not Reach Zordon
Make sure the Zordon server is running and reachable from Blender.
Check from a terminal:
```bash
curl http://localhost:8080/api/heartbeat
```
If the server is remote, replace `localhost` with the server hostname or IP.
### No Cameras Found
Add at least one Blender camera to the scene before submitting.
### Output Files Already Exist
Multiple camera jobs should use camera-specific output names. If they collide,
make sure the Zordon server includes the output-path fix that preserves child
job `output_path` values.
## Packaging
The add-on source lives at:
```text
addons/blender/zordon_blender/
```
The installable archive is:
```text
addons/blender/zordon_blender.zip
```
Rebuild the archive from `addons/blender`:
```bash
python3 -m zipfile -c zordon_blender.zip zordon_blender
```
## Notes
- The add-on is dependency-free and uses Blender's bundled Python standard
library.
- Only `http://` Zordon servers are currently supported.
- `Save Before Submit` is enabled by default so the uploaded file matches the
current Blender state.
+621
View File
@@ -0,0 +1,621 @@
bl_info = {
'name': 'Zordon Render Submitter',
'author': 'Zordon',
'version': (0, 1, 0),
'blender': (3, 6, 0),
'location': 'Properties > Render > Zordon',
'description': 'Submit the current Blender file to a Zordon render server.',
'category': 'Render',
}
import getpass
import http.client
import json
import os
import re
import socket
import subprocess
import time
from urllib.parse import urlparse
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
import bpy
from bpy.props import BoolProperty, EnumProperty, IntProperty, StringProperty
from bpy.types import AddonPreferences, Operator, Panel, PropertyGroup
DEFAULT_SERVER = 'localhost:8080'
DEFAULT_TIMEOUT = 10
UPLOAD_TIMEOUT = 300
ZORDON_SERVICE_TYPE = '_zordon._tcp.local.'
def _addon_preferences(context):
return context.preferences.addons[__name__].preferences
def _configured_server_items(self, context):
preferences = _addon_preferences(context)
raw_servers = [server.strip() for server in preferences.servers.split(',')]
servers = [server for server in raw_servers if server]
if not servers:
servers = [DEFAULT_SERVER]
return [(server, server, '', index) for index, server in enumerate(servers)]
def _normalize_server_url(server, fallback_port):
if not server:
server = DEFAULT_SERVER
server = server.strip()
if not server.startswith(('http://', 'https://')):
server = f'http://{server}'
parsed = urlparse(server)
hostname = parsed.hostname or 'localhost'
scheme = parsed.scheme or 'http'
port = parsed.port or fallback_port or 8080
return scheme, hostname, int(port)
def _api_url(server, port, path):
scheme, hostname, parsed_port = _normalize_server_url(server, port)
return f'{scheme}://{hostname}:{parsed_port}{path}'
def _get_text(server, port, path, timeout=DEFAULT_TIMEOUT):
request = Request(_api_url(server, port, path), method='GET')
with urlopen(request, timeout=timeout) as response:
return response.status, response.read().decode('utf-8')
def _server_list_from_preferences(preferences):
return [server.strip() for server in preferences.servers.split(',') if server.strip()]
def _save_server_list(preferences, servers):
unique_servers = []
for server in servers:
if server and server not in unique_servers:
unique_servers.append(server)
preferences.servers = ', '.join(unique_servers)
def _discover_servers_with_zeroconf(timeout=3.0):
try:
from zeroconf import ServiceBrowser, Zeroconf
except ImportError:
return []
discovered = []
zeroconf = Zeroconf()
class Listener:
def add_service(self, zc, service_type, name):
info = zc.get_service_info(service_type, name, timeout=1000)
if info and info.server and info.port:
discovered.append(f'{info.server.rstrip(".")}:{info.port}')
def update_service(self, zc, service_type, name):
self.add_service(zc, service_type, name)
def remove_service(self, zc, service_type, name):
pass
try:
ServiceBrowser(zeroconf, ZORDON_SERVICE_TYPE, Listener())
time.sleep(timeout)
finally:
zeroconf.close()
return discovered
def _discover_service_names_with_dns_sd(timeout=2.0):
try:
process = subprocess.Popen(
['dns-sd', '-B', '_zordon', '_tcp', 'local'],
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
text=True,
)
except (FileNotFoundError, OSError):
return []
try:
stdout, _ = process.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
process.terminate()
stdout, _ = process.communicate(timeout=1)
service_names = []
for line in stdout.splitlines():
if ' Add ' not in line:
continue
parts = line.split()
if len(parts) >= 7:
service_name = ' '.join(parts[6:])
if service_name and service_name not in service_names:
service_names.append(service_name)
return service_names
def _resolve_service_with_dns_sd(service_name, timeout=2.0):
try:
process = subprocess.Popen(
['dns-sd', '-L', service_name, '_zordon', '_tcp', 'local'],
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
text=True,
)
except (FileNotFoundError, OSError):
return None
try:
stdout, _ = process.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
process.terminate()
stdout, _ = process.communicate(timeout=1)
for line in stdout.splitlines():
if ' can be reached at ' not in line:
continue
match = re.search(r'can be reached at\s+([^:\s]+):(\d+)', line)
if match:
hostname, port = match.groups()
return f'{hostname.rstrip(".")}:{port}'
return None
def _discover_servers_with_dns_sd():
servers = []
for service_name in _discover_service_names_with_dns_sd():
server = _resolve_service_with_dns_sd(service_name)
if server and server not in servers:
servers.append(server)
return servers
def _discover_zordon_servers():
discovered = _discover_servers_with_zeroconf()
if discovered:
return discovered
return _discover_servers_with_dns_sd()
def _send_multipart_job(server, port, job_data, file_path):
scheme, hostname, parsed_port = _normalize_server_url(server, port)
if scheme != 'http':
raise ValueError('Only http:// Zordon servers are currently supported.')
boundary = f'----ZordonBlender{int(time.time() * 1000)}'
json_payload = json.dumps(job_data).encode('utf-8')
file_name = os.path.basename(file_path)
file_size = os.path.getsize(file_path)
parts_before_file = [
f'--{boundary}\r\n',
'Content-Disposition: form-data; name="json"\r\n',
'Content-Type: application/json\r\n\r\n',
json_payload.decode('utf-8'),
'\r\n',
f'--{boundary}\r\n',
f'Content-Disposition: form-data; name="file"; filename="{file_name}"\r\n',
'Content-Type: application/octet-stream\r\n\r\n',
]
preamble = ''.join(parts_before_file).encode('utf-8')
closing = f'\r\n--{boundary}--\r\n'.encode('utf-8')
content_length = len(preamble) + file_size + len(closing)
connection = http.client.HTTPConnection(hostname, parsed_port, timeout=UPLOAD_TIMEOUT)
try:
connection.putrequest('POST', '/api/jobs')
connection.putheader('Content-Type', f'multipart/form-data; boundary={boundary}')
connection.putheader('Content-Length', str(content_length))
connection.endheaders()
connection.send(preamble)
with open(file_path, 'rb') as upload_file:
while True:
chunk = upload_file.read(1024 * 1024)
if not chunk:
break
connection.send(chunk)
connection.send(closing)
response = connection.getresponse()
response_body = response.read().decode('utf-8', errors='replace')
return response.status, response_body
finally:
connection.close()
def _current_file_path():
return bpy.data.filepath
def _default_job_name():
file_path = _current_file_path()
if file_path:
return os.path.splitext(os.path.basename(file_path))[0]
return 'Blender Render'
def _camera_objects():
return [obj for obj in bpy.data.objects if obj.type == 'CAMERA']
def _default_selected_camera_names(scene):
cameras = _camera_objects()
if not cameras:
return []
if scene.camera:
return [scene.camera.name]
if len(cameras) == 1:
return [cameras[0].name]
return []
def _selected_camera_names(props, scene=None):
if props.selected_cameras:
try:
names = json.loads(props.selected_cameras)
except json.JSONDecodeError:
names = []
existing_names = {camera.name for camera in _camera_objects()}
selected_names = [name for name in names if name in existing_names]
if selected_names:
return selected_names
if scene:
return _default_selected_camera_names(scene)
return []
def _set_selected_camera_names(props, camera_names):
props.selected_cameras = json.dumps(list(camera_names))
def _active_camera_names(scene):
if scene.camera:
return [scene.camera.name]
return _default_selected_camera_names(scene)
def _scene_export_format(scene):
file_format = scene.render.image_settings.file_format
return file_format or 'PNG'
class ZORDON_AddonPreferences(AddonPreferences):
bl_idname = __name__
servers: StringProperty(
name='Servers',
description='Comma-separated Zordon servers. Example: studio-render.local:8080, localhost:8080',
default=DEFAULT_SERVER,
)
def draw(self, context):
layout = self.layout
layout.prop(self, 'servers')
layout.operator('zordon.discover_servers', icon='VIEWZOOM')
class ZORDON_SubmissionProperties(PropertyGroup):
server: EnumProperty(
name='Server',
description='Zordon server to submit this file to',
items=_configured_server_items,
)
port: IntProperty(
name='Fallback Port',
description='Port used when the selected server does not include one',
default=8080,
min=1,
max=65535,
)
job_name: StringProperty(
name='Job Name',
description='Name shown in Zordon',
default='',
)
output_path: StringProperty(
name='Output Name',
description='Output name or path used by the Zordon job',
default='',
)
notes: StringProperty(
name='Notes',
description='Optional notes for the Zordon job',
default='',
)
save_before_submit: BoolProperty(
name='Save Before Submit',
description='Save the current .blend before uploading it',
default=True,
)
use_scene_frame_range: BoolProperty(
name='Use Scene Frame Range',
description='Submit the current scene frame start and end',
default=True,
)
selected_cameras: StringProperty(
name='Selected Cameras',
description='JSON encoded list of selected camera names',
default='',
options={'HIDDEN'},
)
class ZORDON_OT_DiscoverServers(Operator):
bl_idname = 'zordon.discover_servers'
bl_label = 'Discover Servers'
bl_description = 'Find Zordon servers advertised with Zeroconf'
def execute(self, context):
preferences = _addon_preferences(context)
existing_servers = _server_list_from_preferences(preferences)
discovered_servers = _discover_zordon_servers()
if not discovered_servers:
self.report({'WARNING'}, 'No Zordon servers found.')
return {'CANCELLED'}
merged_servers = existing_servers + discovered_servers
_save_server_list(preferences, merged_servers)
props = getattr(context.scene, 'zordon_submission', None)
if props:
try:
props.server = discovered_servers[0]
except (TypeError, ValueError):
pass
self.report({'INFO'}, f'Found {len(discovered_servers)} Zordon server(s).')
return {'FINISHED'}
class ZORDON_OT_TestConnection(Operator):
bl_idname = 'zordon.test_connection'
bl_label = 'Test Connection'
bl_description = 'Check whether the selected Zordon server is reachable'
def execute(self, context):
props = context.scene.zordon_submission
try:
status, body = _get_text(props.server, props.port, '/api/heartbeat')
except (HTTPError, URLError, TimeoutError, OSError) as error:
self.report({'ERROR'}, f'Could not reach Zordon: {error}')
return {'CANCELLED'}
if status == 200:
self.report({'INFO'}, f'Connected to Zordon: {body}')
return {'FINISHED'}
self.report({'ERROR'}, f'Unexpected response from Zordon: HTTP {status}')
return {'CANCELLED'}
class ZORDON_OT_RefreshCameras(Operator):
bl_idname = 'zordon.refresh_cameras'
bl_label = 'Refresh Cameras'
bl_description = 'Refresh the camera checklist from the current Blender scene'
def execute(self, context):
_set_selected_camera_names(
context.scene.zordon_submission,
_default_selected_camera_names(context.scene),
)
return {'FINISHED'}
class ZORDON_OT_ToggleCamera(Operator):
bl_idname = 'zordon.toggle_camera'
bl_label = 'Toggle Camera'
bl_description = 'Toggle whether this camera is submitted to Zordon'
camera_name: StringProperty(name='Camera Name')
def execute(self, context):
props = context.scene.zordon_submission
selected = set(_selected_camera_names(props, context.scene))
if self.camera_name in selected:
if len(selected) == 1:
self.report({'WARNING'}, 'At least one camera must be selected.')
return {'CANCELLED'}
selected.remove(self.camera_name)
else:
selected.add(self.camera_name)
_set_selected_camera_names(props, sorted(selected))
return {'FINISHED'}
class ZORDON_OT_SelectAllCameras(Operator):
bl_idname = 'zordon.select_all_cameras'
bl_label = 'All'
bl_description = 'Select all cameras for submission'
def execute(self, context):
props = context.scene.zordon_submission
_set_selected_camera_names(props, [camera.name for camera in _camera_objects()])
return {'FINISHED'}
class ZORDON_OT_SelectActiveCamera(Operator):
bl_idname = 'zordon.select_active_camera'
bl_label = 'Active Only'
bl_description = 'Select only the active scene camera'
def execute(self, context):
active_cameras = _active_camera_names(context.scene)
if not active_cameras:
self.report({'WARNING'}, 'No cameras found.')
return {'CANCELLED'}
_set_selected_camera_names(context.scene.zordon_submission, active_cameras)
return {'FINISHED'}
class ZORDON_OT_SubmitCurrentFile(Operator):
bl_idname = 'zordon.submit_current_file'
bl_label = 'Submit Current File'
bl_description = 'Upload the current Blender file to the selected Zordon server'
def execute(self, context):
props = context.scene.zordon_submission
scene = context.scene
file_path = _current_file_path()
if not file_path:
self.report({'ERROR'}, 'Save this Blender file before submitting it to Zordon.')
return {'CANCELLED'}
if props.save_before_submit:
bpy.ops.wm.save_as_mainfile(filepath=file_path)
job_name = props.job_name.strip() or _default_job_name()
output_path = props.output_path.strip() or job_name
selected_cameras = _selected_camera_names(props, scene)
if not selected_cameras:
self.report({'ERROR'}, 'Add at least one camera before submitting to Zordon.')
return {'CANCELLED'}
resolution = (
int(scene.render.resolution_x),
int(scene.render.resolution_y),
)
job_data = {
'owner': f'{getpass.getuser()}@{socket.gethostname()}',
'engine_name': 'blender',
'engine_version': 'latest',
'name': job_name,
'output_path': output_path,
'start_frame': int(scene.frame_start) if props.use_scene_frame_range else int(scene.frame_current),
'end_frame': int(scene.frame_end) if props.use_scene_frame_range else int(scene.frame_current),
'priority': 1,
'notes': props.notes,
'enable_split_jobs': False,
'split_jobs_same_os': False,
'args': {
'raw': '',
'export_format': _scene_export_format(scene),
'resolution': resolution,
'fps': scene.render.fps,
},
}
if len(selected_cameras) == 1:
job_data['args']['camera'] = selected_cameras[0]
elif len(selected_cameras) > 1:
child_jobs = []
for camera_name in selected_cameras:
camera_suffix = camera_name.replace(' ', '-')
child_args = dict(job_data['args'])
child_args['camera'] = camera_name
child_jobs.append({
'name': f'{job_name}_{camera_suffix}',
'output_path': f'{output_path}_{camera_suffix}',
'args': child_args,
})
job_data['child_jobs'] = child_jobs
try:
status, body = _send_multipart_job(props.server, props.port, job_data, file_path)
except (ValueError, TimeoutError, OSError) as error:
self.report({'ERROR'}, f'Error submitting to Zordon: {error}')
return {'CANCELLED'}
if 200 <= status < 300:
self.report({'INFO'}, f'Submitted "{job_name}" to Zordon.')
return {'FINISHED'}
self.report({'ERROR'}, f'Zordon returned HTTP {status}: {body}')
return {'CANCELLED'}
class ZORDON_PT_SubmitPanel(Panel):
bl_label = 'Zordon'
bl_idname = 'ZORDON_PT_submit_panel'
bl_space_type = 'PROPERTIES'
bl_region_type = 'WINDOW'
bl_context = 'render'
def draw(self, context):
props = context.scene.zordon_submission
layout = self.layout
default_name = _default_job_name()
cameras = _camera_objects()
selected_cameras = set(_selected_camera_names(props, context.scene))
layout.prop(props, 'server')
layout.prop(props, 'port')
row = layout.row(align=True)
row.operator('zordon.discover_servers', icon='VIEWZOOM')
row.operator('zordon.test_connection', icon='LINKED')
layout.separator()
if not props.job_name:
layout.label(text=f'Job Name Default: {default_name}')
layout.prop(props, 'job_name')
if not props.output_path:
layout.label(text=f'Output Name Default: {default_name}')
layout.prop(props, 'output_path')
layout.prop(props, 'notes')
layout.prop(props, 'save_before_submit')
layout.prop(props, 'use_scene_frame_range')
if props.use_scene_frame_range:
layout.label(text=f'Frames: {context.scene.frame_start} - {context.scene.frame_end}')
else:
layout.label(text=f'Frame: {context.scene.frame_current}')
layout.label(text=f'Format: {_scene_export_format(context.scene)}')
layout.separator()
camera_header = layout.row(align=True)
camera_header.label(text='Cameras')
camera_header.operator('zordon.refresh_cameras', text='', icon='FILE_REFRESH')
camera_header.operator('zordon.select_all_cameras', text='All')
camera_header.operator('zordon.select_active_camera', text='Active Only')
if cameras:
for camera in cameras:
icon = 'CHECKBOX_HLT' if camera.name in selected_cameras else 'CHECKBOX_DEHLT'
label = f'{camera.name} - {camera.data.lens:g}mm'
row = layout.row()
op = row.operator('zordon.toggle_camera', text=label, icon=icon, depress=camera.name in selected_cameras)
op.camera_name = camera.name
else:
layout.label(text='No cameras found')
layout.operator('zordon.submit_current_file', icon='RENDER_ANIMATION')
classes = (
ZORDON_AddonPreferences,
ZORDON_SubmissionProperties,
ZORDON_OT_DiscoverServers,
ZORDON_OT_TestConnection,
ZORDON_OT_RefreshCameras,
ZORDON_OT_ToggleCamera,
ZORDON_OT_SelectAllCameras,
ZORDON_OT_SelectActiveCamera,
ZORDON_OT_SubmitCurrentFile,
ZORDON_PT_SubmitPanel,
)
def register():
for cls in classes:
bpy.utils.register_class(cls)
bpy.types.Scene.zordon_submission = bpy.props.PointerProperty(type=ZORDON_SubmissionProperties)
def unregister():
del bpy.types.Scene.zordon_submission
for cls in reversed(classes):
bpy.utils.unregister_class(cls)
if __name__ == '__main__':
register()
+138 -101
View File
@@ -1,121 +1,158 @@
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_all
# - get version from version file
from PyInstaller.utils.hooks import collect_all
from pathlib import Path
import os
import sys
import platform
src_path = os.path.abspath("src")
sys.path.insert(0, src_path)
from version import APP_NAME, APP_VERSION, APP_AUTHOR
sys.path.insert(0, os.path.abspath('.'))
datas = [('resources', 'resources'), ('src/engines/blender/scripts/', 'src/engines/blender/scripts')]
# ------------------------------------------------------------
# Project paths
# ------------------------------------------------------------
project_root = Path(SPECPATH).resolve()
src_dir = project_root / "src"
# Ensure `src.*` imports work during analysis
sys.path.insert(0, str(project_root))
# ------------------------------------------------------------
# Import version info
# ------------------------------------------------------------
from src.version import APP_NAME, APP_VERSION, APP_AUTHOR
APP_NAME = f"{APP_NAME}-client"
# ------------------------------------------------------------
# PyInstaller data / imports
# ------------------------------------------------------------
datas = [
("resources", "resources"),
("src/engines/blender/scripts", "src/engines/blender/scripts"),
]
binaries = []
hiddenimports = ['zeroconf']
tmp_ret = collect_all('zeroconf')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
hiddenimports = ["zeroconf", "src.version"]
tmp_ret = collect_all("zeroconf")
datas += tmp_ret[0]
binaries += tmp_ret[1]
hiddenimports += tmp_ret[2]
# ------------------------------------------------------------
# Analysis
# ------------------------------------------------------------
a = Analysis(
['client.py'],
pathex=[],
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=1, # fyi: optim level 2 breaks on windows
["client.py"],
pathex=[str(project_root)],
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=1, # optimize=2 breaks Windows builds
)
pyz = PYZ(a.pure)
if platform.system() == 'Darwin': # macOS
# ------------------------------------------------------------
# Platform targets
# ------------------------------------------------------------
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
app = BUNDLE(
exe,
a.binaries,
a.datas,
strip=True,
name=f'{APP_NAME}.app',
icon='resources/Server.png',
bundle_identifier=None,
version=APP_VERSION
)
if platform.system() == "Darwin": # macOS
elif platform.system() == 'Windows':
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
import pyinstaller_versionfile
import tempfile
app = BUNDLE(
exe,
a.binaries,
a.datas,
strip=True,
name=f"{APP_NAME}.app",
icon="resources/Server.png",
bundle_identifier=None,
version=APP_VERSION,
)
version_file_path = os.path.join(tempfile.gettempdir(), 'versionfile.txt')
elif platform.system() == "Windows":
pyinstaller_versionfile.create_versionfile(
output_file=version_file_path,
version=APP_VERSION,
company_name=APP_AUTHOR,
file_description=APP_NAME,
internal_name=APP_NAME,
legal_copyright=f"© {APP_AUTHOR}",
original_filename=f"{APP_NAME}.exe",
product_name=APP_NAME
)
import pyinstaller_versionfile
import tempfile
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
version=version_file_path
)
version_file_path = os.path.join(
tempfile.gettempdir(), "versionfile.txt"
)
else: # linux
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None
)
pyinstaller_versionfile.create_versionfile(
output_file=version_file_path,
version=APP_VERSION,
company_name=APP_AUTHOR,
file_description=APP_NAME,
internal_name=APP_NAME,
legal_copyright=f"© {APP_AUTHOR}",
original_filename=f"{APP_NAME}.exe",
product_name=APP_NAME,
)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
version=version_file_path,
)
else: # Linux
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
+487
View File
@@ -0,0 +1,487 @@
# Zordon API Reference
Zordon exposes a Flask API from `src/api/api_server.py`. The server is started by
`start_api_server()` and listens on `Config.port_number` with all application
routes mounted under `/api`.
The in-repo client wrapper is `src/api/server_proxy.py`. Most UI and distributed
rendering code should prefer `RenderServerProxy` instead of constructing request
URLs directly.
## Versioning
- Current API version: `0.1`
- `RenderServerProxy.request()` sends `X-API-Version` with the current
`API_VERSION`, but the server does not currently validate this header.
## Response Conventions
- JSON endpoints return Flask-serialized dictionaries or lists.
- File endpoints return `send_file()` responses.
- Most error responses are plain text with HTTP `400`, `404`, `500`, or `503`.
- `JobNotFoundError` is mapped to HTTP `400`.
- Unhandled exceptions are mapped to HTTP `500` with a plain-text message.
## Jobs
### `GET /api/jobs`
Returns all render jobs and a cache token.
Response:
```json
{
"jobs": [
{
"id": "job-id",
"name": "job name",
"status": "running"
}
],
"token": "cache-token"
}
```
Known callers:
- `RenderServerProxy.get_jobs()`
- `src/ui/main_window.py`
### `GET /api/jobs/long_poll`
Long-polls the job list until the supplied cache token changes or 30 seconds
elapse.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `token` | No | Cache token returned by `/api/jobs`. |
Responses:
- `200` with the same shape as `/api/jobs` when jobs changed.
- `204` with an empty body when no changes arrive before timeout.
Known callers:
- `RenderServerProxy.get_jobs()` through the background cache updater.
### `GET /api/jobs/status/<status_val>`
Returns jobs matching a render status.
Path parameters:
| Name | Description |
| --- | --- |
| `status_val` | Status string converted by `string_to_status()`. |
Responses:
- `200` with a list of job JSON objects when matches exist.
- `400` when no jobs match the requested status.
Review note: this route is not currently wrapped by `RenderServerProxy` and no
in-repo callers were found.
### `GET /api/jobs/<job_id>`
Returns one job as JSON.
Known callers:
- `RenderServerProxy.get_job()`
- `add_job.py`
- `src/ui/main_window.py`
- `src/distributed_job_manager.py`
- `tests/job_creation_tests.py`
### `GET /api/jobs/<job_id>/logs`
Returns the job log file as `text/plain`.
Known callers:
- `src/ui/main_window.py` opens this URL directly.
### `GET /api/jobs/<job_id>/files`
Returns a list of output filenames for the job.
Known callers:
- `RenderServerProxy.get_job_files()`
- `src/utilities/server_helper.py`
### `GET /api/jobs/<job_id>/download`
Downloads one output file.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `filename` | Yes | Case-insensitive filename from the job file list. |
Responses:
- `200` with the requested file as an attachment.
- `400` when `filename` is missing.
- `404` when the file is not found.
Known callers:
- `RenderServerProxy.download_job_file()`
- `src/utilities/server_helper.py`
### `GET /api/jobs/<job_id>/download_all`
Creates a temporary zip of the job output directory and downloads it.
Known callers:
- `RenderServerProxy.download_all_job_files()`
- `src/ui/main_window.py`
- `src/utilities/server_helper.py`
### `GET /api/jobs/<job_id>/thumbnail`
Returns a generated preview image or video for a job.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `size` | No | `big` selects the larger preview path. Currently parsed but not applied. |
| `video_ok` | No | If truthy and a video preview exists, video can be returned. |
Responses:
- `200` with `image/jpeg` or `video/mp4`.
- `404` when no thumbnail is available.
- `500` on preview generation errors.
Known callers:
- `src/ui/main_window.py`
Review note: `size=big` is parsed into `big_thumb` but not used.
## Job Lifecycle
### `POST /api/jobs`
Adds one or more render jobs.
Request formats:
- JSON request body.
- Multipart form with a `json` field and optional `file` upload.
Common job fields include:
| Name | Description |
| --- | --- |
| `name` | Display name for the render job. |
| `renderer` | Render engine name such as `blender` or `ffmpeg`. |
| `start_frame` | First frame to render. |
| `end_frame` | Last frame to render. |
| `args` | Engine-specific render arguments. |
| `enable_split_jobs` | Whether distributed subjobs may be created. |
| `child_jobs` | Optional subjob definitions. |
| `local_path` | Local file path used when posting to localhost. |
Responses:
- `200` with created job data.
- `400` for invalid or missing job data.
- `500` for unexpected processing or creation errors.
Known callers:
- `RenderServerProxy.create_job()`
- `add_job.py`
- `src/ui/add_job_window.py`
- `src/distributed_job_manager.py`
- `tests/job_creation_tests.py`
### `POST /api/jobs/<job_id>/cancel`
Cancels a job.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `confirm` | Yes | Must be truthy or the request is rejected. |
| `redirect` | No | If truthy, redirects to `index`. |
Known callers:
- `RenderServerProxy.cancel_job()`
- `src/ui/main_window.py`
- `src/distributed_job_manager.py`
### `POST /api/jobs/<job_id>/delete`
Deletes a job, stops it first, deletes previews, and removes owned upload/output
directories when safe.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `confirm` | Yes | Must be truthy or the request is rejected. |
Known callers:
- `RenderServerProxy.delete_job()`
- `src/ui/main_window.py`
### `POST /api/jobs/<job_id>/subjob_update`
Notifies a parent job that a child/subjob changed state.
Request body:
- JSON representation of a subjob.
Known callers:
- `RenderServerProxy.send_subjob_update_notification()`
- `src/distributed_job_manager.py`
## Status and Environment
### `GET /api/heartbeat`
Returns the current timestamp as plain text. Used for fast connectivity checks.
Known callers:
- `RenderServerProxy.check_connection()`
### `GET /api/status`
Returns local system and queue status.
Response includes:
- timestamp
- operating system and version
- CPU brand, count, and current utilization
- memory totals and current utilization
- job counts
- hostname and port
- app and API versions
Known callers:
- `RenderServerProxy.get_status()`
### `GET /api/presets`
Returns `config/presets.yaml`.
Review note: no in-repo callers were found.
### `GET /api/cpu_benchmark`
Runs a CPU benchmark for 10 seconds and returns the score as plain text.
Known callers:
- `src/utilities/server_helper.py`
### `GET /api/disk_benchmark`
Runs a disk I/O benchmark and returns write/read speeds.
Review note: no in-repo callers were found.
## Engines
### `GET /api/engines/for_filename`
Returns the engine name suitable for a project filename.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `filename` | Yes | Project filename or path. The client currently sends only the basename. |
Known callers:
- `RenderServerProxy.get_engine_for_filename()`
- `src/ui/add_job_window.py`
### `GET /api/engines`
Returns installed engine data keyed by engine name.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `response_type` | No | `standard` or `full`; defaults to `standard`. |
`full` responses also include supported extensions, supported export formats,
system info, and UI options.
Known callers:
- `RenderServerProxy.get_engines()`
- `src/ui/settings_window.py`
- `src/ui/engine_browser.py`
### `GET /api/engines/names`
Returns installed engine names as a list without instantiating engine classes.
Use this for lightweight selection UIs that only need engine names.
Known callers:
- `RenderServerProxy.get_engine_names()`
- `src/ui/add_job_window.py`
### `GET /api/engines/<engine_name>`
Returns installed version data for a single engine.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `response_type` | No | `standard` or `full`; defaults to `standard`. |
`full` responses also include supported extensions, supported export formats,
system info, and UI options.
Known callers:
- `RenderServerProxy.get_engine()`
- `src/ui/add_job_window.py`
### `GET /api/engines/<engine_name>/availability`
Returns whether an engine can accept jobs on this server, plus CPU count,
installed versions, and hostname.
Known callers:
- `RenderServerProxy.get_engine_availability()`
- `src/distributed_job_manager.py`
### `GET /api/engines/<engine_name>/args`
Returns engine arguments.
Review note: no in-repo callers were found.
### `GET /api/engines/<engine_name>/help`
Returns engine help text.
Known callers:
- `src/ui/add_job_window.py` opens this URL directly.
### `GET /api/engines/download_available`
Checks whether a managed engine version is available to download.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `engine` | Yes | Engine name. |
| `version` | Yes | Engine version. |
| `system_os` | No | Target OS. |
| `cpu` | No | Target CPU architecture. |
Review note: no in-repo callers were found.
### `GET /api/engines/most_recent_version`
Finds the most recent downloadable version for an engine.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `engine` | Yes | Engine name. |
| `system_os` | No | Target OS. |
| `cpu` | No | Target CPU architecture. |
Review note: no in-repo callers were found.
### `POST /api/engines/download`
Downloads a managed engine version.
Query parameters:
| Name | Required | Description |
| --- | --- | --- |
| `engine` | Yes | Engine name. |
| `version` | Yes | Engine version. |
| `system_os` | No | Target OS. |
| `cpu` | No | Target CPU architecture. |
Review note: no in-repo callers were found. Settings currently calls
`EngineManager.download_engine()` directly instead of this API route.
### `POST /api/engines/delete`
Deletes a managed engine download.
JSON body:
| Name | Required | Description |
| --- | --- | --- |
| `engine` | Yes | Engine name. |
| `version` | Yes | Engine version. |
| `system_os` | No | Target OS. |
| `cpu` | No | Target CPU architecture. |
Known callers:
- `RenderServerProxy.delete_engine_download()`
- `src/ui/engine_browser.py`
## Debug
### `GET /api/_debug/detected_clients`
Returns hostnames detected by Zeroconf.
Review note: development/debug only, with an inline comment saying it probably
should not ship.
### `POST /api/_debug/clear_history`
Clears render queue history and returns `success`.
Review note: development/debug only.
## Redundancy and Cleanup Review
Routes with no in-repo callers found:
- `GET /api/jobs/status/<status_val>`
- `GET /api/presets`
- `GET /api/disk_benchmark`
- `GET /api/engines/<engine_name>/args`
- `GET /api/engines/download_available`
- `GET /api/engines/most_recent_version`
- `POST /api/engines/download`
- `GET /api/_debug/detected_clients`
- `POST /api/_debug/clear_history`
Routes or methods with cleanup risks:
- `job_thumbnail()` parses `size=big` but never uses the resulting `big_thumb`
value.
+577
View File
@@ -0,0 +1,577 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Zordon API Reference</title>
<style>
:root {
--bg: #0d1117;
--surface: #161b22;
--surface2: #1c2333;
--border: #30363d;
--text: #e6edf3;
--text-muted: #8b949e;
--blue: #58a6ff;
--green: #3fb950;
--orange: #d29922;
--red: #f85149;
--purple: #bc8cff;
--cyan: #39d2c0;
}
* { box-sizing: border-box; }
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
background: var(--bg);
color: var(--text);
line-height: 1.6;
}
header {
border-bottom: 1px solid var(--border);
padding: 56px 24px 36px;
text-align: center;
}
header h1 {
margin: 0 0 10px;
font-size: 2.7rem;
font-weight: 800;
color: var(--text);
}
header p {
max-width: 760px;
margin: 0 auto;
color: var(--text-muted);
font-size: 1.05rem;
}
main {
display: grid;
grid-template-columns: 250px minmax(0, 1fr);
gap: 32px;
max-width: 1180px;
margin: 0 auto;
padding: 36px 24px 56px;
}
nav {
position: sticky;
top: 18px;
align-self: start;
background: var(--surface);
border: 1px solid var(--border);
border-radius: 8px;
padding: 16px;
}
nav strong {
display: block;
margin-bottom: 8px;
color: var(--text);
}
nav a {
display: block;
padding: 5px 0;
color: var(--text-muted);
text-decoration: none;
font-size: .92rem;
}
nav a:hover { color: var(--blue); }
section { margin-bottom: 44px; }
h2 {
display: inline-block;
margin: 0 0 18px;
padding-bottom: 8px;
border-bottom: 2px solid var(--blue);
font-size: 1.45rem;
}
h3 {
margin: 0 0 10px;
color: var(--cyan);
font-size: 1.08rem;
}
p, li { color: var(--text-muted); }
code {
font-family: "SFMono-Regular", Consolas, "Liberation Mono", Menlo, monospace;
background: var(--surface2);
border-radius: 4px;
padding: 2px 5px;
color: var(--cyan);
font-size: .9em;
}
pre {
margin: 12px 0;
padding: 14px;
overflow-x: auto;
background: var(--surface);
border: 1px solid var(--border);
border-radius: 8px;
color: var(--text);
}
pre code {
padding: 0;
background: transparent;
color: inherit;
}
table {
width: 100%;
margin: 12px 0;
border-collapse: collapse;
font-size: .92rem;
}
th, td {
padding: 9px 10px;
border: 1px solid var(--border);
text-align: left;
vertical-align: top;
}
th {
background: var(--surface2);
color: var(--text);
font-weight: 600;
}
td { color: var(--text-muted); }
.meta-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
gap: 14px;
margin-bottom: 24px;
}
.panel, .endpoint {
background: var(--surface);
border: 1px solid var(--border);
border-radius: 8px;
padding: 18px;
}
.panel h3 { color: var(--text); }
.panel ul { margin: 8px 0 0; padding-left: 20px; }
.endpoint {
margin-bottom: 16px;
}
.endpoint-header {
display: flex;
align-items: baseline;
gap: 10px;
flex-wrap: wrap;
margin-bottom: 8px;
}
.method {
display: inline-block;
min-width: 52px;
padding: 2px 8px;
border-radius: 4px;
text-align: center;
font-weight: 700;
font-size: .76rem;
letter-spacing: .02em;
}
.get { background: rgba(88, 166, 255, .16); color: var(--blue); border: 1px solid rgba(88, 166, 255, .35); }
.post { background: rgba(63, 185, 80, .16); color: var(--green); border: 1px solid rgba(63, 185, 80, .35); }
.multi { background: rgba(210, 153, 34, .16); color: var(--orange); border: 1px solid rgba(210, 153, 34, .35); }
.path {
color: var(--text);
font-family: "SFMono-Regular", Consolas, "Liberation Mono", Menlo, monospace;
font-weight: 650;
overflow-wrap: anywhere;
}
.note {
margin-top: 12px;
padding: 10px 12px;
border-left: 3px solid var(--orange);
background: rgba(210, 153, 34, .08);
color: var(--text-muted);
}
.danger {
border-left-color: var(--red);
background: rgba(248, 81, 73, .08);
}
.callers {
margin: 10px 0 0;
padding-left: 20px;
}
.callers li { margin: 2px 0; }
@media (max-width: 820px) {
main { grid-template-columns: 1fr; }
nav { position: static; }
header h1 { font-size: 2.1rem; }
}
</style>
</head>
<body>
<header>
<h1>Zordon API Reference</h1>
<p>Flask endpoints exposed by <code>src/api/api_server.py</code>, with the in-repo client wrapper in <code>src/api/server_proxy.py</code>.</p>
</header>
<main>
<nav aria-label="API sections">
<strong>Sections</strong>
<a href="#overview">Overview</a>
<a href="#jobs">Jobs</a>
<a href="#job-lifecycle">Job Lifecycle</a>
<a href="#status">Status and Environment</a>
<a href="#engines">Engines</a>
<a href="#debug">Debug</a>
<a href="#cleanup">Cleanup Review</a>
</nav>
<div>
<section id="overview">
<h2>Overview</h2>
<div class="meta-grid">
<div class="panel">
<h3>Versioning</h3>
<ul>
<li>Current API version: <code>0.1</code></li>
<li><code>RenderServerProxy.request()</code> sends <code>X-API-Version</code>.</li>
<li>The server does not currently validate that header.</li>
</ul>
</div>
<div class="panel">
<h3>Response Conventions</h3>
<ul>
<li>JSON endpoints return Flask-serialized dictionaries or lists.</li>
<li>File endpoints return <code>send_file()</code> responses.</li>
<li>Most errors are plain text with <code>400</code>, <code>404</code>, <code>500</code>, or <code>503</code>.</li>
<li><code>JobNotFoundError</code> maps to HTTP <code>400</code>.</li>
</ul>
</div>
</div>
</section>
<section id="jobs">
<h2>Jobs</h2>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs</span></div>
<p>Returns all render jobs and a cache token.</p>
<pre><code>{
"jobs": [{"id": "job-id", "name": "job name", "status": "running"}],
"token": "cache-token"
}</code></pre>
<p>Known callers:</p>
<ul class="callers">
<li><code>RenderServerProxy.get_jobs()</code></li>
<li><code>src/ui/main_window.py</code></li>
</ul>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/long_poll</span></div>
<p>Long-polls the job list until the supplied cache token changes or 30 seconds elapse.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>token</code></td><td>No</td><td>Cache token returned by <code>/api/jobs</code>.</td></tr>
</table>
<p>Returns <code>200</code> with job data when changed, or <code>204</code> when no change arrives before timeout.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/status/&lt;status_val&gt;</span></div>
<p>Returns jobs matching a render status converted by <code>string_to_status()</code>.</p>
<div class="note">No <code>RenderServerProxy</code> wrapper or in-repo caller was found.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/&lt;job_id&gt;</span></div>
<p>Returns one job as JSON.</p>
<p>Known callers include <code>RenderServerProxy.get_job()</code>, <code>add_job.py</code>, <code>src/ui/main_window.py</code>, <code>src/distributed_job_manager.py</code>, and <code>tests/job_creation_tests.py</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/&lt;job_id&gt;/logs</span></div>
<p>Returns the job log file as <code>text/plain</code>. The main window opens this URL directly.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/&lt;job_id&gt;/files</span></div>
<p>Returns a list of output filenames for the job.</p>
<p>Known callers: <code>RenderServerProxy.get_job_files()</code> and <code>src/utilities/server_helper.py</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/&lt;job_id&gt;/download</span></div>
<p>Downloads one output file.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>filename</code></td><td>Yes</td><td>Case-insensitive filename from the job file list.</td></tr>
</table>
<p>Returns <code>200</code> with an attachment, <code>400</code> when <code>filename</code> is missing, or <code>404</code> when the file is not found.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/&lt;job_id&gt;/download_all</span></div>
<p>Creates a temporary zip of the job output directory and downloads it.</p>
<p>Known callers: <code>RenderServerProxy.download_all_job_files()</code>, <code>src/ui/main_window.py</code>, and <code>src/utilities/server_helper.py</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/jobs/&lt;job_id&gt;/thumbnail</span></div>
<p>Returns a generated preview image or video for a job.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>size</code></td><td>No</td><td><code>big</code> is parsed but not currently applied.</td></tr>
<tr><td><code>video_ok</code></td><td>No</td><td>If truthy and a video preview exists, video can be returned.</td></tr>
</table>
<div class="note">Cleanup note: <code>size=big</code> is parsed into <code>big_thumb</code> but not used.</div>
</article>
</section>
<section id="job-lifecycle">
<h2>Job Lifecycle</h2>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/jobs</span></div>
<p>Adds one or more render jobs. Accepts either a JSON request body or multipart form data with a <code>json</code> field and optional <code>file</code> upload.</p>
<table>
<tr><th>Common Field</th><th>Description</th></tr>
<tr><td><code>name</code></td><td>Display name for the render job.</td></tr>
<tr><td><code>renderer</code></td><td>Render engine name such as <code>blender</code> or <code>ffmpeg</code>.</td></tr>
<tr><td><code>start_frame</code></td><td>First frame to render.</td></tr>
<tr><td><code>end_frame</code></td><td>Last frame to render.</td></tr>
<tr><td><code>args</code></td><td>Engine-specific render arguments.</td></tr>
<tr><td><code>enable_split_jobs</code></td><td>Whether distributed subjobs may be created.</td></tr>
<tr><td><code>child_jobs</code></td><td>Optional subjob definitions.</td></tr>
<tr><td><code>local_path</code></td><td>Local file path used when posting to localhost.</td></tr>
</table>
<p>Known callers include <code>RenderServerProxy.create_job()</code>, <code>add_job.py</code>, <code>src/ui/add_job_window.py</code>, <code>src/distributed_job_manager.py</code>, and integration tests.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/jobs/&lt;job_id&gt;/cancel</span></div>
<p>Cancels a job. Requires a truthy <code>confirm</code> query parameter.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>confirm</code></td><td>Yes</td><td>Must be truthy or the request is rejected.</td></tr>
<tr><td><code>redirect</code></td><td>No</td><td>If truthy, redirects to <code>index</code>.</td></tr>
</table>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/jobs/&lt;job_id&gt;/delete</span></div>
<p>Deletes a job, stops it first, deletes previews, and removes owned upload/output directories when safe.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>confirm</code></td><td>Yes</td><td>Must be truthy or the request is rejected.</td></tr>
</table>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/jobs/&lt;job_id&gt;/subjob_update</span></div>
<p>Notifies a parent job that a child/subjob changed state. The request body is the JSON representation of the subjob.</p>
</article>
</section>
<section id="status">
<h2>Status and Environment</h2>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/heartbeat</span></div>
<p>Returns the current timestamp as plain text. Used by <code>RenderServerProxy.check_connection()</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/status</span></div>
<p>Returns local system and queue status, including operating system, CPU, memory, job counts, hostname, port, app version, and API version.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/presets</span></div>
<p>Returns <code>config/presets.yaml</code>.</p>
<div class="note">No in-repo callers were found.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/cpu_benchmark</span></div>
<p>Runs a CPU benchmark for 10 seconds and returns the score as plain text. Used by <code>src/utilities/server_helper.py</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/disk_benchmark</span></div>
<p>Runs a disk I/O benchmark and returns write/read speeds.</p>
<div class="note">No in-repo callers were found.</div>
</article>
</section>
<section id="engines">
<h2>Engines</h2>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/for_filename</span></div>
<p>Returns the engine name suitable for a project filename.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>filename</code></td><td>Yes</td><td>Project filename or path. The client currently sends only the basename.</td></tr>
</table>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines</span></div>
<p>Returns installed engine data keyed by engine name.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>response_type</code></td><td>No</td><td><code>standard</code> or <code>full</code>; defaults to <code>standard</code>.</td></tr>
</table>
<p><code>full</code> responses include supported extensions, supported export formats, system info, and UI options.</p>
<p>Known callers: <code>RenderServerProxy.get_engines()</code>, <code>src/ui/settings_window.py</code>, and <code>src/ui/engine_browser.py</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/names</span></div>
<p>Returns installed engine names as a list without instantiating engine classes. Use this for lightweight selection UIs that only need engine names.</p>
<p>Known callers: <code>RenderServerProxy.get_engine_names()</code> and <code>src/ui/add_job_window.py</code>.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/&lt;engine_name&gt;</span></div>
<p>Returns installed version data for a single engine.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>response_type</code></td><td>No</td><td><code>standard</code> or <code>full</code>; defaults to <code>standard</code>.</td></tr>
</table>
<p><code>full</code> responses include supported extensions, supported export formats, system info, and UI options.</p>
<p>Known caller: <code>RenderServerProxy.get_engine()</code> in the add-job window.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/&lt;engine_name&gt;/availability</span></div>
<p>Returns whether an engine can accept jobs on this server, plus CPU count, installed versions, and hostname.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/&lt;engine_name&gt;/args</span></div>
<p>Returns engine arguments.</p>
<div class="note">No in-repo callers were found.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/&lt;engine_name&gt;/help</span></div>
<p>Returns engine help text. The add-job window opens this URL directly.</p>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/download_available</span></div>
<p>Checks whether a managed engine version is available to download.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>engine</code></td><td>Yes</td><td>Engine name.</td></tr>
<tr><td><code>version</code></td><td>Yes</td><td>Engine version.</td></tr>
<tr><td><code>system_os</code></td><td>No</td><td>Target OS.</td></tr>
<tr><td><code>cpu</code></td><td>No</td><td>Target CPU architecture.</td></tr>
</table>
<div class="note">No in-repo callers were found.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/engines/most_recent_version</span></div>
<p>Finds the most recent downloadable version for an engine.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>engine</code></td><td>Yes</td><td>Engine name.</td></tr>
<tr><td><code>system_os</code></td><td>No</td><td>Target OS.</td></tr>
<tr><td><code>cpu</code></td><td>No</td><td>Target CPU architecture.</td></tr>
</table>
<div class="note">No in-repo callers were found.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/engines/download</span></div>
<p>Downloads a managed engine version.</p>
<table>
<tr><th>Query Parameter</th><th>Required</th><th>Description</th></tr>
<tr><td><code>engine</code></td><td>Yes</td><td>Engine name.</td></tr>
<tr><td><code>version</code></td><td>Yes</td><td>Engine version.</td></tr>
<tr><td><code>system_os</code></td><td>No</td><td>Target OS.</td></tr>
<tr><td><code>cpu</code></td><td>No</td><td>Target CPU architecture.</td></tr>
</table>
<div class="note">Settings currently calls <code>EngineManager.download_engine()</code> directly instead of this API route.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/engines/delete</span></div>
<p>Deletes a managed engine download.</p>
<table>
<tr><th>JSON Field</th><th>Required</th><th>Description</th></tr>
<tr><td><code>engine</code></td><td>Yes</td><td>Engine name.</td></tr>
<tr><td><code>version</code></td><td>Yes</td><td>Engine version.</td></tr>
<tr><td><code>system_os</code></td><td>No</td><td>Target OS.</td></tr>
<tr><td><code>cpu</code></td><td>No</td><td>Target CPU architecture.</td></tr>
</table>
</article>
</section>
<section id="debug">
<h2>Debug</h2>
<article class="endpoint">
<div class="endpoint-header"><span class="method get">GET</span><span class="path">/api/_debug/detected_clients</span></div>
<p>Returns hostnames detected by Zeroconf.</p>
<div class="note">Development/debug only, with an inline comment saying it probably should not ship.</div>
</article>
<article class="endpoint">
<div class="endpoint-header"><span class="method post">POST</span><span class="path">/api/_debug/clear_history</span></div>
<p>Clears render queue history and returns <code>success</code>.</p>
<div class="note">Development/debug only.</div>
</article>
</section>
<section id="cleanup">
<h2>Cleanup Review</h2>
<div class="panel">
<h3>Routes With No In-Repo Callers Found</h3>
<ul>
<li><code>GET /api/jobs/status/&lt;status_val&gt;</code></li>
<li><code>GET /api/presets</code></li>
<li><code>GET /api/disk_benchmark</code></li>
<li><code>GET /api/engines/&lt;engine_name&gt;/args</code></li>
<li><code>GET /api/engines/download_available</code></li>
<li><code>GET /api/engines/most_recent_version</code></li>
<li><code>POST /api/engines/download</code></li>
<li><code>GET /api/_debug/detected_clients</code></li>
<li><code>POST /api/_debug/clear_history</code></li>
</ul>
</div>
<div class="panel" style="margin-top: 14px;">
<h3>Cleanup Risks</h3>
<ul>
<li><code>job_thumbnail()</code> parses <code>size=big</code> but never uses the resulting <code>big_thumb</code> value.</li>
</ul>
</div>
</section>
</div>
</main>
</body>
</html>
+3 -1
View File
@@ -1,2 +1,4 @@
[pytest]
norecursedirs = src/engines/aerender .git build dist *.egg venv .venv env .env __pycache__ .pytest_cache
norecursedirs = src/engines/aerender .git build dist *.egg venv .venv env .env __pycache__ .pytest_cache
testpaths = tests
python_files = test_*.py job_creation_tests.py
+57 -29
View File
@@ -3,6 +3,7 @@ import multiprocessing
import os
import socket
import threading
from pathlib import Path
import psutil
@@ -10,12 +11,13 @@ from src.api.api_server import API_VERSION
from src.api.api_server import start_api_server
from src.api.preview_manager import PreviewManager
from src.api.serverproxy_manager import ServerProxyManager
from src.application_context import ApplicationContext
from src.distributed_job_manager import DistributedJobManager
from src.engines.engine_manager import EngineManager
from src.render_queue import RenderQueue
from src.utilities.config import Config
from src.utilities.misc_helper import (get_gpu_info, system_safe_path, current_system_cpu, current_system_os,
current_system_os_version, current_system_cpu_brand, check_for_updates)
from src.utilities.misc_helper import (get_gpu_info, current_system_cpu, current_system_os,
current_system_os_version, current_system_cpu_brand)
from src.utilities.zeroconf_server import ZeroconfServer
from src.version import APP_NAME, APP_VERSION
@@ -25,31 +27,53 @@ logger = logging.getLogger()
class ZordonServer:
def __init__(self):
self.ctx = ApplicationContext()
# setup logging
logging.basicConfig(format='%(asctime)s: %(levelname)s: %(module)s: %(message)s', datefmt='%d-%b-%y %H:%M:%S',
level=Config.server_log_level.upper())
logging.getLogger("requests").setLevel(logging.WARNING) # suppress noisy requests/urllib3 logging
logging.getLogger("urllib3").setLevel(logging.WARNING)
# Load Config YAML
# ---- Bootstrap Config ----
Config.setup_config_dir()
Config.load_config(system_safe_path(os.path.join(Config.config_dir(), 'config.yaml')))
config_path = Path(Config.config_dir()) / "config.yaml"
self.ctx.config = Config()
self.ctx.config.load(config_path)
Config._default_instance = self.ctx.config
Config._sync_class()
# configure default paths
EngineManager.engines_path = system_safe_path(
os.path.join(os.path.join(os.path.expanduser(Config.upload_folder),
'engines')))
os.makedirs(EngineManager.engines_path, exist_ok=True)
PreviewManager.storage_path = system_safe_path(
os.path.join(os.path.expanduser(Config.upload_folder), 'previews'))
# ---- Engine Manager ----
self.ctx.engine_manager = EngineManager()
self.ctx.engine_manager.engines_path = Path(Config.upload_folder).expanduser() / "engines"
os.makedirs(self.ctx.engine_manager.engines_path, exist_ok=True)
EngineManager._default_instance = self.ctx.engine_manager
EngineManager._sync_class()
# ---- Preview Manager ----
self.ctx.preview_manager = PreviewManager()
self.ctx.preview_manager.storage_path = Path(Config.upload_folder).expanduser() / "previews"
PreviewManager._default_instance = self.ctx.preview_manager
PreviewManager._sync_class()
# ---- Render Queue ----
self.ctx.render_queue = RenderQueue()
RenderQueue._default_instance = self.ctx.render_queue
RenderQueue._sync_class()
RenderQueue.load_state(database_directory=Path(Config.upload_folder).expanduser())
# ---- Distributed Job Manager ----
self.ctx.distributed_job_manager = DistributedJobManager()
DistributedJobManager._default_instance = self.ctx.distributed_job_manager
DistributedJobManager._sync_class()
DistributedJobManager.subscribe_to_listener()
self.api_server = None
self.server_hostname = None
self.server_hostname: str = socket.gethostname()
def start_server(self):
def existing_process(process_name):
import psutil
current_pid = os.getpid()
current_process = psutil.Process(current_pid)
for proc in psutil.process_iter(['pid', 'name', 'ppid']):
@@ -72,15 +96,13 @@ class ZordonServer:
# main start
logger.info(f"Starting {APP_NAME} Render Server ({APP_VERSION})")
logger.debug(f"Upload directory: {os.path.expanduser(Config.upload_folder)}")
logger.debug(f"Upload directory: {Path(Config.upload_folder).expanduser()}")
logger.debug(f"Thumbs directory: {PreviewManager.storage_path}")
logger.debug(f"Engines directory: {EngineManager.engines_path}")
# Set up the RenderQueue object
RenderQueue.load_state(database_directory=system_safe_path(os.path.expanduser(Config.upload_folder)))
ServerProxyManager.subscribe_to_listener()
DistributedJobManager.subscribe_to_listener()
# get hostname
ServerProxyManager.subscribe_to_listener()
# update hostname
self.server_hostname = socket.gethostname()
# configure and start API server
@@ -89,16 +111,21 @@ class ZordonServer:
self.api_server.start()
# start zeroconf server
ZeroconfServer.configure(f"_{APP_NAME.lower()}._tcp.local.", self.server_hostname, Config.port_number)
ZeroconfServer.properties = {'system_cpu': current_system_cpu(),
'system_cpu_brand': current_system_cpu_brand(),
'system_cpu_cores': multiprocessing.cpu_count(),
'system_os': current_system_os(),
'system_os_version': current_system_os_version(),
'system_memory': round(psutil.virtual_memory().total / (1024**3)), # in GB
'gpu_info': get_gpu_info(),
'api_version': API_VERSION}
ZeroconfServer.start()
ctx = self.ctx
ctx.zeroconf_server = ZeroconfServer()
ctx.zeroconf_server._configure(f"_{APP_NAME.lower()}._tcp.local.", self.server_hostname, Config.port_number)
ctx.zeroconf_server.properties = {'system_cpu': current_system_cpu(),
'system_cpu_brand': current_system_cpu_brand(),
'system_cpu_cores': multiprocessing.cpu_count(),
'system_os': current_system_os(),
'system_os_version': current_system_os_version(),
'system_memory': round(psutil.virtual_memory().total / (1024**3)),
'gpu_info': get_gpu_info(),
'api_version': API_VERSION}
ZeroconfServer._default_instance = ctx.zeroconf_server
ZeroconfServer._sync_class()
ctx.zeroconf_server._start()
logger.info(f"{APP_NAME} Render Server started - Hostname: {self.server_hostname}")
RenderQueue.start() # Start evaluating the render queue
@@ -114,6 +141,7 @@ class ZordonServer:
logger.exception(f"Exception during prepare for shutdown: {e}")
logger.info(f"{APP_NAME} Render Server has shut down")
if __name__ == '__main__':
server = ZordonServer()
try:
+140 -72
View File
@@ -1,90 +1,158 @@
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_all
# - get version from version file
from PyInstaller.utils.hooks import collect_all
from pathlib import Path
import os
import sys
import platform
sys.path.insert(0, os.path.abspath('.'))
from version import APP_NAME, APP_VERSION, APP_AUTHOR
APP_NAME = APP_NAME + " Server"
datas = [('resources', 'resources'), ('src/engines/blender/scripts/', 'src/engines/blender/scripts')]
# ------------------------------------------------------------
# Project paths
# ------------------------------------------------------------
project_root = Path(SPECPATH).resolve()
src_dir = project_root / "src"
# Ensure `src.*` imports work during analysis
sys.path.insert(0, str(project_root))
# ------------------------------------------------------------
# Import version info
# ------------------------------------------------------------
from src.version import APP_NAME, APP_VERSION, APP_AUTHOR
APP_NAME = f"{APP_NAME}-server"
# ------------------------------------------------------------
# PyInstaller data / imports
# ------------------------------------------------------------
datas = [
("resources", "resources"),
("src/engines/blender/scripts", "src/engines/blender/scripts"),
]
binaries = []
hiddenimports = ['zeroconf']
tmp_ret = collect_all('zeroconf')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
hiddenimports = ["zeroconf", "src.version"]
tmp_ret = collect_all("zeroconf")
datas += tmp_ret[0]
binaries += tmp_ret[1]
hiddenimports += tmp_ret[2]
# ------------------------------------------------------------
# Analysis
# ------------------------------------------------------------
a = Analysis(
['server.py'],
pathex=[],
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=1, # fyi: optim level 2 breaks on windows
["server.py"],
pathex=[str(project_root)],
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=1, # optimize=2 breaks Windows builds
)
pyz = PYZ(a.pure)
if platform.system() == 'Windows':
# ------------------------------------------------------------
# Platform targets
# ------------------------------------------------------------
import pyinstaller_versionfile
import tempfile
if platform.system() == "Darwin": # macOS
version_file_path = os.path.join(tempfile.gettempdir(), 'versionfile.txt')
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
pyinstaller_versionfile.create_versionfile(
output_file=version_file_path,
version=APP_VERSION,
company_name=APP_AUTHOR,
file_description=APP_NAME,
internal_name=APP_NAME,
legal_copyright=f"© {APP_AUTHOR}",
original_filename=f"{APP_NAME}.exe",
product_name=APP_NAME
)
app = BUNDLE(
exe,
a.binaries,
a.datas,
strip=True,
name=f"{APP_NAME}.app",
icon="resources/Server.png",
bundle_identifier=None,
version=APP_VERSION,
)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
version=version_file_path
)
elif platform.system() == "Windows":
else: # linux / macOS
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None
)
import pyinstaller_versionfile
import tempfile
version_file_path = os.path.join(
tempfile.gettempdir(), "versionfile.txt"
)
pyinstaller_versionfile.create_versionfile(
output_file=version_file_path,
version=APP_VERSION,
company_name=APP_AUTHOR,
file_description=APP_NAME,
internal_name=APP_NAME,
legal_copyright=f"© {APP_AUTHOR}",
original_filename=f"{APP_NAME}.exe",
product_name=APP_NAME,
)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
version=version_file_path,
)
else: # Linux
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.datas,
[],
name=APP_NAME,
debug=False,
bootloader_ignore_signals=False,
strip=True,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
+225 -166
View File
@@ -2,20 +2,22 @@
import concurrent.futures
import json
import logging
import os
import pathlib
import shutil
import socket
import ssl
import tempfile
import time
import traceback
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, Optional
import cpuinfo
import psutil
import yaml
from flask import Flask, request, send_file, after_this_request, Response, redirect, url_for
from sqlalchemy.orm.exc import DetachedInstanceError
from werkzeug.exceptions import HTTPException
from src.api.job_import_handler import JobImportHandler
from src.api.preview_manager import PreviewManager
@@ -23,7 +25,7 @@ from src.distributed_job_manager import DistributedJobManager
from src.engines.engine_manager import EngineManager
from src.render_queue import RenderQueue, JobNotFoundError
from src.utilities.config import Config
from src.utilities.misc_helper import system_safe_path, current_system_os, current_system_cpu, \
from src.utilities.misc_helper import current_system_os, current_system_cpu, \
current_system_os_version, num_to_alphanumeric, get_gpu_info
from src.utilities.status_utils import string_to_status
from src.version import APP_VERSION
@@ -32,9 +34,9 @@ logger = logging.getLogger()
server = Flask(__name__)
ssl._create_default_https_context = ssl._create_unverified_context # disable SSL for downloads
API_VERSION = "0.1"
API_VERSION = "1.0"
def start_api_server(hostname=None):
def start_api_server(hostname: Optional[str] = None, bind_host: str = '0.0.0.0') -> None:
# get hostname
if not hostname:
@@ -44,7 +46,7 @@ def start_api_server(hostname=None):
# load flask settings
server.config['HOSTNAME'] = hostname
server.config['PORT'] = int(Config.port_number)
server.config['UPLOAD_FOLDER'] = system_safe_path(os.path.expanduser(Config.upload_folder))
server.config['UPLOAD_FOLDER'] = str(Path(Config.upload_folder).expanduser())
server.config['MAX_CONTENT_PATH'] = Config.max_content_path
server.config['enable_split_jobs'] = Config.enable_split_jobs
@@ -52,9 +54,9 @@ def start_api_server(hostname=None):
flask_log = logging.getLogger('werkzeug')
flask_log.setLevel(Config.flask_log_level.upper())
logger.debug('Starting API server')
logger.debug(f'Starting API server on {bind_host}:{server.config["PORT"]}')
try:
server.run(host=hostname, port=server.config['PORT'], debug=Config.flask_debug_enable, use_reloader=False,
server.run(host=bind_host, port=server.config['PORT'], debug=Config.flask_debug_enable, use_reloader=False,
threaded=True)
finally:
logger.debug('Stopping API server')
@@ -65,7 +67,7 @@ def start_api_server(hostname=None):
# --------------------------------------------
@server.get('/api/jobs')
def jobs_json():
def jobs_json() -> Dict[str, Any]:
"""Retrieves all jobs from the render queue in JSON format.
This endpoint fetches all jobs currently in the render queue, converts them to JSON format,
@@ -82,7 +84,7 @@ def jobs_json():
return {'jobs': all_jobs, 'token': job_cache_token}
@server.get('/api/jobs_long_poll')
@server.get('/api/jobs/long_poll')
def long_polling_jobs():
hash_token = request.args.get('token', None)
start_time = time.time()
@@ -96,7 +98,7 @@ def long_polling_jobs():
time.sleep(1)
@server.get('/api/jobs/<status_val>')
@server.get('/api/jobs/status/<status_val>')
def filtered_jobs_json(status_val):
state = string_to_status(status_val)
jobs = [x.json() for x in RenderQueue.jobs_with_status(state)]
@@ -110,7 +112,7 @@ def filtered_jobs_json(status_val):
# Job Details / File Handling
# --------------------------------------------
@server.get('/api/job/<job_id>')
@server.get('/api/jobs/<job_id>')
def get_job_details(job_id):
"""Retrieves the details of a requested job in JSON format
@@ -123,7 +125,7 @@ def get_job_details(job_id):
return RenderQueue.job_with_id(job_id).json()
@server.get('/api/job/<job_id>/logs')
@server.get('/api/jobs/<job_id>/logs')
def get_job_logs(job_id):
"""Retrieves the log file for a specific render job.
@@ -134,60 +136,62 @@ def get_job_logs(job_id):
Response: The log file's content as plain text, or an empty response if the log file is not found.
"""
found_job = RenderQueue.job_with_id(job_id)
log_path = system_safe_path(found_job.log_path())
log_path = Path(found_job.log_path())
log_data = None
if log_path and os.path.exists(log_path):
if log_path and log_path.exists():
with open(log_path) as file:
log_data = file.read()
return Response(log_data, mimetype='text/plain')
@server.get('/api/job/<job_id>/file_list')
def get_file_list(job_id):
return [os.path.basename(x) for x in RenderQueue.job_with_id(job_id).file_list()]
@server.get('/api/jobs/<job_id>/files')
def get_job_files(job_id):
return [Path(p).name for p in RenderQueue.job_with_id(job_id).file_list()]
@server.route('/api/job/<job_id>/download')
@server.route('/api/jobs/<job_id>/download')
def download_requested_file(job_id):
requested_filename = request.args.get('filename')
requested_filename = request.args.get("filename")
if not requested_filename:
return 'Filename required', 400
return "Filename required", 400
found_job = RenderQueue.job_with_id(job_id)
for job_filename in found_job.file_list():
if os.path.basename(job_filename).lower() == requested_filename.lower():
return send_file(job_filename, as_attachment=True, )
for job_file in found_job.file_list():
p = Path(job_file)
if p.name.lower() == requested_filename.lower():
return send_file(str(p), as_attachment=True)
return f"File '{requested_filename}' not found", 404
@server.route('/api/job/<job_id>/download_all')
@server.route('/api/jobs/<job_id>/download_all')
def download_all_files(job_id):
zip_filename = None
@after_this_request
def clear_zip(response):
if zip_filename and os.path.exists(zip_filename):
if zip_filename and zip_filename.exists():
try:
os.remove(zip_filename)
zip_filename.unlink()
except Exception as e:
logger.warning(f"Error removing zip file '{zip_filename}': {e}")
return response
found_job = RenderQueue.job_with_id(job_id)
output_dir = os.path.dirname(found_job.output_path)
if os.path.exists(output_dir):
from zipfile import ZipFile
zip_filename = system_safe_path(os.path.join(tempfile.gettempdir(),
pathlib.Path(found_job.input_path).stem + '.zip'))
with ZipFile(zip_filename, 'w') as zipObj:
for f in os.listdir(output_dir):
zipObj.write(filename=system_safe_path(os.path.join(output_dir, f)),
arcname=os.path.basename(f))
return send_file(zip_filename, mimetype="zip", as_attachment=True, )
else:
return f'Cannot find project files for job {job_id}', 500
output_dir = Path(found_job.output_path).parent
if not output_dir.exists():
return f"Cannot find project files for job {job_id}", 500
zip_filename = Path(tempfile.gettempdir()) / f"{Path(found_job.input_path).stem}.zip"
from zipfile import ZipFile
with ZipFile(zip_filename, "w") as zipObj:
for f in output_dir.iterdir():
if f.is_file():
zipObj.write(f, arcname=f.name)
return send_file(str(zip_filename), mimetype="zip", as_attachment=True)
# --------------------------------------------
@@ -195,36 +199,13 @@ def download_all_files(job_id):
# --------------------------------------------
@server.get('/api/presets')
def presets():
presets_path = system_safe_path('config/presets.yaml')
def presets() -> Dict[str, Any]:
presets_path = Path('config/presets.yaml')
with open(presets_path) as f:
loaded_presets = yaml.load(f, Loader=yaml.FullLoader)
return loaded_presets
@server.get('/api/full_status')
def full_status():
full_results = {'timestamp': datetime.now().isoformat(), 'servers': {}}
try:
snapshot_results = snapshot()
server_data = {'status': snapshot_results.get('status', {}), 'jobs': snapshot_results.get('jobs', {}),
'is_online': True}
full_results['servers'][server.config['HOSTNAME']] = server_data
except Exception as e:
logger.error(f"Exception fetching full status: {e}")
return full_results
@server.get('/api/snapshot')
def snapshot():
server_status = status()
server_jobs = [x.json() for x in RenderQueue.all_jobs()]
server_data = {'status': server_status, 'jobs': server_jobs, 'timestamp': datetime.now().isoformat()}
return server_data
@server.route('/api/status')
def status():
return {"timestamp": datetime.now().isoformat(),
@@ -250,10 +231,10 @@ def status():
# Job Lifecyle (Create, Cancel, Delete)
# --------------------------------------------
@server.post('/api/add_job')
def add_job_handler():
@server.post('/api/jobs')
def create_jobs_handler():
"""
POST /api/add_job
POST /api/jobs
Add a render job to the queue.
**Request Formats**
@@ -292,30 +273,18 @@ def add_job_handler():
err_msg = f"Error processing job data: {e}"
return err_msg, 400
except Exception as e:
traceback.print_exception(e)
err_msg = f"Unknown error processing data: {e}"
return err_msg, 500
try:
loaded_project_local_path = processed_job_data['__loaded_project_local_path']
created_jobs = []
if processed_job_data.get("child_jobs"):
for child_job_diffs in processed_job_data["child_jobs"]:
processed_child_job_data = processed_job_data.copy()
processed_child_job_data.pop("child_jobs")
processed_child_job_data.update(child_job_diffs)
child_job = DistributedJobManager.create_render_job(processed_child_job_data, loaded_project_local_path)
created_jobs.append(child_job)
else:
new_job = DistributedJobManager.create_render_job(processed_job_data, loaded_project_local_path)
created_jobs.append(new_job)
return [x.json() for x in created_jobs]
return JobImportHandler.create_jobs_from_processed_data(processed_job_data)
except Exception as e:
logger.exception(f"Error creating render job: {e}")
return 'unknown error', 500
@server.get('/api/job/<job_id>/cancel')
@server.post('/api/jobs/<job_id>/cancel')
def cancel_job(job_id):
if not request.args.get('confirm', False):
return 'Confirmation required to cancel job', 400
@@ -329,16 +298,21 @@ def cancel_job(job_id):
return "Unknown error", 500
@server.route('/api/job/<job_id>/delete', methods=['POST', 'GET'])
@server.post('/api/jobs/<job_id>/delete')
def delete_job(job_id):
try:
if not request.args.get('confirm', False):
return 'Confirmation required to delete job', 400
if not request.args.get("confirm", False):
return "Confirmation required to delete job", 400
# Check if we can remove the 'output' directory
found_job = RenderQueue.job_with_id(job_id)
project_dir = os.path.dirname(os.path.dirname(found_job.input_path))
output_dir = os.path.dirname(found_job.output_path)
input_path = Path(found_job.input_path)
output_path = Path(found_job.output_path)
upload_root = Path(server.config["UPLOAD_FOLDER"])
project_dir = input_path.parent.parent
output_dir = output_path.parent
found_job.stop()
try:
@@ -346,97 +320,187 @@ def delete_job(job_id):
except Exception as e:
logger.error(f"Error deleting previews for {found_job}: {e}")
# finally delete the job
RenderQueue.delete_job(found_job)
# delete the output_dir
if server.config['UPLOAD_FOLDER'] in output_dir and os.path.exists(output_dir):
shutil.rmtree(output_dir)
# See if we own the project_dir (i.e. was it uploaded) - if so delete the directory
try:
if server.config['UPLOAD_FOLDER'] in project_dir and os.path.exists(project_dir):
# check to see if any other projects are sharing the same project file
project_dir_files = [f for f in os.listdir(project_dir) if not f.startswith('.')]
if len(project_dir_files) == 0 or (len(project_dir_files) == 1 and 'source' in project_dir_files[0]):
logger.info(f"Removing project directory: {project_dir}")
shutil.rmtree(project_dir)
except Exception as e:
logger.error(f"Error removing project files: {e}")
if _has_remaining_jobs_for_project(project_dir):
_delete_job_output(output_path, project_dir, upload_root)
elif project_dir.exists() and project_dir.is_relative_to(upload_root):
logger.info(f"Removing project directory: {project_dir}")
shutil.rmtree(project_dir)
return "Job deleted", 200
except Exception as e:
logger.error(f"Error deleting job: {e}")
return f"Error deleting job: {e}", 500
def _has_remaining_jobs_for_project(project_dir):
for job in RenderQueue.all_jobs():
try:
if Path(job.input_path).parent.parent == project_dir:
return True
except (AttributeError, TypeError):
continue
return False
def _delete_job_output(output_path, project_dir, upload_root):
output_dir = output_path.parent
project_output_dir = project_dir / 'output'
if not output_dir.exists() or not output_dir.is_relative_to(upload_root):
return
if output_dir != project_output_dir:
shutil.rmtree(output_dir)
return
output_prefix = output_path.stem
for output_file in output_dir.iterdir():
if _output_file_matches_prefix(output_file.name, output_prefix) and output_file.is_file():
output_file.unlink()
def _output_file_matches_prefix(filename, output_prefix):
return (
filename == output_prefix or
filename.startswith(f'{output_prefix}_') or
filename.startswith(f'{output_prefix}.')
)
# --------------------------------------------
# Engine Info and Management:
# --------------------------------------------
@server.get('/api/engine_info')
def engine_info():
@server.get('/api/engines/for_filename')
def get_engine_for_filename():
filename = request.args.get("filename")
if not filename:
return "Error: filename is required", 400
found_engine = EngineManager.engine_class_for_project_path(filename)
if not found_engine:
return f"Error: cannot find a suitable engine for '{filename}'", 400
return found_engine.name()
def _validated_engine_response_type():
response_type = request.args.get('response_type', 'standard')
if response_type not in ['full', 'standard']:
raise ValueError(f"Invalid response_type: {response_type}")
return response_type
def process_engine(engine):
try:
# Get all installed versions of the engine
installed_versions = EngineManager.all_versions_for_engine(engine.name())
if not installed_versions:
return None
system_installed_versions = [v for v in installed_versions if v['type'] == 'system']
install_path = system_installed_versions[0]['path'] if system_installed_versions else installed_versions[0]['path']
def _engine_info_for_engine(engine_class, response_type='standard'):
try:
installed_versions = EngineManager.all_version_data_for_engine(engine_class.name())
if not installed_versions:
return None
en = engine(install_path)
engine_name = en.name()
result = {
engine_name: {
'is_available': RenderQueue.is_available_for_job(engine_name),
'versions': installed_versions
system_installed_versions = [v for v in installed_versions if v['type'] == 'system']
install_path = (
system_installed_versions[0]['path'] if system_installed_versions else installed_versions[0]['path']
)
engine = engine_class(install_path)
engine_name = engine.name()
result = {
'is_available': RenderQueue.is_available_for_job(engine_name),
'versions': installed_versions
}
if response_type == 'full':
with concurrent.futures.ThreadPoolExecutor() as executor:
future_results = {
'supported_extensions': executor.submit(engine.supported_extensions),
'supported_export_formats': executor.submit(engine.get_output_formats),
'system_info': executor.submit(engine.system_info),
'options': executor.submit(engine.ui_options)
}
}
if response_type == 'full':
with concurrent.futures.ThreadPoolExecutor() as executor:
future_results = {
'supported_extensions': executor.submit(en.supported_extensions),
'supported_export_formats': executor.submit(en.get_output_formats),
'system_info': executor.submit(en.system_info)
}
for key, future in future_results.items():
result[key] = future.result()
for key, future in future_results.items():
result[engine_name][key] = future.result()
return result
return result
except Exception as e:
logger.error(f"Error fetching details for engine '{engine_class.name()}': {e}")
return {}
except Exception as e:
logger.error(f"Error fetching details for engine '{engine.name()}': {e}")
raise e
@server.get('/api/engines')
def get_engines_info():
response_type = _validated_engine_response_type()
engine_data = {}
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = {executor.submit(process_engine, engine): engine.name() for engine in EngineManager.supported_engines()}
futures = {
executor.submit(_engine_info_for_engine, engine, response_type): engine.name()
for engine in EngineManager.supported_engines()
}
for future in concurrent.futures.as_completed(futures):
result = future.result()
if result:
engine_data.update(result)
engine_data[futures[future]] = result
return engine_data
@server.get('/api/<engine_name>/is_available')
def is_engine_available(engine_name):
@server.get('/api/engines/names')
def get_engine_names():
result = []
for engine_class in EngineManager.supported_engines():
data = EngineManager.all_version_data_for_engine(engine_class.name())
if data:
result.append(engine_class.name())
return result
@server.get('/api/engines/<engine_name>')
def get_engine(engine_name):
try:
response_type = _validated_engine_response_type()
engine_class = EngineManager.engine_class_with_name(engine_name)
return _engine_info_for_engine(engine_class, response_type) or {}
except Exception as e:
logger.error(f"Error fetching details for engine '{engine_name}': {e}")
return {}
@server.get('/api/engines/<engine_name>/availability')
def get_engine_availability(engine_name):
return {'engine': engine_name, 'available': RenderQueue.is_available_for_job(engine_name),
'cpu_count': int(psutil.cpu_count(logical=False)),
'versions': EngineManager.all_versions_for_engine(engine_name),
'hostname': server.config['HOSTNAME']}
'versions': EngineManager.all_version_data_for_engine(engine_name),
'hostname': server.config.get('HOSTNAME', socket.gethostname())}
@server.get('/api/is_engine_available_to_download')
@server.get('/api/engines/<engine_name>/args')
def get_engine_args(engine_name):
try:
engine_class = EngineManager.engine_class_with_name(engine_name)
if not engine_class:
return f"Cannot find engine '{engine_name}'", 400
return engine_class().get_arguments()
except LookupError:
return f"Cannot find engine '{engine_name}'", 400
@server.get('/api/engines/<engine_name>/help')
def get_engine_help(engine_name):
try:
engine_class = EngineManager.engine_class_with_name(engine_name)
if not engine_class:
return f"Cannot find engine '{engine_name}'", 400
return engine_class().get_help()
except LookupError:
return f"Cannot find engine '{engine_name}'", 400
# --------------------------------------------
# Engine Downloads and Updates:
# --------------------------------------------
@server.get('/api/engines/download_available')
def is_engine_available_to_download():
available_result = EngineManager.version_is_available_to_download(request.args.get('engine'),
request.args.get('version'),
@@ -446,7 +510,7 @@ def is_engine_available_to_download():
(f"Cannot find available download for {request.args.get('engine')} {request.args.get('version')}", 500)
@server.get('/api/find_most_recent_version')
@server.get('/api/engines/most_recent_version')
def find_most_recent_version():
most_recent = EngineManager.find_most_recent_version(request.args.get('engine'),
request.args.get('system_os'),
@@ -455,7 +519,7 @@ def find_most_recent_version():
(f"Error finding most recent version of {request.args.get('engine')}", 500)
@server.post('/api/download_engine')
@server.post('/api/engines/download')
def download_engine():
download_result = EngineManager.download_engine(request.args.get('engine'),
request.args.get('version'),
@@ -465,7 +529,7 @@ def download_engine():
(f"Error downloading {request.args.get('engine')} {request.args.get('version')}", 500)
@server.post('/api/delete_engine')
@server.post('/api/engines/delete')
def delete_engine_download():
json_data = request.json
delete_result = EngineManager.delete_engine_download(json_data.get('engine'),
@@ -476,24 +540,6 @@ def delete_engine_download():
(f"Error deleting {json_data.get('engine')} {json_data.get('version')}", 500)
@server.get('/api/engine/<engine_name>/args')
def get_engine_args(engine_name):
try:
engine_class = EngineManager.engine_with_name(engine_name)
return engine_class().get_arguments()
except LookupError:
return f"Cannot find engine '{engine_name}'", 400
@server.get('/api/engine/<engine_name>/help')
def get_engine_help(engine_name):
try:
engine_class = EngineManager.engine_with_name(engine_name)
return engine_class().get_help()
except LookupError:
return f"Cannot find engine '{engine_name}'", 400
# --------------------------------------------
# Miscellaneous:
# --------------------------------------------
@@ -501,14 +547,14 @@ def get_engine_help(engine_name):
def heartbeat():
return datetime.now().isoformat(), 200
@server.post('/api/job/<job_id>/send_subjob_update_notification')
@server.post('/api/jobs/<job_id>/subjob_update')
def subjob_update_notification(job_id):
subjob_details = request.json
DistributedJobManager.handle_subjob_update_notification(RenderQueue.job_with_id(job_id), subjob_data=subjob_details)
return Response(status=200)
@server.route('/api/job/<job_id>/thumbnail')
@server.route('/api/jobs/<job_id>/thumbnail')
def job_thumbnail(job_id):
try:
@@ -568,8 +614,21 @@ def handle_detached_instance(_):
return "Unavailable", 503
@server.errorhandler(404)
def handle_404(error):
url = request.url
err_msg = f"404 Not Found: {url}"
if 'favicon' not in url:
logger.warning(err_msg)
return err_msg, 404
@server.errorhandler(Exception)
def handle_general_error(general_error):
if isinstance(general_error, HTTPException):
return general_error.description, general_error.code
traceback.print_exception(type(general_error), general_error, general_error.__traceback__)
err_msg = f"Server error: {general_error}"
logger.error(err_msg)
return err_msg, 500
@@ -586,7 +645,7 @@ def detected_clients():
return ZeroconfServer.found_hostnames()
@server.get('/api/_debug/clear_history')
@server.post('/api/_debug/clear_history')
def clear_history():
RenderQueue.clear_history()
return 'success'
+109 -32
View File
@@ -5,18 +5,78 @@ import shutil
import tempfile
import zipfile
from datetime import datetime
from pathlib import Path
import requests
from tqdm import tqdm
from werkzeug.utils import secure_filename
from src.distributed_job_manager import DistributedJobManager
logger = logging.getLogger()
class JobImportHandler:
"""Handles job import operations for rendering projects.
This class provides functionality to validate, download, and process
job data and project files for the rendering queue system.
"""
@classmethod
def validate_job_data(cls, new_job_data, upload_directory, uploaded_file=None):
def create_jobs_from_processed_data(cls, processed_job_data: dict) -> list[dict]:
""" Takes processed job data and creates new jobs
Args: processed_job_data: Dictionary containing job information"""
loaded_project_local_path = processed_job_data['__loaded_project_local_path']
# prepare child job data
job_data_to_create = []
if processed_job_data.get("child_jobs"):
for child_job_diffs in processed_job_data["child_jobs"]:
processed_child_job_data = processed_job_data.copy()
processed_child_job_data.pop("child_jobs")
processed_child_job_data.update(child_job_diffs)
processed_child_job_data['__use_output_subdir'] = True
job_data_to_create.append(processed_child_job_data)
else:
job_data_to_create.append(processed_job_data)
# create the jobs
created_jobs = []
for job_data in job_data_to_create:
new_job = DistributedJobManager.create_render_job(job_data, loaded_project_local_path)
created_jobs.append(new_job)
# Save notes to .txt
if processed_job_data.get("notes"):
parent_dir = Path(loaded_project_local_path).parent.parent
notes_name = processed_job_data['name'] + "-notes.txt"
with (Path(parent_dir) / notes_name).open("w") as f:
f.write(processed_job_data["notes"])
return [x.json() for x in created_jobs]
@classmethod
def validate_job_data(cls, new_job_data: dict, upload_directory: Path, uploaded_file=None) -> dict:
"""Validates and prepares job data for import.
This method validates the job data dictionary, handles project file
acquisition (upload, download, or local copy), and prepares the job
directory structure.
Args:
new_job_data: Dictionary containing job information including
'name', 'engine_name', and optionally 'url' or 'local_path'.
upload_directory: Base directory for storing uploaded jobs.
uploaded_file: Optional uploaded file object from the request.
Returns:
The validated job data dictionary with additional metadata.
Raises:
KeyError: If required fields 'name' or 'engine_name' are missing.
FileNotFoundError: If no valid project file can be found.
"""
loaded_project_local_path = None
# check for required keys
@@ -24,7 +84,7 @@ class JobImportHandler:
engine_name = new_job_data.get('engine_name')
if not job_name:
raise KeyError("Missing job name")
elif not engine_name:
if not engine_name:
raise KeyError("Missing engine name")
project_url = new_job_data.get('url', None)
@@ -43,28 +103,34 @@ class JobImportHandler:
raise FileNotFoundError("Cannot find any valid project paths")
# Prepare the local filepath
cleaned_path_name = os.path.splitext(referred_name)[0].replace(' ', '-')
job_dir = os.path.join(upload_directory, '-'.join(
[datetime.now().strftime("%Y.%m.%d_%H.%M.%S"), engine_name, cleaned_path_name]))
cleaned_path_name = job_name.replace(' ', '-')
folder_name = f"{cleaned_path_name}-{engine_name}-{datetime.now().strftime('%Y.%m.%d_%H.%M.%S')}"
job_dir = Path(upload_directory) / folder_name
os.makedirs(job_dir, exist_ok=True)
project_source_dir = os.path.join(job_dir, 'source')
project_source_dir = Path(job_dir) / 'source'
os.makedirs(project_source_dir, exist_ok=True)
# Move projects to their work directories
if uploaded_file and uploaded_file.filename:
loaded_project_local_path = os.path.join(project_source_dir, secure_filename(uploaded_file.filename))
uploaded_file.save(loaded_project_local_path)
logger.info(f"Transfer complete for {loaded_project_local_path.split(upload_directory)[-1]}")
elif project_url:
loaded_project_local_path = os.path.join(project_source_dir, referred_name)
shutil.move(downloaded_file_url, loaded_project_local_path)
logger.info(f"Download complete for {loaded_project_local_path.split(upload_directory)[-1]}")
elif local_path:
loaded_project_local_path = os.path.join(project_source_dir, referred_name)
shutil.copy(local_path, loaded_project_local_path)
logger.info(f"Import complete for {loaded_project_local_path.split(upload_directory)[-1]}")
# Handle file uploading
filename = secure_filename(uploaded_file.filename)
loaded_project_local_path = Path(project_source_dir) / filename
uploaded_file.save(str(loaded_project_local_path))
logger.info(f"Transfer complete for {loaded_project_local_path.relative_to(upload_directory)}")
if loaded_project_local_path.lower().endswith('.zip'):
elif project_url:
# Handle downloading project from a URL
loaded_project_local_path = Path(project_source_dir) / referred_name
shutil.move(str(downloaded_file_url), str(loaded_project_local_path))
logger.info(f"Download complete for {loaded_project_local_path.relative_to(upload_directory)}")
elif local_path:
# Handle local files
loaded_project_local_path = Path(project_source_dir) / referred_name
shutil.copy(str(local_path), str(loaded_project_local_path))
logger.info(f"Import complete for {loaded_project_local_path.relative_to(upload_directory)}")
if loaded_project_local_path.suffix == ".zip":
loaded_project_local_path = cls.process_zipped_project(loaded_project_local_path)
new_job_data["__loaded_project_local_path"] = loaded_project_local_path
@@ -72,13 +138,25 @@ class JobImportHandler:
return new_job_data
@staticmethod
def download_project_from_url(project_url):
def download_project_from_url(project_url: str):
"""Downloads a project file from the given URL.
Downloads the file from the specified URL to a temporary directory
with progress tracking. Returns the filename and temporary path.
Args:
project_url: The URL to download the project file from.
Returns:
A tuple of (filename, temp_file_path) if successful,
(None, None) if download fails.
"""
# This nested function is to handle downloading from a URL
logger.info(f"Downloading project from url: {project_url}")
referred_name = os.path.basename(project_url)
try:
response = requests.get(project_url, stream=True)
response = requests.get(project_url, stream=True, timeout=300)
if response.status_code == 200:
# Get the total file size from the "Content-Length" header
file_size = int(response.headers.get("Content-Length", 0))
@@ -101,30 +179,29 @@ class JobImportHandler:
return None, None
@staticmethod
def process_zipped_project(zip_path):
def process_zipped_project(zip_path: Path) -> Path:
"""
Processes a zipped project.
This method takes a path to a zip file, extracts its contents, and returns the path to the extracted project file.
If the zip file contains more than one project file or none, an error is raised.
This method takes a path to a zip file, extracts its contents, and returns the path to the extracted project
file. If the zip file contains more than one project file or none, an error is raised.
Args:
zip_path (str): The path to the zip file.
zip_path (Path): The path to the zip file.
Raises:
ValueError: If there's more than 1 project file or none in the zip file.
Returns:
str: The path to the main project file.
Path: The path to the main project file.
"""
work_path = os.path.dirname(zip_path)
work_path = zip_path.parent
try:
with zipfile.ZipFile(zip_path, 'r') as myzip:
myzip.extractall(work_path)
myzip.extractall(str(work_path))
project_files = [x for x in os.listdir(work_path) if os.path.isfile(os.path.join(work_path, x))]
project_files = [x for x in project_files if '.zip' not in x]
project_files = [p for p in work_path.iterdir() if p.is_file() and p.suffix.lower() != ".zip"]
logger.debug(f"Zip files: {project_files}")
@@ -134,12 +211,12 @@ class JobImportHandler:
# If there's more than 1 project file or none, raise an error
if len(project_files) != 1:
raise ValueError(f'Cannot find a valid project file in {os.path.basename(zip_path)}')
raise ValueError(f'Cannot find a valid project file in {zip_path.name}')
extracted_project_path = os.path.join(work_path, project_files[0])
extracted_project_path = work_path / project_files[0]
logger.info(f"Extracted zip file to {extracted_project_path}")
except (zipfile.BadZipFile, zipfile.LargeZipFile) as e:
logger.error(f"Error processing zip file: {e}")
raise ValueError(f"Error processing zip file: {e}")
raise ValueError(f'Error processing zip file: {e}') from e
return extracted_project_path
+40 -16
View File
@@ -3,6 +3,7 @@ import os
import subprocess
import threading
from pathlib import Path
from typing import Dict, Optional
from src.utilities.ffmpeg_helper import generate_thumbnail, save_first_frame
@@ -12,12 +13,20 @@ supported_image_formats = ['.jpg', '.png', '.exr', '.tif', '.tga', '.bmp', '.web
class PreviewManager:
_default_instance: Optional['PreviewManager'] = None
storage_path = None
_running_jobs = {}
storage_path: Optional[str] = None
def __init__(self) -> None:
self.storage_path = None
self._running_jobs: Dict = {}
@classmethod
def __generate_job_preview_worker(cls, job, replace_existing=False, max_width=480):
def _sync_class(cls) -> None:
if cls._default_instance is not None:
cls.storage_path = cls._default_instance.storage_path
def _generate_job_preview_worker(self, job, replace_existing=False, max_width=480):
# Determine best source file to use for thumbs
job_file_list = job.file_list()
@@ -33,8 +42,8 @@ class PreviewManager:
logger.warning(f"No valid image or video files found in files from job: {job}")
return
os.makedirs(cls.storage_path, exist_ok=True)
base_path = os.path.join(cls.storage_path, f"{job.id}-{preview_label}-{max_width}")
os.makedirs(self.storage_path, exist_ok=True)
base_path = os.path.join(self.storage_path, f"{job.id}-{preview_label}-{max_width}")
preview_video_path = base_path + '.mp4'
preview_image_path = base_path + '.jpg'
@@ -65,25 +74,23 @@ class PreviewManager:
except subprocess.CalledProcessError as e:
logger.error(f"Error generating video preview for {job}: {e}")
@classmethod
def update_previews_for_job(cls, job, replace_existing=False, wait_until_completion=False, timeout=None):
job_thread = cls._running_jobs.get(job.id)
def _update_previews_for_job(self, job, replace_existing=False, wait_until_completion=False, timeout=None):
job_thread = self._running_jobs.get(job.id)
if job_thread and job_thread.is_alive():
logger.debug(f'Preview generation job already running for {job}')
else:
job_thread = threading.Thread(target=cls.__generate_job_preview_worker, args=(job, replace_existing,))
job_thread = threading.Thread(target=self._generate_job_preview_worker, args=(job, replace_existing,))
job_thread.start()
cls._running_jobs[job.id] = job_thread
self._running_jobs[job.id] = job_thread
if wait_until_completion:
job_thread.join(timeout=timeout)
@classmethod
def get_previews_for_job(cls, job):
def _get_previews_for_job(self, job):
results = {}
try:
directory_path = Path(cls.storage_path)
directory_path = Path(self.storage_path)
preview_files_for_job = [f for f in directory_path.iterdir() if f.is_file() and f.name.startswith(job.id)]
for preview_filename in preview_files_for_job:
@@ -101,9 +108,8 @@ class PreviewManager:
pass
return results
@classmethod
def delete_previews_for_job(cls, job):
all_previews = cls.get_previews_for_job(job)
def _delete_previews_for_job(self, job):
all_previews = self.get_previews_for_job(job)
flattened_list = [item for sublist in all_previews.values() for item in sublist]
for preview in flattened_list:
try:
@@ -111,3 +117,21 @@ class PreviewManager:
os.remove(preview['filename'])
except OSError as e:
logger.error(f"Error removing preview '{preview.get('filename')}': {e}")
# --- Forwarders for backward compatibility ---
@classmethod
def update_previews_for_job(cls, job, replace_existing=False, wait_until_completion=False, timeout=None):
if cls._default_instance is not None:
cls._default_instance._update_previews_for_job(job, replace_existing, wait_until_completion, timeout)
@classmethod
def get_previews_for_job(cls, job):
if cls._default_instance is not None:
return cls._default_instance._get_previews_for_job(job)
return {}
@classmethod
def delete_previews_for_job(cls, job):
if cls._default_instance is not None:
cls._default_instance._delete_previews_for_job(job)
+76 -45
View File
@@ -3,6 +3,7 @@ import logging
import os
import threading
import time
from pathlib import Path
import requests
from requests_toolbelt.multipart import MultipartEncoder, MultipartEncoderMonitor
@@ -21,6 +22,8 @@ categories = [RenderStatus.RUNNING, RenderStatus.WAITING_FOR_SUBJOBS, RenderStat
logger = logging.getLogger()
OFFLINE_MAX = 4
JOB_UPLOAD_TIMEOUT = (10, 1800)
FILE_DOWNLOAD_TIMEOUT = (10, 1800)
class RenderServerProxy:
@@ -108,6 +111,11 @@ class RenderServerProxy:
return requests.get(f'http://{self.hostname}:{self.port}/api/{payload}', timeout=timeout,
headers={"X-API-Version": str(API_VERSION)})
def _post(self, payload, timeout=5, **kwargs):
from src.api.api_server import API_VERSION
return requests.post(f'http://{self.hostname}:{self.port}/api/{payload}', timeout=timeout,
headers={"X-API-Version": str(API_VERSION)}, **kwargs)
# --------------------------------------------
# Background Updates:
# --------------------------------------------
@@ -133,7 +141,7 @@ class RenderServerProxy:
if self.__offline_flags: # if we're offline, don't bother with the long poll
ignore_token = True
url = f'jobs_long_poll?token={self.__jobs_cache_token}' if (self.__jobs_cache_token and
url = f'jobs/long_poll?token={self.__jobs_cache_token}' if (self.__jobs_cache_token and
not ignore_token) else 'jobs'
status_result = self.request_data(url, timeout=timeout)
if status_result is not None:
@@ -152,13 +160,10 @@ class RenderServerProxy:
# Get System Info:
# --------------------------------------------
def get_all_jobs(self, timeout=5, ignore_token=False):
def get_jobs(self, timeout=5, ignore_token=False):
if not self.__update_in_background or ignore_token:
self.__update_job_cache(timeout, ignore_token)
return self.__jobs_cache.copy() if self.__jobs_cache else None
def get_data(self, timeout=5):
return self.request_data('full_status', timeout=timeout)
return self.__jobs_cache.copy()
def get_status(self):
status = self.request_data('status')
@@ -174,62 +179,70 @@ class RenderServerProxy:
# Get Job Info:
# --------------------------------------------
def get_job_info(self, job_id, timeout=5):
return self.request_data(f'job/{job_id}', timeout=timeout)
def get_job(self, job_id, timeout=5):
return self.request_data(f'jobs/{job_id}', timeout=timeout)
def get_job_files_list(self, job_id):
return self.request_data(f"job/{job_id}/file_list")
def get_job_files(self, job_id):
return self.request_data(f'jobs/{job_id}/files')
# --------------------------------------------
# Job Lifecycle:
# --------------------------------------------
def post_job_to_server(self, file_path, job_data, callback=None):
def create_job(self, file_path: Path, job_data, callback=None, timeout=JOB_UPLOAD_TIMEOUT):
"""
Posts a job to the server.
Args:
file_path (str): The path to the file to upload.
file_path (Path): The path to the file to upload.
job_data (dict): A dict of jobs data.
callback (function, optional): A callback function to call during the upload. Defaults to None.
timeout (float | tuple, optional): Requests timeout. Defaults to a 10-second connect timeout and
30-minute read timeout for large uploads.
Returns:
Response: The response from the server.
"""
# Check if file exists
if not os.path.exists(file_path):
if not file_path.exists():
raise FileNotFoundError(f"File not found: {file_path}")
# Bypass uploading file if posting to localhost
if self.is_localhost:
job_data['local_path'] = file_path
url = urljoin(f'http://{self.hostname}:{self.port}', '/api/add_job')
job_data['local_path'] = str(file_path)
url = urljoin(f'http://{self.hostname}:{self.port}', '/api/jobs')
headers = {'Content-Type': 'application/json'}
return requests.post(url, data=json.dumps(job_data), headers=headers)
return requests.post(url, data=json.dumps(job_data), headers=headers, timeout=timeout)
# Prepare the form data for remote host
with open(file_path, 'rb') as file:
encoder = MultipartEncoder({
'file': (os.path.basename(file_path), file, 'application/octet-stream'),
'file': (file_path.name, file, 'application/octet-stream'),
'json': (None, json.dumps(job_data), 'application/json'),
})
# Create a monitor that will track the upload progress
monitor = MultipartEncoderMonitor(encoder, callback) if callback else MultipartEncoderMonitor(encoder)
headers = {'Content-Type': monitor.content_type}
url = urljoin(f'http://{self.hostname}:{self.port}', '/api/add_job')
url = urljoin(f'http://{self.hostname}:{self.port}', '/api/jobs')
# Send the request with proper resource management
with requests.post(url, data=monitor, headers=headers) as response:
with requests.post(url, data=monitor, headers=headers, timeout=timeout) as response:
return response
def cancel_job(self, job_id, confirm=False):
return self.request_data(f'job/{job_id}/cancel?confirm={confirm}')
response = self._post(f'jobs/{job_id}/cancel', params={'confirm': confirm})
if response.ok:
self.__update_job_cache(timeout=5, ignore_token=True)
return response
def delete_job(self, job_id, confirm=False):
return self.request_data(f'job/{job_id}/delete?confirm={confirm}')
response = self._post(f'jobs/{job_id}/delete', params={'confirm': confirm})
if response.ok:
self.__update_job_cache(timeout=5, ignore_token=True)
return response
def send_subjob_update_notification(self, parent_id, subjob):
def send_subjob_update_notification(self, parent_id, subjob, timeout=5):
"""
Notifies the parent job of an update in a subjob.
@@ -240,21 +253,24 @@ class RenderServerProxy:
Returns:
Response: The response from the server.
"""
return requests.post(f'http://{self.hostname}:{self.port}/api/job/{parent_id}/send_subjob_update_notification',
json=subjob.json())
return requests.post(f'http://{self.hostname}:{self.port}/api/jobs/{parent_id}/subjob_update',
json=subjob.json(), timeout=timeout)
# --------------------------------------------
# Engines:
# --------------------------------------------
def is_engine_available(self, engine_name):
return self.request_data(f'{engine_name}/is_available')
def get_engine_for_filename(self, filename:str, timeout=5):
response = self.request(f'engines/for_filename?filename={os.path.basename(filename)}', timeout)
return response.text
def get_all_engines(self):
# todo: this doesnt work
return self.request_data('all_engines')
def get_engine_availability(self, engine_name:str, timeout=5):
return self.request_data(f'engines/{engine_name}/availability', timeout)
def get_engine_info(self, response_type='standard', timeout=5):
def get_engine_names(self, timeout=5):
return self.request_data('engines/names', timeout=timeout)
def get_engines(self, response_type='standard', timeout=5):
"""
Fetches engine information from the server.
@@ -265,39 +281,54 @@ class RenderServerProxy:
Returns:
dict: A dictionary containing the engine information.
"""
all_data = self.request_data(f"engine_info?response_type={response_type}", timeout=timeout)
all_data = self.request_data(f'engines?response_type={response_type}', timeout=timeout)
return all_data
def delete_engine(self, engine, version, system_cpu=None):
def get_engine(self, engine_name:str, response_type='standard', timeout=5):
"""
Sends a request to the server to delete a specific engine.
Fetches specific engine information from the server.
Args:
engine (str): The name of the engine to delete.
engine_name (str): Name of the engine
response_type (str, optional): Returns standard or full version of engine info
timeout (int, optional): The number of seconds to wait for a response from the server. Defaults to 5.
Returns:
dict: A dictionary containing the engine information.
"""
return self.request_data(f'engines/{engine_name}?response_type={response_type}', timeout)
def delete_engine_download(self, engine_name:str, version:str, system_os=None, cpu=None):
"""
Sends a request to the server to delete a specific engine download.
Args:
engine_name (str): The name of the engine to delete.
version (str): The version of the engine to delete.
system_cpu (str, optional): The system CPU type. Defaults to None.
system_os (str, optional): The system OS. Defaults to None.
cpu (str, optional): The system CPU type. Defaults to None.
Returns:
Response: The response from the server.
"""
form_data = {'engine': engine, 'version': version, 'system_cpu': system_cpu}
return requests.post(f'http://{self.hostname}:{self.port}/api/delete_engine', json=form_data)
form_data = {'engine': engine_name, 'version': version, 'system_os': system_os, 'cpu': cpu}
return self._post('engines/delete', json=form_data)
# --------------------------------------------
# Download Files:
# --------------------------------------------
def download_all_job_files(self, job_id, save_path):
url = f"http://{self.hostname}:{self.port}/api/job/{job_id}/download_all"
return self.__download_file_from_url(url, output_filepath=save_path)
def download_all_job_files(self, job_id, save_path, timeout=FILE_DOWNLOAD_TIMEOUT):
url = f'http://{self.hostname}:{self.port}/api/jobs/{job_id}/download_all'
return self.__download_file_from_url(url, output_filepath=save_path, timeout=timeout)
def download_job_file(self, job_id, job_filename, save_path):
url = f"http://{self.hostname}:{self.port}/api/job/{job_id}/download?filename={job_filename}"
return self.__download_file_from_url(url, output_filepath=save_path)
def download_job_file(self, job_id, job_filename, save_path, timeout=FILE_DOWNLOAD_TIMEOUT):
url = f'http://{self.hostname}:{self.port}/api/jobs/{job_id}/download?filename={job_filename}'
return self.__download_file_from_url(url, output_filepath=save_path, timeout=timeout)
@staticmethod
def __download_file_from_url(url, output_filepath):
with requests.get(url, stream=True) as r:
def __download_file_from_url(url, output_filepath, timeout=FILE_DOWNLOAD_TIMEOUT):
with requests.get(url, stream=True, timeout=timeout) as r:
r.raise_for_status()
with open(output_filepath, 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
+23
View File
@@ -0,0 +1,23 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Optional
if TYPE_CHECKING:
from src.api.preview_manager import PreviewManager
from src.distributed_job_manager import DistributedJobManager
from src.engines.engine_manager import EngineManager
from src.render_queue import RenderQueue
from src.utilities.config import Config
from src.utilities.zeroconf_server import ZeroconfServer
class ApplicationContext:
"""Holds all service instances. Single source of truth for wiring."""
def __init__(self) -> None:
self.config: Optional[Config] = None
self.engine_manager: Optional[EngineManager] = None
self.preview_manager: Optional[PreviewManager] = None
self.zeroconf_server: Optional[ZeroconfServer] = None
self.render_queue: Optional[RenderQueue] = None
self.distributed_job_manager: Optional[DistributedJobManager] = None
+84 -158
View File
@@ -3,6 +3,8 @@ import os
import socket
import threading
import time
from typing import Optional
from pathlib import Path
from plyer import notification
from pubsub import pub
@@ -20,47 +22,32 @@ logger = logging.getLogger()
class DistributedJobManager:
def __init__(self):
pass
_default_instance: Optional['DistributedJobManager'] = None
@classmethod
def subscribe_to_listener(cls):
"""
Subscribes the private class method '__local_job_status_changed' to the 'status_change' pubsub message.
This should be called once, typically during the initialization phase.
"""
pub.subscribe(cls.__local_job_status_changed, 'status_change')
pub.subscribe(cls.__local_job_frame_complete, 'frame_complete')
def _sync_class(cls) -> None:
if cls._default_instance is not None:
pass # no class-level attributes to sync
@classmethod
def __local_job_frame_complete(cls, job_id, frame_number, update_interval=5):
def __init__(self) -> None:
self.background_worker: Optional[threading.Thread] = None
"""
Responds to the 'frame_complete' pubsub message for local jobs.
Args:
job_id (str): The ID of the job that has changed status.
old_status (str): The previous status of the job.
new_status (str): The new (current) status of the job.
Note: Do not call directly. Instead, call via the 'frame_complete' pubsub message.
"""
def _subscribe_to_listener(self) -> None:
pub.subscribe(self._local_job_status_changed, 'status_change')
pub.subscribe(self._local_job_frame_complete, 'frame_complete')
def _local_job_frame_complete(self, job_id, frame_number, update_interval=5) -> None:
render_job = RenderQueue.job_with_id(job_id, none_ok=True)
if not render_job: # ignore jobs not in the queue
if not render_job:
return
logger.debug(f"Job {job_id} has completed frame #{frame_number}")
replace_existing_previews = (frame_number % update_interval) == 0
cls.__job_update_shared(render_job, replace_existing_previews)
self._job_update_shared(render_job, replace_existing_previews)
@classmethod
def __job_update_shared(cls, render_job, replace_existing_previews=False):
# update previews
def _job_update_shared(self, render_job, replace_existing_previews=False) -> None:
PreviewManager.update_previews_for_job(job=render_job, replace_existing=replace_existing_previews)
# notify parent to allow individual frames to be copied instead of waiting until the end
if render_job.parent:
parent_id, parent_hostname = render_job.parent.split('@')[0], render_job.parent.split('@')[-1]
try:
@@ -69,57 +56,41 @@ class DistributedJobManager:
except Exception as e:
logger.error(f"Error notifying parent {parent_hostname} about update in subjob {render_job.id}: {e}")
@classmethod
def __local_job_status_changed(cls, job_id, old_status, new_status):
"""
Responds to the 'status_change' pubsub message for local jobs.
If it's a child job, it notifies the parent job about the status change.
Args:
job_id (str): The ID of the job that has changed status.
old_status (str): The previous status of the job.
new_status (str): The new (current) status of the job.
Note: Do not call directly. Instead, call via the 'status_change' pubsub message.
"""
def _local_job_status_changed(self, job_id: str, old_status: str, new_status: str) -> None:
render_job = RenderQueue.job_with_id(job_id, none_ok=True)
if not render_job: # ignore jobs created but not yet added to queue
if not render_job:
return
logger.debug(f"Job {job_id} status change: {old_status} -> {new_status}")
self._job_update_shared(render_job, replace_existing_previews=(render_job.status == RenderStatus.COMPLETED))
cls.__job_update_shared(render_job, replace_existing_previews=(render_job.status == RenderStatus.COMPLETED))
# Handle children
if render_job.children:
if new_status in [RenderStatus.CANCELLED, RenderStatus.ERROR]: # Cancel children if necessary
if new_status in (RenderStatus.CANCELLED, RenderStatus.ERROR):
for child in render_job.children:
child_id, child_hostname = child.split('@')
RenderServerProxy(child_hostname).cancel_job(child_id, confirm=True)
# UI Notifications
try:
if new_status == RenderStatus.COMPLETED:
logger.debug("Show render complete notification")
notification.notify(
title='Render Job Complete',
message=f'{render_job.name} completed succesfully',
timeout=10 # Display time in seconds
timeout=10
)
elif new_status == RenderStatus.ERROR:
logger.debug("Show render error notification")
notification.notify(
title='Render Job Failed',
message=f'{render_job.name} failed rendering',
timeout=10 # Display time in seconds
timeout=10
)
elif new_status == RenderStatus.RUNNING:
logger.debug("Show render started notification")
notification.notify(
title='Render Job Started',
message=f'{render_job.name} started rendering',
timeout=10 # Display time in seconds
timeout=10
)
except Exception as e:
logger.debug(f"Unable to show UI notification: {e}")
@@ -128,34 +99,17 @@ class DistributedJobManager:
# Create Job
# --------------------------------------------
@classmethod
def create_render_job(cls, new_job_attributes, loaded_project_local_path):
"""Creates render jobs. Pass in dict of job_data and the local path to the project. It creates and returns a new
render job.
def _create_render_job(self, new_job_attributes: dict, loaded_project_local_path: Path):
requested_output_path = new_job_attributes.get('output_path')
output_filename = Path(str(requested_output_path)).name if requested_output_path else loaded_project_local_path.stem
Args:
new_job_attributes (dict): Dict of desired attributes for new job (frame count, renderer, output path, etc)
loaded_project_local_path (str): The local path to the loaded project.
Returns:
worker: Created job worker
"""
# get new output path in output_dir
output_path = new_job_attributes.get('output_path')
if not output_path:
loaded_project_filename = os.path.basename(loaded_project_local_path)
output_filename = os.path.splitext(loaded_project_filename)[0]
else:
output_filename = os.path.basename(output_path)
# Prepare output path
output_dir = os.path.join(os.path.dirname(os.path.dirname(loaded_project_local_path)), 'output')
output_path = os.path.join(output_dir, output_filename)
output_dir = loaded_project_local_path.parent.parent / "output"
if new_job_attributes.get('__use_output_subdir'):
output_dir = output_dir / output_filename
output_path = output_dir / output_filename
os.makedirs(output_dir, exist_ok=True)
logger.debug(f"New job output path: {output_path}")
# create & configure jobs
worker = EngineManager.create_worker(engine_name=new_job_attributes['engine_name'],
input_path=loaded_project_local_path,
output_path=output_path,
@@ -163,16 +117,15 @@ class DistributedJobManager:
args=new_job_attributes.get('args', {}),
parent=new_job_attributes.get('parent'),
name=new_job_attributes.get('name'))
worker.status = new_job_attributes.get("initial_status", worker.status) # todo: is this necessary?
worker.status = new_job_attributes.get("initial_status", worker.status)
worker.priority = int(new_job_attributes.get('priority', worker.priority))
worker.start_frame = int(new_job_attributes.get("start_frame", worker.start_frame))
worker.end_frame = int(new_job_attributes.get("end_frame", worker.end_frame))
worker.watchdog_timeout = Config.worker_process_timeout
worker.hostname = socket.gethostname()
# determine if we can / should split the job
if new_job_attributes.get("enable_split_jobs", False) and (worker.total_frames > 1) and not worker.parent:
cls.split_into_subjobs_async(worker, new_job_attributes, loaded_project_local_path)
self.split_into_subjobs_async(worker, new_job_attributes, loaded_project_local_path)
else:
worker.status = RenderStatus.NOT_STARTED
@@ -185,15 +138,7 @@ class DistributedJobManager:
# Handling Subjobs
# --------------------------------------------
@classmethod
def handle_subjob_update_notification(cls, local_job, subjob_data):
"""Responds to a notification from a remote subjob and the host requests any subsequent updates from the subjob.
Args:
local_job (BaseRenderWorker): The local parent job worker.
subjob_data (dict): Subjob data sent from the remote server.
"""
def _handle_subjob_update_notification(self, local_job, subjob_data: dict) -> None:
subjob_status = string_to_status(subjob_data['status'])
subjob_id = subjob_data['id']
subjob_hostname = subjob_data['hostname']
@@ -209,19 +154,10 @@ class DistributedJobManager:
if subjob_data['status'] == 'completed' and download_success:
local_job.children[subjob_key]['download_status'] = 'completed'
@classmethod
def wait_for_subjobs(cls, parent_job):
"""Check the status of subjobs and waits until they are all finished. Download rendered frames from subjobs
when they are completed.
Args:
parent_job: Worker object that has child jobs
Returns:
"""
def _wait_for_subjobs(self, parent_job) -> None:
logger.debug(f"Waiting for subjobs for job {parent_job}")
parent_job.status = RenderStatus.WAITING_FOR_SUBJOBS
statuses_to_download = [RenderStatus.CANCELLED, RenderStatus.ERROR, RenderStatus.COMPLETED]
statuses_to_download = (RenderStatus.CANCELLED, RenderStatus.ERROR, RenderStatus.COMPLETED)
def subjobs_not_downloaded():
return {k: v for k, v in parent_job.children.items() if 'download_status' not in v or
@@ -233,21 +169,17 @@ class DistributedJobManager:
sleep_counter = 0
while parent_job.status == RenderStatus.WAITING_FOR_SUBJOBS:
if sleep_counter % server_delay == 0: # only ping servers every x seconds
for child_key, subjob_cached_data in subjobs_not_downloaded().items():
if sleep_counter % server_delay == 0:
for child_key in subjobs_not_downloaded():
subjob_id = child_key.split('@')[0]
subjob_hostname = child_key.split('@')[-1]
# Fetch info from server and handle failing case
subjob_data = RenderServerProxy(subjob_hostname).get_job_info(subjob_id)
subjob_data = RenderServerProxy(subjob_hostname).get_job(subjob_id)
if not subjob_data:
logger.warning(f"No response from {subjob_hostname}")
# timeout / missing server situations
parent_job.children[child_key]['download_status'] = f'error: No response from {subjob_hostname}'
continue
# Update parent job cache but keep the download status
download_status = parent_job.children[child_key].get('download_status', None)
parent_job.children[child_key] = subjob_data
parent_job.children[child_key]['download_status'] = download_status
@@ -257,8 +189,7 @@ class DistributedJobManager:
f"{float(subjob_data.get('percent_complete')) * 100.0}%"
logger.debug(status_msg)
# Check if job is finished, but has not had files copied yet over yet
if download_status is None and subjob_data['file_count'] and status in statuses_to_download:
if download_status is None and subjob_data.get('file_count') and status in statuses_to_download:
try:
download_missing_frames_from_subjob(parent_job, subjob_id, subjob_hostname)
parent_job.children[child_key]['download_status'] = 'complete'
@@ -266,7 +197,6 @@ class DistributedJobManager:
logger.error(f"Error downloading missing frames from subjob: {e}")
parent_job.children[child_key]['download_status'] = 'error: {}'
# Any finished jobs not successfully downloaded at this point are skipped
if parent_job.children[child_key].get('download_status', None) is None and \
status in statuses_to_download:
logger.warning(f"Skipping waiting on downloading from subjob: {child_key}")
@@ -277,42 +207,22 @@ class DistributedJobManager:
f"{', '.join(list(subjobs_not_downloaded().keys()))}")
time.sleep(1)
sleep_counter += 1
else: # exit the loop
else:
parent_job.status = RenderStatus.RUNNING
# --------------------------------------------
# Creating Subjobs
# --------------------------------------------
@classmethod
def split_into_subjobs_async(cls, parent_worker, new_job_attributes, project_path, system_os=None):
# todo: I don't love this
def _split_into_subjobs_async(self, parent_worker, new_job_attributes, project_path, system_os=None) -> None:
parent_worker.status = RenderStatus.CONFIGURING
cls.background_worker = threading.Thread(target=cls.split_into_subjobs, args=(parent_worker, new_job_attributes,
project_path, system_os))
cls.background_worker.start()
self.background_worker = threading.Thread(target=self.split_into_subjobs, args=(
parent_worker, new_job_attributes, project_path, system_os))
self.background_worker.start()
@classmethod
def split_into_subjobs(cls, parent_worker, new_job_attributes, project_path, system_os=None, specific_servers=None):
"""
Splits a job into subjobs and distributes them among available servers.
This method checks the availability of servers, distributes the work among them, and creates subjobs on each
server. If a server is the local host, it adjusts the frame range of the parent job instead of creating a
subjob.
Args:
parent_worker (Worker): The parent job what we're creating the subjobs for.
new_job_attributes (dict): Dict of desired attributes for new job (frame count, engine, output path, etc)
project_path (str): The path to the project.
system_os (str, optional): Required OS. Default is any.
specific_servers (list, optional): List of specific servers to split work between. Defaults to all found.
"""
# Check availability
available_servers = specific_servers if specific_servers else cls.find_available_servers(parent_worker.engine_name,
system_os)
# skip if theres no external servers found
def split_into_subjobs(self, parent_worker, new_job_attributes, project_path, system_os=None, specific_servers=None) -> None:
available_servers = specific_servers if specific_servers else self.find_available_servers(
parent_worker.engine_name, system_os)
external_servers = [x for x in available_servers if x['hostname'] != parent_worker.hostname]
if not external_servers:
parent_worker.status = RenderStatus.NOT_STARTED
@@ -321,34 +231,29 @@ class DistributedJobManager:
logger.debug(f"Splitting into subjobs - Available servers: {[x['hostname'] for x in available_servers]}")
all_subjob_server_data = distribute_server_work(parent_worker.start_frame, parent_worker.end_frame, available_servers)
# Prep and submit these sub-jobs
logger.info(f"Job {parent_worker.id} split plan: {all_subjob_server_data}")
try:
for subjob_data in all_subjob_server_data:
subjob_hostname = subjob_data['hostname']
post_results = cls.__create_subjob(new_job_attributes, project_path, subjob_data, subjob_hostname,
post_results = self._create_subjob(new_job_attributes, project_path, subjob_data, subjob_hostname,
parent_worker)
if not post_results.ok:
ValueError(f"Failed to create subjob on {subjob_hostname}")
# save child info
submission_results = post_results.json()[0]
child_key = f"{submission_results['id']}@{subjob_hostname}"
parent_worker.children[child_key] = submission_results
# start subjobs
logger.debug(f"Created {len(all_subjob_server_data)} subjobs successfully")
parent_worker.name = f"{parent_worker.name} (Parent)"
parent_worker.status = RenderStatus.NOT_STARTED # todo: this won't work with scheduled starts
parent_worker.status = RenderStatus.NOT_STARTED
except Exception as e:
# cancel all the subjobs
logger.error(f"Failed to split job into subjobs: {e}")
logger.debug(f"Cancelling {len(all_subjob_server_data) - 1} attempted subjobs")
RenderServerProxy(parent_worker.hostname).cancel_job(parent_worker.id, confirm=True)
@staticmethod
def __create_subjob(new_job_attributes, project_path, server_data, server_hostname, parent_worker):
"""Convenience method to create subjobs for a parent worker"""
def _create_subjob(new_job_attributes: dict, project_path, server_data, server_hostname, parent_worker):
subjob = new_job_attributes.copy()
subjob['name'] = f"{parent_worker.name}[{server_data['frame_range'][0]}-{server_data['frame_range'][-1]}]"
subjob['parent'] = f"{parent_worker.id}@{parent_worker.hostname}"
@@ -357,7 +262,7 @@ class DistributedJobManager:
subjob['engine_version'] = parent_worker.engine_version
logger.debug(f"Posting subjob with frames {subjob['start_frame']}-"
f"{subjob['end_frame']} to {server_hostname}")
post_results = RenderServerProxy(server_hostname).post_job_to_server(
post_results = RenderServerProxy(server_hostname).create_job(
file_path=project_path, job_data=subjob)
return post_results
@@ -366,25 +271,46 @@ class DistributedJobManager:
# --------------------------------------------
@staticmethod
def find_available_servers(engine_name, system_os=None):
"""
Scan the Zeroconf network for currently available render servers supporting a specific engine.
:param engine_name: str, The engine type to search for
:param system_os: str, Restrict results to servers running a specific OS
:return: A list of dictionaries with each dict containing hostname and cpu_count of available servers
"""
from api.api_server import API_VERSION
available_servers = []
def find_available_servers(engine_name: str, system_os=None):
from src.api.api_server import API_VERSION
found_available_servers = []
for hostname in ZeroconfServer.found_hostnames():
host_properties = ZeroconfServer.get_hostname_properties(hostname)
if host_properties.get('api_version') == API_VERSION:
if not system_os or (system_os and system_os == host_properties.get('system_os')):
response = RenderServerProxy(hostname).is_engine_available(engine_name)
response = RenderServerProxy(hostname).get_engine_availability(engine_name)
if response and response.get('available', False):
available_servers.append(response)
found_available_servers.append(response)
return available_servers
return found_available_servers
# --- Forwarders for backward compatibility ---
@classmethod
def subscribe_to_listener(cls):
if cls._default_instance is not None:
cls._default_instance._subscribe_to_listener()
@classmethod
def create_render_job(cls, new_job_attributes, loaded_project_local_path):
if cls._default_instance is not None:
return cls._default_instance._create_render_job(new_job_attributes, loaded_project_local_path)
raise RuntimeError("DistributedJobManager is not initialized")
@classmethod
def handle_subjob_update_notification(cls, local_job, subjob_data):
if cls._default_instance is not None:
cls._default_instance._handle_subjob_update_notification(local_job, subjob_data)
@classmethod
def wait_for_subjobs(cls, parent_job):
if cls._default_instance is not None:
cls._default_instance._wait_for_subjobs(parent_job)
@classmethod
def split_into_subjobs_async(cls, parent_worker, new_job_attributes, project_path, system_os=None):
if cls._default_instance is not None:
cls._default_instance._split_into_subjobs_async(parent_worker, new_job_attributes, project_path, system_os)
if __name__ == '__main__':
+1 -1
View File
@@ -70,7 +70,7 @@ class BlenderDownloader(EngineDownloader):
@staticmethod
def __find_LTS_versions():
response = requests.get('https://www.blender.org/download/lts/')
response = requests.get('https://www.blender.org/download/lts/', timeout=5)
response.raise_for_status()
lts_pattern = r'https://www.blender.org/download/lts/(\d+-\d+)/'
+15 -12
View File
@@ -1,8 +1,8 @@
import json
import re
from pathlib import Path
from src.engines.core.base_engine import *
from src.utilities.misc_helper import system_safe_path
logger = logging.getLogger()
@@ -24,10 +24,12 @@ class Blender(BaseRenderEngine):
from src.engines.blender.blender_worker import BlenderRenderWorker
return BlenderRenderWorker
@staticmethod
def ui_options(system_info):
from src.engines.blender.blender_ui import BlenderUI
return BlenderUI.get_options(system_info)
def ui_options(self):
options = [
{'name': 'engine', 'options': self.supported_render_engines()},
{'name': 'render_device', 'options': ['Any', 'GPU', 'CPU']},
]
return options
def supported_extensions(self):
return ['blend']
@@ -87,7 +89,7 @@ class Blender(BaseRenderEngine):
scene_info = {}
try:
script_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'scripts', 'get_file_info.py')
results = self.run_python_script(project_path=project_path, script_path=system_safe_path(script_path),
results = self.run_python_script(project_path=project_path, script_path=Path(script_path),
timeout=timeout)
result_text = results.stdout.decode()
for line in result_text.splitlines():
@@ -108,7 +110,7 @@ class Blender(BaseRenderEngine):
try:
logger.info(f"Starting to pack Blender file: {project_path}")
script_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'scripts', 'pack_project.py')
results = self.run_python_script(project_path=project_path, script_path=system_safe_path(script_path),
results = self.run_python_script(project_path=project_path, script_path=Path(script_path),
timeout=timeout)
result_text = results.stdout.decode()
@@ -117,7 +119,7 @@ class Blender(BaseRenderEngine):
# report any missing textures
not_found = re.findall("(Unable to pack file, source path .*)\n", result_text)
for err in not_found:
logger.error(err)
raise ChildProcessError(err)
p = re.compile('Saved to: (.*)\n')
match = p.search(result_text)
@@ -125,6 +127,7 @@ class Blender(BaseRenderEngine):
new_path = os.path.join(dir_name, match.group(1).strip())
logger.info(f'Blender file packed successfully to {new_path}')
return new_path
return project_path
except Exception as e:
msg = f'Error packing .blend file: {e}'
logger.error(msg)
@@ -164,13 +167,13 @@ class Blender(BaseRenderEngine):
return options
def system_info(self):
return {'render_devices': self.get_render_devices()}
return {'render_devices': self.get_render_devices(), 'engines': self.supported_render_engines()}
def get_render_devices(self):
script_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'scripts', 'get_system_info.py')
results = self.run_python_script(script_path=script_path)
output = results.stdout.decode()
match = re.search(r"GPU DATA:(\[[\s\S]*\])", output)
match = re.search(r"GPU DATA:(\[.*?\])", output, re.DOTALL)
if match:
gpu_data_json = match.group(1)
gpus_info = json.loads(gpu_data_json)
@@ -179,7 +182,7 @@ class Blender(BaseRenderEngine):
logger.error("GPU data not found in the output.")
def supported_render_engines(self):
engine_output = subprocess.run([self.engine_path(), '-E', 'help'], timeout=SUBPROCESS_TIMEOUT,
engine_output = subprocess.run([self.engine_path(), '-b', '-E', 'help'], timeout=SUBPROCESS_TIMEOUT,
capture_output=True, creationflags=_creationflags).stdout.decode('utf-8').strip()
render_engines = [x.strip() for x in engine_output.split('Blender Engine Listing:')[-1].strip().splitlines()]
return render_engines
@@ -190,5 +193,5 @@ class Blender(BaseRenderEngine):
if __name__ == "__main__":
x = Blender().get_render_devices()
x = Blender().system_info()
print(x)
-9
View File
@@ -1,9 +0,0 @@
class BlenderUI:
@staticmethod
def get_options(system_info):
options = [
{'name': 'engine', 'options': system_info.get('engines', [])},
{'name': 'render_device', 'options': ['Any', 'GPU', 'CPU']},
]
return options
+23 -9
View File
@@ -18,14 +18,13 @@ class BlenderRenderWorker(BaseRenderWorker):
self.__frame_percent_complete = 0.0
# Scene Info
self.scene_info = Blender(engine_path).get_project_info(input_path)
self.start_frame = int(self.scene_info.get('start_frame', 1))
self.end_frame = int(self.scene_info.get('end_frame', self.start_frame))
self.project_length = (self.end_frame - self.start_frame) + 1
self.scene_info = {}
self.current_frame = -1
def generate_worker_subprocess(self):
self.scene_info = Blender(self.engine_path).get_project_info(self.input_path)
cmd = [self.engine_path]
if self.args.get('background', True): # optionally run render not in background
cmd.append('-b')
@@ -41,10 +40,16 @@ class BlenderRenderWorker(BaseRenderWorker):
cmd.append('--python-expr')
python_exp = 'import bpy; bpy.context.scene.render.use_overwrite = False;'
# Setup Custom Resolution
if self.args.get('resolution'):
res = self.args.get('resolution')
python_exp += 'bpy.context.scene.render.resolution_percentage = 100;'
python_exp += f'bpy.context.scene.render.resolution_x={res[0]}; bpy.context.scene.render.resolution_y={res[1]};'
# Setup Custom Camera
custom_camera = self.args.get('camera', None)
if custom_camera:
python_exp = python_exp + f"bpy.context.scene.camera = bpy.data.objects['{custom_camera}'];"
python_exp += f"bpy.context.scene.camera = bpy.data.objects['{custom_camera}'];"
# Set Render Device for Cycles (gpu/cpu/any)
if blender_engine == 'CYCLES':
@@ -85,11 +90,15 @@ class BlenderRenderWorker(BaseRenderWorker):
def _parse_stdout(self, line):
pattern = re.compile(
cycles_pattern = re.compile(
r'Fra:(?P<frame>\d*).*Mem:(?P<memory>\S+).*Time:(?P<time>\S+)(?:.*Remaining:)?(?P<remaining>\S*)')
found = pattern.search(line)
if found:
stats = found.groupdict()
cycles_match = cycles_pattern.search(line)
eevee_pattern = re.compile(
r"Rendering\s+(?P<current>\d+)\s*/\s*(?P<total>\d+)\s+samples"
)
eevee_match = eevee_pattern.search(line)
if cycles_match:
stats = cycles_match.groupdict()
memory_use = stats['memory']
time_elapsed = stats['time']
time_remaining = stats['remaining'] or 'Unknown'
@@ -104,6 +113,11 @@ class BlenderRenderWorker(BaseRenderWorker):
logger.debug(
'Frame:{0} | Mem:{1} | Time:{2} | Remaining:{3}'.format(self.current_frame, memory_use,
time_elapsed, time_remaining))
elif eevee_match:
self.__frame_percent_complete = int(eevee_match.groups()[0]) / int(eevee_match.groups()[1])
logger.debug(f'Frame:{self.current_frame} | Samples:{eevee_match.groups()[0]}/{eevee_match.groups()[1]}')
elif "Rendering frame" in line: # used for eevee
self.current_frame = int(line.split("Rendering frame")[-1].strip())
elif "file doesn't exist" in line.lower():
self.log_error(line, halt_render=True)
elif line.lower().startswith('error'):
@@ -1,6 +1,10 @@
import bpy
import json
# Force CPU rendering
bpy.context.preferences.addons["cycles"].preferences.compute_device_type = "NONE"
bpy.context.scene.cycles.device = "CPU"
# Ensure Cycles is available
bpy.context.preferences.addons['cycles'].preferences.get_devices()
+63 -23
View File
@@ -2,9 +2,10 @@ import logging
import os
import platform
import subprocess
from typing import Optional, List, Dict, Any, Type
logger = logging.getLogger()
SUBPROCESS_TIMEOUT = 5
SUBPROCESS_TIMEOUT = 10
class BaseRenderEngine(object):
@@ -17,14 +18,23 @@ class BaseRenderEngine(object):
executable on different operating systems or environments.
"""
install_paths = []
install_paths: List[str] = []
binary_names: Dict[str, str] = {}
# --------------------------------------------
# Required Overrides for Subclasses:
# --------------------------------------------
def __init__(self, custom_path=None):
self.custom_engine_path = custom_path
def __init__(self, custom_path: Optional[str] = None) -> None:
"""Initialize the render engine.
Args:
custom_path: Optional custom path to the engine executable.
Raises:
FileNotFoundError: If the engine executable cannot be found.
"""
self.custom_engine_path: Optional[str] = custom_path
if not self.engine_path() or not os.path.exists(self.engine_path()):
raise FileNotFoundError(f"Cannot find path to engine for {self.name()} instance: {self.engine_path()}")
@@ -32,7 +42,7 @@ class BaseRenderEngine(object):
logger.warning(f"Path is not executable. Setting permissions to 755 for {self.engine_path()}")
os.chmod(self.engine_path(), 0o755)
def version(self):
def version(self) -> str:
"""Return the version number as a string.
Returns:
@@ -43,7 +53,7 @@ class BaseRenderEngine(object):
"""
raise NotImplementedError(f"version not implemented for {self.__class__.__name__}")
def get_project_info(self, project_path, timeout=10):
def get_project_info(self, project_path: str, timeout: int = 10) -> Dict[str, Any]:
"""Extracts detailed project information from the given project path.
Args:
@@ -59,7 +69,7 @@ class BaseRenderEngine(object):
raise NotImplementedError(f"get_project_info not implemented for {self.__class__.__name__}")
@classmethod
def get_output_formats(cls):
def get_output_formats(cls) -> List[str]:
"""Returns a list of available output formats supported by the engine.
Returns:
@@ -68,21 +78,22 @@ class BaseRenderEngine(object):
raise NotImplementedError(f"get_output_formats not implemented for {cls.__name__}")
@staticmethod
def worker_class(): # override when subclassing to link worker class
def worker_class() -> Type[Any]: # override when subclassing to link worker class
raise NotImplementedError("Worker class not implemented")
# --------------------------------------------
# Optional Overrides for Subclasses:
# --------------------------------------------
def supported_extensions(self):
"""
def supported_extensions(self) -> List[str]:
"""Return a list of file extensions supported by this engine.
Returns:
list[str]: list of supported extensions
List[str]: List of supported file extensions (e.g., ['.blend', '.mp4']).
"""
return []
def get_help(self):
def get_help(self) -> str:
"""Retrieves the help documentation for the engine.
This method runs the engine's help command (default: '-h') and captures the output.
@@ -102,7 +113,7 @@ class BaseRenderEngine(object):
timeout=SUBPROCESS_TIMEOUT, creationflags=creationflags).decode('utf-8')
return help_doc
def system_info(self):
def system_info(self) -> Dict[str, Any]:
"""Return additional information about the system specfic to the engine (configured GPUs, render engines, etc)
Returns:
@@ -110,7 +121,7 @@ class BaseRenderEngine(object):
"""
return {}
def perform_presubmission_tasks(self, project_path):
def perform_presubmission_tasks(self, project_path: str) -> str:
"""Perform any pre-submission tasks on a project file before uploading it to a server (pack textures, etc.)
Override this method to:
@@ -126,31 +137,60 @@ class BaseRenderEngine(object):
"""
return project_path
def get_arguments(self):
pass
def get_arguments(self) -> Dict[str, Any]:
"""Return command-line arguments for this engine.
Returns:
Dict[str, Any]: Dictionary of argument specifications.
"""
return {}
@staticmethod
def downloader(): # override when subclassing if using a downloader class
def downloader() -> Optional[Any]:
"""Return the downloader class for this engine.
Returns:
Optional[Any]: Downloader class instance, or None if no downloader is used.
"""
return None
@staticmethod
def ui_options(system_info): # override to return options for ui
def ui_options(self) -> Dict[str, Any]:
"""Return UI configuration options for this engine.
Returns:
Dict[str, Any]: Dictionary of UI options and their configurations.
"""
return {}
# --------------------------------------------
# Do Not Override These Methods:
# --------------------------------------------
def engine_path(self):
def engine_path(self) -> Optional[str]:
"""Return the path to the engine executable.
Returns:
Optional[str]: Path to the engine executable, or None if not found.
"""
return self.custom_engine_path or self.default_engine_path()
@classmethod
def name(cls):
def name(cls) -> str:
"""Return the name of this engine.
Returns:
str: Engine name in lowercase.
"""
return str(cls.__name__).lower()
@classmethod
def default_engine_path(cls):
path = None
def default_engine_path(cls) -> Optional[str]:
"""Find the default path to the engine executable.
Returns:
Optional[str]: Default path to the engine executable, or None if not found.
"""
path: Optional[str] = None
try: # Linux and macOS
path = subprocess.check_output(['which', cls.name()], timeout=SUBPROCESS_TIMEOUT).decode('utf-8').strip()
except (subprocess.CalledProcessError, FileNotFoundError):
+99 -10
View File
@@ -8,6 +8,7 @@ import subprocess
import threading
import time
from datetime import datetime
from typing import Optional, Dict, Any, List, Union
import psutil
from pubsub import pub
@@ -36,7 +37,6 @@ class BaseRenderWorker(Base):
engine_version = Column(String)
engine_path = Column(String)
priority = Column(Integer)
project_length = Column(Integer)
start_frame = Column(Integer)
end_frame = Column(Integer, nullable=True)
parent = Column(String, nullable=True)
@@ -50,9 +50,25 @@ class BaseRenderWorker(Base):
# Required Overrides for Subclasses:
# --------------------------------------------
def __init__(self, input_path, output_path, engine_path, priority=2, args=None, ignore_extensions=True, parent=None,
name=None):
def __init__(self, input_path: str, output_path: str, engine_path: str, priority: int = 2,
args: Optional[Dict[str, Any]] = None, ignore_extensions: bool = True,
parent: Optional[str] = None, name: Optional[str] = None) -> None:
"""Initialize a render worker.
Args:
input_path: Path to the input project file.
output_path: Path where output files will be saved.
engine_path: Path to the render engine executable.
priority: Job priority level (default: 2).
args: Additional arguments for the render job.
ignore_extensions: Whether to ignore file extension validation.
parent: Parent job ID for distributed jobs.
name: Custom name for the job.
Raises:
ValueError: If file extension is not supported.
NotImplementedError: If engine is not defined.
"""
if not ignore_extensions:
if not any(ext in input_path for ext in self.engine.supported_extensions()):
err_meg = f"Cannot find valid project with supported file extension for '{self.engine.name()}'"
@@ -62,6 +78,7 @@ class BaseRenderWorker(Base):
raise NotImplementedError(f"Engine not defined for {self.__class__.__name__}")
def generate_id():
"""Generate a unique job ID."""
import uuid
return str(uuid.uuid4()).split('-')[0]
@@ -83,7 +100,6 @@ class BaseRenderWorker(Base):
self.maximum_attempts = 3
# Frame Ranges
self.project_length = 0 # is this necessary?
self.current_frame = 0
self.start_frame = 0
self.end_frame = None
@@ -105,15 +121,15 @@ class BaseRenderWorker(Base):
self.__last_output_time = None
self.watchdog_timeout = 120
def generate_worker_subprocess(self):
def generate_worker_subprocess(self) -> List[str]:
"""Generate a return a list of the command line arguments necessary to perform requested job
Returns:
list[str]: list of command line arguments
"""
raise NotImplementedError("generate_worker_subprocess not implemented")
def _parse_stdout(self, line):
def _parse_stdout(self, line: str) -> None:
"""Parses a line of standard output from the engine.
This method should be overridden in a subclass to implement the logic for processing
@@ -135,13 +151,18 @@ class BaseRenderWorker(Base):
# Optional Overrides for Subclasses:
# --------------------------------------------
def percent_complete(self):
def percent_complete(self) -> float:
"""Return the completion percentage of this job.
Returns:
float: Completion percentage between 0.0 and 1.0.
"""
# todo: fix this
if self.status == RenderStatus.COMPLETED:
return 1.0
return 0
def post_processing(self):
def post_processing(self) -> None:
"""Override to perform any engine-specific postprocessing"""
pass
@@ -150,11 +171,16 @@ class BaseRenderWorker(Base):
# --------------------------------------------
def __repr__(self):
"""Return string representation of the worker.
Returns:
str: String representation showing job details.
"""
return f"<Job id:{self.id} p{self.priority} {self.engine_name}-{self.engine_version} '{self.name}' status:{self.status.value}>"
@property
def total_frames(self):
return (self.end_frame or self.project_length) - self.start_frame + 1
return max(self.end_frame, 1) - self.start_frame + 1
@property
def status(self):
@@ -178,6 +204,14 @@ class BaseRenderWorker(Base):
pub.sendMessage('frame_complete', job_id=self.id, frame_number=self.current_frame)
def generate_subprocess(self):
"""Generate subprocess command arguments.
Returns:
List[str]: List of command line arguments for the subprocess.
Raises:
ValueError: If argument conflicts are detected.
"""
# Convert raw args from string if available and catch conflicts
generated_args = [str(x) for x in self.generate_worker_subprocess()]
generated_args_flags = [x for x in generated_args if x.startswith('-')]
@@ -188,6 +222,11 @@ class BaseRenderWorker(Base):
return generated_args
def get_raw_args(self):
"""Parse raw command line arguments from args dictionary.
Returns:
Optional[List[str]]: Parsed raw arguments, or None if no raw args.
"""
raw_args_string = self.args.get('raw', '')
raw_args = None
if raw_args_string:
@@ -196,12 +235,20 @@ class BaseRenderWorker(Base):
return raw_args
def log_path(self):
"""Generate the log file path for this job.
Returns:
str: Path to the log file.
"""
filename = (self.name or os.path.basename(self.input_path)) + '_' + \
self.date_created.strftime("%Y.%m.%d_%H.%M.%S") + '.log'
return os.path.join(os.path.dirname(self.input_path), filename)
def start(self):
"""Start the render job.
Validates input paths and engine availability, then starts the render thread.
"""
if self.status not in [RenderStatus.SCHEDULED, RenderStatus.NOT_STARTED, RenderStatus.CONFIGURING]:
logger.error(f"Trying to start job with status: {self.status}")
return
@@ -431,17 +478,33 @@ class BaseRenderWorker(Base):
logger.error(f"Error stopping the process: {e}")
def is_running(self):
"""Check if the render job is currently running.
Returns:
bool: True if the job is running, False otherwise.
"""
if hasattr(self, '__thread'):
return self.__thread.is_alive()
return False
def log_error(self, error_line, halt_render=False):
"""Log an error and optionally halt the render.
Args:
error_line: Error message to log.
halt_render: Whether to stop the render due to this error.
"""
logger.error(error_line)
self.errors.append(error_line)
if halt_render:
self.stop(is_error=True)
def stop(self, is_error=False):
"""Stop the render job.
Args:
is_error: Whether this stop is due to an error.
"""
logger.debug(f"Stopping {self}")
# cleanup status
@@ -459,9 +522,19 @@ class BaseRenderWorker(Base):
self.__thread.join(timeout=5)
def time_elapsed(self):
"""Get elapsed time for this job.
Returns:
str: Formatted time elapsed string.
"""
return get_time_elapsed(self.start_time, self.end_time)
def file_list(self):
"""Get list of output files for this job.
Returns:
List[str]: List of output file paths.
"""
try:
job_dir = os.path.dirname(self.output_path)
file_list = [
@@ -475,6 +548,11 @@ class BaseRenderWorker(Base):
return []
def json(self):
"""Convert worker to JSON-serializable dictionary.
Returns:
Dict[str, Any]: Dictionary representation of worker data.
"""
job_dict = {
'id': self.id,
'name': self.name,
@@ -504,8 +582,10 @@ class BaseRenderWorker(Base):
# convert to json and back to auto-convert dates to iso format
def date_serializer(o):
"""Serialize datetime objects to ISO format."""
if isinstance(o, datetime):
return o.isoformat()
return None
json_convert = json.dumps(job_dict, default=date_serializer)
worker_json = json.loads(json_convert)
@@ -513,6 +593,15 @@ class BaseRenderWorker(Base):
def timecode_to_frames(timecode, frame_rate):
"""Convert timecode string to frame number.
Args:
timecode: Timecode in format HH:MM:SS:FF.
frame_rate: Frames per second.
Returns:
int: Frame number corresponding to timecode.
"""
e = [int(x) for x in timecode.split(':')]
seconds = (((e[0] * 60) + e[1] * 60) + e[2])
frames = (seconds * frame_rate) + e[-1] + 1
+249 -168
View File
@@ -3,97 +3,96 @@ import os
import shutil
import threading
import concurrent.futures
from pathlib import Path
from typing import Type, List, Dict, Any, Optional
from src.engines.core.base_engine import BaseRenderEngine
from src.engines.blender.blender_engine import Blender
from src.engines.ffmpeg.ffmpeg_engine import FFMPEG
from src.utilities.misc_helper import system_safe_path, current_system_os, current_system_cpu
from src.utilities.misc_helper import current_system_os, current_system_cpu
logger = logging.getLogger()
ENGINE_CLASSES = [Blender, FFMPEG]
class EngineManager:
"""Class that manages different versions of installed render engines and handles fetching and downloading new versions,
if possible.
"""
engines_path = None
download_tasks = []
_default_instance: Optional['EngineManager'] = None
engines_path: Optional[str] = None
download_tasks: List[Any] = []
def __init__(self) -> None:
self.engines_path: Optional[str] = None
self.download_tasks: List[Any] = []
@classmethod
def _sync_class(cls) -> None:
if cls._default_instance is not None:
cls.engines_path = cls._default_instance.engines_path
cls.download_tasks = cls._default_instance.download_tasks
@staticmethod
def supported_engines():
return [Blender, FFMPEG]
def supported_engines() -> list[type[BaseRenderEngine]]:
return ENGINE_CLASSES
@classmethod
def downloadable_engines(cls):
return [engine for engine in cls.supported_engines() if hasattr(engine, "downloader") and engine.downloader()]
# --- Installed Engines ---
@classmethod
def active_downloads(cls) -> list:
return [x for x in cls.download_tasks if x.is_alive()]
def _engine_class_for_project_path(self, path: str) -> Type[BaseRenderEngine]:
_, extension = os.path.splitext(path)
extension = extension.lower().strip('.')
for engine_class in self.supported_engines():
engine = self.get_latest_engine_instance(engine_class)
if extension in engine.supported_extensions():
return engine_class
undefined_renderer_support = [x for x in self.supported_engines() if not self.get_latest_engine_instance(x).supported_extensions()]
return undefined_renderer_support[0]
@classmethod
def engine_with_name(cls, engine_name):
for obj in cls.supported_engines():
def _engine_class_with_name(self, engine_name: str) -> Optional[Type[BaseRenderEngine]]:
for obj in self.supported_engines():
if obj.name().lower() == engine_name.lower():
return obj
return None
@classmethod
def update_all_engines(cls):
for engine in cls.downloadable_engines():
update_available = cls.is_engine_update_available(engine)
if update_available:
update_available['name'] = engine.name()
cls.download_engine(engine.name(), update_available['version'], background=True)
def _get_latest_engine_instance(self, engine_class: Type[BaseRenderEngine]) -> BaseRenderEngine:
newest = self.newest_installed_engine_data(engine_class.name())
engine = engine_class(newest["path"])
return engine
@classmethod
def get_engines(cls, filter_name=None, include_corrupt=False, ignore_system=False):
if not cls.engines_path:
def _get_installed_engine_data(self, filter_name: Optional[str] = None, include_corrupt: bool = False,
ignore_system: bool = False) -> List[Dict[str, Any]]:
if not self.engines_path:
raise FileNotFoundError("Engine path is not set")
# Parse downloaded engine directory
results = []
try:
all_items = os.listdir(cls.engines_path)
all_directories = [item for item in all_items if os.path.isdir(os.path.join(cls.engines_path, item))]
keys = ["engine", "version", "system_os", "cpu"] # Define keys for result dictionary
all_items = os.listdir(self.engines_path)
all_directories = [item for item in all_items if os.path.isdir(os.path.join(self.engines_path, item))]
keys = ["engine", "version", "system_os", "cpu"]
for directory in all_directories:
# Split directory name into segments
segments = directory.split('-')
# Create a dictionary mapping keys to corresponding segments
result_dict = {keys[i]: segments[i] for i in range(min(len(keys), len(segments)))}
result_dict['type'] = 'managed'
# Initialize binary_name with engine name
binary_name = result_dict['engine'].lower()
# Determine the correct binary name based on the engine and system_os
eng = cls.engine_with_name(result_dict['engine'])
binary_name = eng.binary_names.get(result_dict['system_os'], binary_name)
eng = self.engine_class_with_name(result_dict['engine'])
if eng:
binary_name = eng.binary_names.get(result_dict['system_os'], binary_name)
# Find the path to the binary file
path = next(
(os.path.join(root, binary_name) for root, _, files in
os.walk(system_safe_path(os.path.join(cls.engines_path, directory))) if binary_name in files),
None
)
search_root = Path(self.engines_path) / directory
match = next((p for p in search_root.rglob(binary_name) if p.is_file()), None)
path = str(match) if match else None
result_dict['path'] = path
# fetch version number from binary - helps detect corrupted downloads - disabled due to perf issues
# binary_version = eng(path).version()
# if not binary_version:
# logger.warning(f"Possible corrupt {eng.name()} {result_dict['version']} install detected: {path}")
# if not include_corrupt:
# continue
# result_dict['version'] = binary_version or 'error'
# Add the result dictionary to results if it matches the filter_name or if no filter is applied
if not filter_name or filter_name == result_dict['engine']:
results.append(result_dict)
except FileNotFoundError as e:
logger.warning(f"Cannot find local engines download directory: {e}")
# add system installs to this list - use bg thread because it can be slow
def fetch_engine_details(eng, include_corrupt=False):
version = eng().version()
if not version and not include_corrupt:
@@ -112,7 +111,7 @@ class EngineManager:
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = {
executor.submit(fetch_engine_details, eng, include_corrupt): eng.name()
for eng in cls.supported_engines()
for eng in self.supported_engines()
if eng.default_engine_path() and (not filter_name or filter_name == eng.name())
}
@@ -123,114 +122,57 @@ class EngineManager:
return results
@classmethod
def all_versions_for_engine(cls, engine_name, include_corrupt=False, ignore_system=False):
versions = cls.get_engines(filter_name=engine_name, include_corrupt=include_corrupt, ignore_system=ignore_system)
# --- Check for Updates ---
def _update_all_engines(self) -> None:
for engine in self.downloadable_engines():
update_available = self.is_engine_update_available(engine)
if update_available:
update_available['name'] = engine.name()
self.download_engine(engine.name(), update_available['version'], background=True)
def _all_version_data_for_engine(self, engine_name: str, include_corrupt=False, ignore_system=False) -> list:
versions = self.get_installed_engine_data(filter_name=engine_name, include_corrupt=include_corrupt, ignore_system=ignore_system)
sorted_versions = sorted(versions, key=lambda x: x['version'], reverse=True)
return sorted_versions
@classmethod
def newest_engine_version(cls, engine, system_os=None, cpu=None, ignore_system=None):
def _newest_installed_engine_data(self, engine_name: str, system_os=None, cpu=None, ignore_system=None) -> list:
system_os = system_os or current_system_os()
cpu = cpu or current_system_cpu()
try:
filtered = [x for x in cls.all_versions_for_engine(engine, ignore_system=ignore_system)
filtered = [x for x in self.all_version_data_for_engine(engine_name, ignore_system=ignore_system)
if x['system_os'] == system_os and x['cpu'] == cpu]
return filtered[0]
except IndexError:
logger.error(f"Cannot find newest engine version for {engine}-{system_os}-{cpu}")
return None
logger.error(f"Cannot find newest engine version for {engine_name}-{system_os}-{cpu}")
return []
@classmethod
def is_version_downloaded(cls, engine, version, system_os=None, cpu=None, ignore_system=False):
def _is_version_installed(self, engine_name: str, version: str, system_os=None, cpu=None, ignore_system=False):
system_os = system_os or current_system_os()
cpu = cpu or current_system_cpu()
filtered = [x for x in cls.get_engines(filter_name=engine, ignore_system=ignore_system) if
filtered = [x for x in self.get_installed_engine_data(filter_name=engine_name, ignore_system=ignore_system) if
x['system_os'] == system_os and x['cpu'] == cpu and x['version'] == version]
return filtered[0] if filtered else False
@classmethod
def version_is_available_to_download(cls, engine, version, system_os=None, cpu=None):
def _version_is_available_to_download(self, engine_name: str, version, system_os=None, cpu=None):
try:
downloader = cls.engine_with_name(engine).downloader()
downloader = self.engine_class_with_name(engine_name).downloader()
return downloader.version_is_available_to_download(version=version, system_os=system_os, cpu=cpu)
except Exception as e:
logger.debug(f"Exception in version_is_available_to_download: {e}")
return None
@classmethod
def find_most_recent_version(cls, engine=None, system_os=None, cpu=None, lts_only=False):
def _find_most_recent_version(self, engine_name: str, system_os=None, cpu=None, lts_only=False) -> dict:
try:
downloader = cls.engine_with_name(engine).downloader()
downloader = self.engine_class_with_name(engine_name).downloader()
return downloader.find_most_recent_version(system_os=system_os, cpu=cpu)
except Exception as e:
logger.debug(f"Exception in find_most_recent_version: {e}")
return None
return {}
@classmethod
def get_existing_download_task(cls, engine, version, system_os=None, cpu=None):
for task in cls.download_tasks:
task_parts = task.name.split('-')
task_engine, task_version, task_system_os, task_cpu = task_parts[:4]
if engine == task_engine and version == task_version:
if system_os in (task_system_os, None) and cpu in (task_cpu, None):
return task
return None
@classmethod
def download_engine(cls, engine, version, system_os=None, cpu=None, background=False, ignore_system=False):
engine_to_download = cls.engine_with_name(engine)
existing_task = cls.get_existing_download_task(engine, version, system_os, cpu)
if existing_task:
logger.debug(f"Already downloading {engine} {version}")
if not background:
existing_task.join() # If download task exists, wait until it's done downloading
return None
elif not engine_to_download.downloader():
logger.warning("No valid downloader for this engine. Please update this software manually.")
return None
elif not cls.engines_path:
raise FileNotFoundError("Engines path must be set before requesting downloads")
thread = EngineDownloadWorker(engine, version, system_os, cpu)
cls.download_tasks.append(thread)
thread.start()
if background:
return thread
thread.join()
found_engine = cls.is_version_downloaded(engine, version, system_os, cpu, ignore_system) # Check that engine downloaded
if not found_engine:
logger.error(f"Error downloading {engine}")
return found_engine
@classmethod
def delete_engine_download(cls, engine, version, system_os=None, cpu=None):
logger.info(f"Requested deletion of engine: {engine}-{version}")
found = cls.is_version_downloaded(engine, version, system_os, cpu)
if found and found['type'] == 'managed': # don't delete system installs
# find the root directory of the engine executable
root_dir_name = '-'.join([engine, version, found['system_os'], found['cpu']])
remove_path = os.path.join(found['path'].split(root_dir_name)[0], root_dir_name)
# delete the file path
logger.info(f"Deleting engine at path: {remove_path}")
shutil.rmtree(remove_path, ignore_errors=False)
logger.info(f"Engine {engine}-{version}-{found['system_os']}-{found['cpu']} successfully deleted")
return True
elif found: # these are managed by the system / user. Don't delete these.
logger.error(f'Cannot delete requested {engine} {version}. Managed externally.')
else:
logger.error(f"Cannot find engine: {engine}-{version}")
return False
@classmethod
def is_engine_update_available(cls, engine_class, ignore_system_installs=False):
def _is_engine_update_available(self, engine_class: Type[BaseRenderEngine], ignore_system_installs=False):
logger.debug(f"Checking for updates to {engine_class.name()}")
latest_version = engine_class.downloader().find_most_recent_version()
@@ -239,24 +181,83 @@ class EngineManager:
return None
version_num = latest_version.get('version')
if cls.is_version_downloaded(engine_class.name(), version_num, ignore_system=ignore_system_installs):
if self.is_version_installed(engine_class.name(), version_num, ignore_system=ignore_system_installs):
logger.debug(f"Latest version of {engine_class.name()} ({version_num}) already downloaded")
return None
return latest_version
# --- Downloads ---
@classmethod
def create_worker(cls, engine_name, input_path, output_path, engine_version=None, args=None, parent=None, name=None):
def _downloadable_engines(self):
return [engine for engine in self.supported_engines() if hasattr(engine, "downloader") and engine.downloader()]
worker_class = cls.engine_with_name(engine_name).worker_class()
def _get_existing_download_task(self, engine_name, version, system_os=None, cpu=None):
for task in self.download_tasks:
task_parts = task.name.split('-')
task_engine, task_version, task_system_os, task_cpu = task_parts[:4]
# check to make sure we have versions installed
all_versions = cls.all_versions_for_engine(engine_name)
if engine_name == task_engine and version == task_version:
if system_os in (task_system_os, None) and cpu in (task_cpu, None):
return task
return None
def _download_engine(self, engine_name, version, system_os=None, cpu=None, background=False, ignore_system=False):
engine_to_download = self.engine_class_with_name(engine_name)
existing_task = self.get_existing_download_task(engine_name, version, system_os, cpu)
if existing_task:
logger.debug(f"Already downloading {engine_name} {version}")
if not background:
existing_task.join()
return None
elif not engine_to_download.downloader():
logger.warning("No valid downloader for this engine. Please update this software manually.")
return None
elif not self.engines_path:
raise FileNotFoundError("Engines path must be set before requesting downloads")
thread = EngineDownloadWorker(engine_name, version, system_os, cpu)
self.download_tasks.append(thread)
thread.start()
if background:
return thread
thread.join()
found_engine = self.is_version_installed(engine_name, version, system_os, cpu, ignore_system)
if not found_engine:
logger.error(f"Error downloading {engine_name}")
return found_engine
def _delete_engine_download(self, engine_name, version, system_os=None, cpu=None):
logger.info(f"Requested deletion of engine: {engine_name}-{version}")
found = self.is_version_installed(engine_name, version, system_os, cpu)
if found and found['type'] == 'managed':
root_dir_name = '-'.join([engine_name, version, found['system_os'], found['cpu']])
remove_path = os.path.join(found['path'].split(root_dir_name)[0], root_dir_name)
logger.info(f"Deleting engine at path: {remove_path}")
shutil.rmtree(remove_path, ignore_errors=False)
logger.info(f"Engine {engine_name}-{version}-{found['system_os']}-{found['cpu']} successfully deleted")
return True
elif found:
logger.error(f'Cannot delete requested {engine_name} {version}. Managed externally.')
else:
logger.error(f"Cannot find engine: {engine_name}-{version}")
return False
# --- Background Tasks ---
def _active_downloads(self) -> list:
return [x for x in self.download_tasks if x.is_alive()]
def _create_worker(self, engine_name: str, input_path: Path, output_path: Path, engine_version=None, args=None, parent=None, name=None):
worker_class = self.engine_class_with_name(engine_name).worker_class()
all_versions = self.all_version_data_for_engine(engine_name)
if not all_versions:
raise FileNotFoundError(f"Cannot find any installed '{engine_name}' engines")
# Find the path to the requested engine version or use default
engine_path = None
if engine_version and engine_version != 'latest':
for ver in all_versions:
@@ -264,9 +265,8 @@ class EngineManager:
engine_path = ver['path']
break
# Download the required engine if not found locally
if not engine_path:
download_result = cls.download_engine(engine_name, engine_version)
download_result = self.download_engine(engine_name, engine_version)
if not download_result:
raise FileNotFoundError(f"Cannot download requested version: {engine_name} {engine_version}")
engine_path = download_result['path']
@@ -278,32 +278,111 @@ class EngineManager:
if not engine_path:
raise FileNotFoundError(f"Cannot find requested engine version {engine_version}")
return worker_class(input_path=input_path, output_path=output_path, engine_path=engine_path, args=args,
return worker_class(input_path=str(input_path), output_path=str(output_path), engine_path=engine_path, args=args,
parent=parent, name=name)
# --- Forwarders for backward compatibility ---
@classmethod
def engine_for_project_path(cls, path):
_, extension = os.path.splitext(path)
extension = extension.lower().strip('.')
for engine in cls.supported_engines():
if extension in engine().supported_extensions():
return engine
undefined_renderer_support = [x for x in cls.supported_engines() if not x().supported_extensions()]
return undefined_renderer_support[0]
def engine_class_for_project_path(cls, path):
if cls._default_instance is not None:
return cls._default_instance._engine_class_for_project_path(path)
@classmethod
def engine_class_with_name(cls, engine_name):
if cls._default_instance is not None:
return cls._default_instance._engine_class_with_name(engine_name)
@classmethod
def get_latest_engine_instance(cls, engine_class):
if cls._default_instance is not None:
return cls._default_instance._get_latest_engine_instance(engine_class)
@classmethod
def get_installed_engine_data(cls, filter_name=None, include_corrupt=False, ignore_system=False):
if cls._default_instance is not None:
return cls._default_instance._get_installed_engine_data(filter_name, include_corrupt, ignore_system)
return []
@classmethod
def update_all_engines(cls):
if cls._default_instance is not None:
cls._default_instance._update_all_engines()
@classmethod
def all_version_data_for_engine(cls, engine_name, include_corrupt=False, ignore_system=False):
if cls._default_instance is not None:
return cls._default_instance._all_version_data_for_engine(engine_name, include_corrupt, ignore_system)
return []
@classmethod
def newest_installed_engine_data(cls, engine_name, system_os=None, cpu=None, ignore_system=None):
if cls._default_instance is not None:
return cls._default_instance._newest_installed_engine_data(engine_name, system_os, cpu, ignore_system)
return []
@classmethod
def is_version_installed(cls, engine_name, version, system_os=None, cpu=None, ignore_system=False):
if cls._default_instance is not None:
return cls._default_instance._is_version_installed(engine_name, version, system_os, cpu, ignore_system)
return False
@classmethod
def version_is_available_to_download(cls, engine_name, version, system_os=None, cpu=None):
if cls._default_instance is not None:
return cls._default_instance._version_is_available_to_download(engine_name, version, system_os, cpu)
return None
@classmethod
def find_most_recent_version(cls, engine_name, system_os=None, cpu=None, lts_only=False):
if cls._default_instance is not None:
return cls._default_instance._find_most_recent_version(engine_name, system_os, cpu, lts_only)
return {}
@classmethod
def is_engine_update_available(cls, engine_class, ignore_system_installs=False):
if cls._default_instance is not None:
return cls._default_instance._is_engine_update_available(engine_class, ignore_system_installs)
return None
@classmethod
def downloadable_engines(cls):
if cls._default_instance is not None:
return cls._default_instance._downloadable_engines()
return []
@classmethod
def get_existing_download_task(cls, engine_name, version, system_os=None, cpu=None):
if cls._default_instance is not None:
return cls._default_instance._get_existing_download_task(engine_name, version, system_os, cpu)
return None
@classmethod
def download_engine(cls, engine_name, version, system_os=None, cpu=None, background=False, ignore_system=False):
if cls._default_instance is not None:
return cls._default_instance._download_engine(engine_name, version, system_os, cpu, background, ignore_system)
return None
@classmethod
def delete_engine_download(cls, engine_name, version, system_os=None, cpu=None):
if cls._default_instance is not None:
return cls._default_instance._delete_engine_download(engine_name, version, system_os, cpu)
return False
@classmethod
def active_downloads(cls):
if cls._default_instance is not None:
return cls._default_instance._active_downloads()
return []
@classmethod
def create_worker(cls, engine_name, input_path, output_path, engine_version=None, args=None, parent=None, name=None):
if cls._default_instance is not None:
return cls._default_instance._create_worker(engine_name, input_path, output_path, engine_version, args, parent, name)
raise RuntimeError("EngineManager is not initialized")
class EngineDownloadWorker(threading.Thread):
"""A thread worker for downloading a specific version of a rendering engine.
This class handles the process of downloading a rendering engine in a separate thread,
ensuring that the download process does not block the main application.
Attributes:
engine (str): The name of the rendering engine to download.
version (str): The version of the rendering engine to download.
system_os (str, optional): The operating system for which to download the engine. Defaults to current OS type.
cpu (str, optional): Requested CPU architecture. Defaults to system CPU type.
"""
def __init__(self, engine, version, system_os=None, cpu=None):
super().__init__()
self.engine = engine
@@ -317,21 +396,23 @@ class EngineDownloadWorker(threading.Thread):
def run(self):
try:
existing_download = EngineManager.is_version_downloaded(self.engine, self.version, self.system_os, self.cpu,
existing_download = EngineManager.is_version_installed(self.engine, self.version, self.system_os, self.cpu,
ignore_system=True)
if existing_download:
logger.info(f"Requested download of {self.engine} {self.version}, but local copy already exists")
return existing_download
# Get the appropriate downloader class based on the engine type
downloader = EngineManager.engine_with_name(self.engine).downloader()
downloader.download_engine( self.version, download_location=EngineManager.engines_path,
downloader = EngineManager.engine_class_with_name(self.engine).downloader()
downloader.download_engine(self.version, download_location=EngineManager.engines_path,
system_os=self.system_os, cpu=self.cpu, timeout=300, progress_callback=self._update_progress)
except Exception as e:
logger.error(f"Error in download worker: {e}")
finally:
# remove itself from the downloader list
EngineManager.download_tasks.remove(self)
try:
if EngineManager._default_instance is not None:
EngineManager._default_instance.download_tasks.remove(self)
except ValueError:
pass
if __name__ == '__main__':
@@ -341,4 +422,4 @@ if __name__ == '__main__':
# EngineManager.delete_engine_download('blender', '3.2.1', 'macos', 'a')
EngineManager.engines_path = "/Users/brettwilliams/zordon-uploads/engines"
# print(EngineManager.is_version_downloaded("ffmpeg", "6.0"))
print(EngineManager.get_engines())
print(EngineManager.get_installed_engine_data())
+14 -5
View File
@@ -1,4 +1,5 @@
import json
import os
import re
from src.engines.core.base_engine import *
@@ -20,8 +21,7 @@ class FFMPEG(BaseRenderEngine):
return FFMPEGRenderWorker
def ui_options(self):
from src.engines.ffmpeg.ffmpeg_ui import FFMPEGUI
return FFMPEGUI.get_options(self)
return []
def supported_extensions(self):
help_text = (subprocess.check_output([self.engine_path(), '-h', 'full'], stderr=subprocess.STDOUT,
@@ -45,10 +45,19 @@ class FFMPEG(BaseRenderEngine):
return version
def get_project_info(self, project_path, timeout=10):
"""Run ffprobe and parse the output as JSON"""
try:
# Run ffprobe and parse the output as JSON
# resolve ffprobe path
engine_dir = os.path.dirname(self.engine_path())
ffprobe_path = os.path.join(engine_dir, 'ffprobe')
if self.engine_path().endswith('.exe'):
ffprobe_path += '.exe'
if not os.path.exists(ffprobe_path): # fallback to system install (if available)
ffprobe_path = 'ffprobe'
# run ffprobe
cmd = [
'ffprobe', '-v', 'quiet', '-print_format', 'json',
ffprobe_path, '-v', 'quiet', '-print_format', 'json',
'-show_streams', '-select_streams', 'v', project_path
]
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, text=True,
@@ -78,7 +87,7 @@ class FFMPEG(BaseRenderEngine):
}
except Exception as e:
print(f"An error occurred: {e}")
print(f"Failed to get FFMPEG project info: {e}")
return None
def get_encoders(self):
-5
View File
@@ -1,5 +0,0 @@
class FFMPEGUI:
@staticmethod
def get_options(system_info):
options = []
return options
+2 -2
View File
@@ -19,8 +19,8 @@ class FFMPEGRenderWorker(BaseRenderWorker):
cmd = [self.engine_path, '-y', '-stats', '-i', self.input_path]
# Resize frame
if self.args.get('x_resolution', None) and self.args.get('y_resolution', None):
cmd.extend(['-vf', f"scale={self.args['x_resolution']}:{self.args['y_resolution']}"])
if self.args.get('resolution', None):
cmd.extend(['-vf', f"scale={self.args['resolution'][0]}:{self.args['resolution'][1]}"])
# Convert raw args from string if available
raw_args = self.args.get('raw', None)
+203 -110
View File
@@ -1,206 +1,299 @@
import logging
import os
import threading
from collections import Counter
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional
from pubsub import pub
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import Session, sessionmaker
from sqlalchemy.orm.exc import DetachedInstanceError
from src.engines.core.base_worker import Base
from src.engines.core.base_worker import Base, BaseRenderWorker
from src.utilities.status_utils import RenderStatus
logger = logging.getLogger()
class JobNotFoundError(Exception):
def __init__(self, job_id, *args):
def __init__(self, job_id: str, *args: Any) -> None:
super().__init__(args)
self.job_id = job_id
def __str__(self):
def __str__(self) -> str:
return f"Cannot find job with ID: {self.job_id}"
class RenderQueue:
engine = None
session = None
job_queue = []
maximum_renderer_instances = {'blender': 1, 'aerender': 1, 'ffmpeg': 4}
last_saved_counts = {}
is_running = False
_default_instance: Optional['RenderQueue'] = None
@classmethod
def _sync_class(cls) -> None:
if cls._default_instance is not None:
pass # no class-level attributes to sync
def __init__(self) -> None:
self.engine: Optional[create_engine] = None
self.session: Optional[Session] = None
self.job_queue: List[BaseRenderWorker] = []
self.maximum_renderer_instances: Dict[str, int] = {'blender': 1, 'aerender': 1, 'ffmpeg': 4}
self.last_saved_counts: Dict[str, int] = {}
self.is_running: bool = False
self._lock = threading.Lock()
# --------------------------------------------
# Render Queue Evaluation:
# --------------------------------------------
@classmethod
def start(cls):
"""Start evaluating the render queue"""
def _start(self) -> None:
logger.debug("Starting render queue updates")
cls.is_running = True
cls.evaluate_queue()
self.is_running = True
self.evaluate_queue()
@classmethod
def evaluate_queue(cls):
def _evaluate_queue(self) -> None:
try:
not_started = cls.jobs_with_status(RenderStatus.NOT_STARTED, priority_sorted=True)
not_started = self.jobs_with_status(RenderStatus.NOT_STARTED, priority_sorted=True)
for job in not_started:
if cls.is_available_for_job(job.engine_name, job.priority):
cls.start_job(job)
if self.is_available_for_job(job.engine_name, job.priority):
self.start_job(job)
scheduled = cls.jobs_with_status(RenderStatus.SCHEDULED, priority_sorted=True)
scheduled = self.jobs_with_status(RenderStatus.SCHEDULED, priority_sorted=True)
for job in scheduled:
if job.scheduled_start <= datetime.now():
logger.debug(f"Starting scheduled job: {job}")
cls.start_job(job)
self.start_job(job)
if cls.last_saved_counts != cls.job_counts():
cls.save_state()
if self.last_saved_counts != self.job_counts():
self.save_state()
except DetachedInstanceError:
pass
@classmethod
def __local_job_status_changed(cls, job_id, old_status, new_status):
render_job = RenderQueue.job_with_id(job_id, none_ok=True)
if render_job and cls.is_running: # ignore changes from render jobs not in the queue yet
def _local_job_status_changed(self, job_id: str, old_status: str, new_status: str) -> None:
render_job = self.job_with_id(job_id, none_ok=True)
if render_job and self.is_running:
logger.debug(f"RenderQueue detected job {job_id} has changed from {old_status} -> {new_status}")
RenderQueue.evaluate_queue()
self.evaluate_queue()
@classmethod
def stop(cls):
def _stop(self) -> None:
logger.debug("Stopping render queue updates")
cls.is_running = False
self.is_running = False
# --------------------------------------------
# Fetch Jobs:
# --------------------------------------------
@classmethod
def all_jobs(cls):
return cls.job_queue
def _all_jobs(self) -> List[BaseRenderWorker]:
return self.job_queue
@classmethod
def running_jobs(cls):
return cls.jobs_with_status(RenderStatus.RUNNING)
def _running_jobs(self) -> List[BaseRenderWorker]:
return self.jobs_with_status(RenderStatus.RUNNING)
@classmethod
def pending_jobs(cls):
pending_jobs = cls.jobs_with_status(RenderStatus.NOT_STARTED)
pending_jobs.extend(cls.jobs_with_status(RenderStatus.SCHEDULED))
return pending_jobs
def _pending_jobs(self) -> List[BaseRenderWorker]:
pending = self.jobs_with_status(RenderStatus.NOT_STARTED)
pending.extend(self.jobs_with_status(RenderStatus.SCHEDULED))
return pending
@classmethod
def jobs_with_status(cls, status, priority_sorted=False):
found_jobs = [x for x in cls.all_jobs() if x.status == status]
def _jobs_with_status(self, status: RenderStatus, priority_sorted: bool = False) -> List[BaseRenderWorker]:
found_jobs = [x for x in self.all_jobs() if x.status == status]
if priority_sorted:
found_jobs = sorted(found_jobs, key=lambda a: a.priority, reverse=False)
return found_jobs
@classmethod
def job_with_id(cls, job_id, none_ok=False):
found_job = next((x for x in cls.all_jobs() if x.id == job_id), None)
def _job_with_id(self, job_id: str, none_ok: bool = False) -> Optional[BaseRenderWorker]:
found_job = next((x for x in self.all_jobs() if x.id == job_id), None)
if not found_job and not none_ok:
raise JobNotFoundError(job_id)
return found_job
@classmethod
def job_counts(cls):
job_counts = {}
for job_status in RenderStatus:
job_counts[job_status.value] = len(cls.jobs_with_status(job_status))
return job_counts
def _job_counts(self) -> Dict[str, int]:
counts = Counter(x.status for x in self.all_jobs())
return {s.value: counts.get(s, 0) for s in RenderStatus}
# --------------------------------------------
# Startup / Shutdown:
# --------------------------------------------
@classmethod
def load_state(cls, database_directory):
if not cls.engine:
cls.engine = create_engine(f"sqlite:///{os.path.join(database_directory, 'database.db')}")
Base.metadata.create_all(cls.engine)
cls.session = sessionmaker(bind=cls.engine)()
def _load_state(self, database_directory: Path) -> None:
self.engine = create_engine(f"sqlite:///{database_directory / 'database.db'}")
Base.metadata.create_all(self.engine)
self.session = sessionmaker(bind=self.engine)()
from src.engines.core.base_worker import BaseRenderWorker
cls.job_queue = cls.session.query(BaseRenderWorker).all()
pub.subscribe(cls.__local_job_status_changed, 'status_change')
self.job_queue = self.session.query(BaseRenderWorker).all()
pub.subscribe(self._local_job_status_changed, 'status_change')
@classmethod
def save_state(cls):
cls.session.commit()
def _save_state(self) -> None:
if self.session:
self.session.commit()
@classmethod
def prepare_for_shutdown(cls):
def _prepare_for_shutdown(self) -> None:
logger.debug("Closing session")
cls.stop()
running_jobs = cls.jobs_with_status(RenderStatus.RUNNING) # cancel all running jobs
[cls.cancel_job(job) for job in running_jobs]
cls.save_state()
cls.session.close()
self.stop()
running_jobs = self.jobs_with_status(RenderStatus.RUNNING)
for job in running_jobs:
self.cancel_job(job)
self.save_state()
if self.session:
self.session.close()
# --------------------------------------------
# Renderer Availability:
# --------------------------------------------
@classmethod
def renderer_instances(cls):
from collections import Counter
all_instances = [x.engine_name for x in cls.running_jobs()]
def renderer_instances(self) -> Counter:
all_instances = [x.engine_name for x in self.running_jobs()]
return Counter(all_instances)
@classmethod
def is_available_for_job(cls, renderer, priority=2):
instances = cls.renderer_instances()
higher_priority_jobs = [x for x in cls.running_jobs() if x.priority < priority]
max_allowed_instances = cls.maximum_renderer_instances.get(renderer, 1)
maxed_out_instances = renderer in instances.keys() and instances[renderer] >= max_allowed_instances
def _is_available_for_job(self, renderer: str, priority: int = 2) -> bool:
instances = self.renderer_instances()
higher_priority_jobs = [x for x in self.running_jobs() if x.priority < priority]
max_allowed_instances = self.maximum_renderer_instances.get(renderer, 1)
maxed_out_instances = renderer in instances and instances[renderer] >= max_allowed_instances
return not maxed_out_instances and not higher_priority_jobs
# --------------------------------------------
# Job Lifecycle Management:
# --------------------------------------------
@classmethod
def add_to_render_queue(cls, render_job, force_start=False):
def _add_to_render_queue(self, render_job: BaseRenderWorker, force_start: bool = False) -> None:
logger.info(f"Adding job to render queue: {render_job}")
cls.job_queue.append(render_job)
if cls.is_running and force_start and render_job.status in (RenderStatus.NOT_STARTED, RenderStatus.SCHEDULED):
cls.start_job(render_job)
cls.session.add(render_job)
cls.save_state()
if cls.is_running:
cls.evaluate_queue()
with self._lock:
self.job_queue.append(render_job)
if force_start and render_job.status in (RenderStatus.NOT_STARTED, RenderStatus.SCHEDULED):
self.start_job(render_job)
self.session.add(render_job)
self.save_state()
if self.is_running:
self.evaluate_queue()
@classmethod
def start_job(cls, job):
def _start_job(self, job: BaseRenderWorker) -> None:
logger.info(f'Starting job: {job}')
job.start()
cls.save_state()
self.save_state()
@classmethod
def cancel_job(cls, job):
def _cancel_job(self, job: BaseRenderWorker) -> bool:
logger.info(f'Cancelling job: {job}')
job.stop()
return job.status == RenderStatus.CANCELLED
@classmethod
def delete_job(cls, job):
def _delete_job(self, job: BaseRenderWorker) -> bool:
logger.info(f"Deleting job: {job}")
job.stop()
cls.job_queue.remove(job)
cls.session.delete(job)
cls.save_state()
with self._lock:
job.stop()
self.job_queue.remove(job)
self.session.delete(job)
self.save_state()
return True
# --------------------------------------------
# Miscellaneous:
# --------------------------------------------
def _clear_history(self) -> None:
for job in list(self.all_jobs()):
if job.status in (RenderStatus.CANCELLED, RenderStatus.COMPLETED, RenderStatus.ERROR):
self.delete_job(job)
self.save_state()
# --- Forwarders for backward compatibility ---
@classmethod
def start(cls):
if cls._default_instance is not None:
cls._default_instance._start()
@classmethod
def evaluate_queue(cls):
if cls._default_instance is not None:
cls._default_instance._evaluate_queue()
@classmethod
def stop(cls):
if cls._default_instance is not None:
cls._default_instance._stop()
@classmethod
def all_jobs(cls):
if cls._default_instance is not None:
return cls._default_instance.job_queue
return []
@classmethod
def running_jobs(cls):
if cls._default_instance is not None:
return cls._default_instance._running_jobs()
return []
@classmethod
def pending_jobs(cls):
if cls._default_instance is not None:
return cls._default_instance._pending_jobs()
return []
@classmethod
def jobs_with_status(cls, status, priority_sorted=False):
if cls._default_instance is not None:
return cls._default_instance._jobs_with_status(status, priority_sorted)
return []
@classmethod
def job_with_id(cls, job_id, none_ok=False):
if cls._default_instance is not None:
return cls._default_instance._job_with_id(job_id, none_ok)
if not none_ok:
raise JobNotFoundError(job_id)
return None
@classmethod
def job_counts(cls):
if cls._default_instance is not None:
return cls._default_instance._job_counts()
return {}
@classmethod
def load_state(cls, database_directory):
if cls._default_instance is not None:
cls._default_instance._load_state(database_directory)
@classmethod
def save_state(cls):
if cls._default_instance is not None:
cls._default_instance._save_state()
@classmethod
def prepare_for_shutdown(cls):
if cls._default_instance is not None:
cls._default_instance._prepare_for_shutdown()
@classmethod
def is_available_for_job(cls, renderer, priority=2):
if cls._default_instance is not None:
return cls._default_instance._is_available_for_job(renderer, priority)
return True
@classmethod
def add_to_render_queue(cls, render_job, force_start=False):
if cls._default_instance is not None:
cls._default_instance._add_to_render_queue(render_job, force_start)
@classmethod
def start_job(cls, job):
if cls._default_instance is not None:
cls._default_instance._start_job(job)
@classmethod
def cancel_job(cls, job):
if cls._default_instance is not None:
return cls._default_instance._cancel_job(job)
return False
@classmethod
def delete_job(cls, job):
if cls._default_instance is not None:
return cls._default_instance._delete_job(job)
return False
@classmethod
def clear_history(cls):
to_remove = [x for x in cls.all_jobs() if x.status in [RenderStatus.CANCELLED,
RenderStatus.COMPLETED, RenderStatus.ERROR]]
for job_to_remove in to_remove:
cls.delete_job(job_to_remove)
cls.save_state()
if cls._default_instance is not None:
cls._default_instance._clear_history()
+277 -141
View File
@@ -1,32 +1,35 @@
import copy
import os.path
import pathlib
import logging
import socket
import threading
from pathlib import Path
import psutil
from PyQt6.QtCore import QThread, pyqtSignal, Qt, pyqtSlot
logger = logging.getLogger(__name__)
from PyQt6.QtWidgets import (
QApplication, QWidget, QVBoxLayout, QHBoxLayout, QLabel, QLineEdit, QPushButton, QFileDialog, QSpinBox, QComboBox,
QGroupBox, QCheckBox, QProgressBar, QPlainTextEdit, QDoubleSpinBox, QMessageBox, QListWidget, QListWidgetItem
QGroupBox, QCheckBox, QProgressBar, QPlainTextEdit, QDoubleSpinBox, QMessageBox, QListWidget, QListWidgetItem,
QTabWidget
)
from requests import Response
from src.api.server_proxy import RenderServerProxy
from src.engines.engine_manager import EngineManager
from src.ui.engine_help_window import EngineHelpViewer
from src.utilities.misc_helper import COMMON_RESOLUTIONS, COMMON_FRAME_RATES
from src.utilities.zeroconf_server import ZeroconfServer
class NewRenderJobForm(QWidget):
def __init__(self, project_path=None):
super().__init__()
self.notes_group = None
self.frame_rate_input = None
self.resolution_options_list = None
self.resolution_x_input = None
self.engine_group = None
self.output_settings_group = None
self.resolution_y_input = None
self.fps_options_list = None
self.fps_input = None
self.engine_group = None
self.notes_group = None
self.output_settings_group = None
self.project_path = project_path
# UI
@@ -55,152 +58,203 @@ class NewRenderJobForm(QWidget):
self.priority_input = None
self.end_frame_input = None
self.start_frame_input = None
self.render_name_input = None
self.job_name_input = None
self.scene_file_input = None
self.scene_file_browse_button = None
self.job_name_input = None
self.tabs = None
# Job / Server Data
self.server_proxy = RenderServerProxy(socket.gethostname())
self.engine_info = None
self.project_info = None
self.installed_engines = []
self.preferred_engine = None
# Setup
self.setWindowTitle("New Job")
self.setup_ui()
self.update_engine_info()
self.setup_project()
# get renderer info in bg thread
# t = threading.Thread(target=self.update_renderer_info)
# t.start()
self.show()
def setup_ui(self):
# Main Layout
# Main widget layout
main_layout = QVBoxLayout(self)
# Loading File Group
# Tabs
self.tabs = QTabWidget()
# ==================== Loading Section (outside tabs) ====================
self.load_file_group = QGroupBox("Loading")
load_file_layout = QVBoxLayout(self.load_file_group)
# progress bar
progress_layout = QHBoxLayout()
self.process_label = QLabel("Processing")
self.process_progress_bar = QProgressBar()
self.process_progress_bar.setMinimum(0)
self.process_progress_bar.setMaximum(0)
self.process_label = QLabel("Processing")
self.process_progress_bar.setMaximum(0) # Indeterminate
progress_layout.addWidget(self.process_label)
progress_layout.addWidget(self.process_progress_bar)
load_file_layout.addLayout(progress_layout)
main_layout.addWidget(self.load_file_group)
# Project Group
self.project_group = QGroupBox("Project")
server_layout = QVBoxLayout(self.project_group)
# File Path
# Scene File
job_overview_group = QGroupBox("Project File")
file_group_layout = QVBoxLayout(job_overview_group)
# Job Name
job_name_layout = QHBoxLayout()
job_name_layout.addWidget(QLabel("Job name:"))
self.job_name_input = QLineEdit()
job_name_layout.addWidget(self.job_name_input)
self.engine_type = QComboBox()
job_name_layout.addWidget(self.engine_type)
file_group_layout.addLayout(job_name_layout)
# Job File
scene_file_picker_layout = QHBoxLayout()
scene_file_picker_layout.addWidget(QLabel("File:"))
self.scene_file_input = QLineEdit()
self.scene_file_input.setText(self.project_path)
self.scene_file_browse_button = QPushButton("Browse...")
self.scene_file_browse_button.clicked.connect(self.browse_scene_file)
scene_file_picker_layout.addWidget(QLabel("File:"))
scene_file_picker_layout.addWidget(self.scene_file_input)
scene_file_picker_layout.addWidget(self.scene_file_browse_button)
server_layout.addLayout(scene_file_picker_layout)
# Server List
file_group_layout.addLayout(scene_file_picker_layout)
main_layout.addWidget(job_overview_group)
main_layout.addWidget(self.load_file_group)
main_layout.addWidget(self.tabs)
# ==================== Tab 1: Job Settings ====================
self.project_group = QWidget()
project_layout = QVBoxLayout(self.project_group) # Fixed: proper parent
# Server / Hostname
server_list_layout = QHBoxLayout()
server_list_layout.setSpacing(0)
server_list_layout.addWidget(QLabel("Render Target:"))
self.server_input = QComboBox()
server_list_layout.addWidget(QLabel("Hostname:"), 1)
server_list_layout.addWidget(self.server_input, 3)
server_layout.addLayout(server_list_layout)
main_layout.addWidget(self.project_group)
self.update_server_list()
self.server_input.currentTextChanged.connect(self.server_changed)
server_list_layout.addWidget(self.server_input)
project_layout.addLayout(server_list_layout)
# Priority
priority_layout = QHBoxLayout()
priority_layout.addWidget(QLabel("Priority:"), 1)
priority_layout.addWidget(QLabel("Priority:"))
self.priority_input = QComboBox()
self.priority_input.addItems(["High", "Medium", "Low"])
self.priority_input.setCurrentIndex(1)
priority_layout.addWidget(self.priority_input, 3)
server_layout.addLayout(priority_layout)
# Splitjobs
self.enable_splitjobs = QCheckBox("Automatically split render across multiple servers")
self.enable_splitjobs.setEnabled(True)
server_layout.addWidget(self.enable_splitjobs)
self.splitjobs_same_os = QCheckBox("Only render on same OS")
self.splitjobs_same_os.setEnabled(True)
server_layout.addWidget(self.splitjobs_same_os)
priority_layout.addWidget(self.priority_input)
project_layout.addLayout(priority_layout)
# Output Settings Group
self.output_settings_group = QGroupBox("Output Settings")
# Split Jobs Options
self.enable_splitjobs = QCheckBox("Automatically split render across multiple servers")
project_layout.addWidget(self.enable_splitjobs)
self.splitjobs_same_os = QCheckBox("Only render on same OS")
project_layout.addWidget(self.splitjobs_same_os)
project_layout.addStretch() # Push everything up
# ==================== Tab 2: Output Settings ====================
self.output_settings_group = QWidget()
output_settings_layout = QVBoxLayout(self.output_settings_group)
# output path
render_name_layout = QHBoxLayout()
render_name_layout.addWidget(QLabel("Render name:"))
self.render_name_input = QLineEdit()
render_name_layout.addWidget(self.render_name_input)
output_settings_layout.addLayout(render_name_layout)
# file format
# File Format
format_group = QGroupBox("Format / Range")
output_settings_layout.addWidget(format_group)
format_group_layout = QVBoxLayout()
format_group.setLayout(format_group_layout)
file_format_layout = QHBoxLayout()
file_format_layout.addWidget(QLabel("Format:"))
self.file_format_combo = QComboBox()
self.file_format_combo.setFixedWidth(200)
file_format_layout.addWidget(self.file_format_combo)
output_settings_layout.addLayout(file_format_layout)
# frame range
frame_range_layout = QHBoxLayout(self.output_settings_group)
file_format_layout.addStretch()
format_group_layout.addLayout(file_format_layout)
# Frame Range
frame_range_layout = QHBoxLayout()
frame_range_layout.addWidget(QLabel("Frames:"))
self.start_frame_input = QSpinBox()
self.start_frame_input.setRange(1, 99999)
self.start_frame_input.setFixedWidth(80)
self.end_frame_input = QSpinBox()
self.end_frame_input.setRange(1, 99999)
frame_range_layout.addWidget(QLabel("Frames:"))
self.end_frame_input.setFixedWidth(80)
frame_range_layout.addWidget(self.start_frame_input)
frame_range_layout.addWidget(QLabel("to"))
frame_range_layout.addWidget(self.end_frame_input)
output_settings_layout.addLayout(frame_range_layout)
# resolution
resolution_layout = QHBoxLayout(self.output_settings_group)
frame_range_layout.addStretch()
format_group_layout.addLayout(frame_range_layout)
# --- Resolution & FPS Group ---
resolution_group = QGroupBox("Resolution / Frame Rate")
output_settings_layout.addWidget(resolution_group)
resolution_group_layout = QVBoxLayout()
resolution_group.setLayout(resolution_group_layout)
# Resolution
resolution_layout = QHBoxLayout()
self.resolution_options_list = QComboBox()
self.resolution_options_list.setFixedWidth(200)
self.resolution_options_list.addItem("Original Size")
for res in COMMON_RESOLUTIONS:
self.resolution_options_list.addItem(res)
self.resolution_options_list.currentIndexChanged.connect(self._resolution_preset_changed)
resolution_layout.addWidget(self.resolution_options_list)
resolution_group_layout.addLayout(resolution_layout)
self.resolution_x_input = QSpinBox()
self.resolution_x_input.setRange(1, 9999) # Assuming max resolution width 9999
self.resolution_x_input.setRange(1, 9999)
self.resolution_x_input.setValue(1920)
self.resolution_y_input = QSpinBox()
self.resolution_y_input.setRange(1, 9999) # Assuming max resolution height 9999
self.resolution_y_input.setValue(1080)
self.frame_rate_input = QDoubleSpinBox()
self.frame_rate_input.setRange(1, 9999) # Assuming max resolution width 9999
self.frame_rate_input.setDecimals(3)
self.frame_rate_input.setValue(23.976)
resolution_layout.addWidget(QLabel("Resolution:"))
self.resolution_x_input.setFixedWidth(80)
resolution_layout.addWidget(self.resolution_x_input)
self.resolution_y_input = QSpinBox()
self.resolution_y_input.setRange(1, 9999)
self.resolution_y_input.setValue(1080)
self.resolution_y_input.setFixedWidth(80)
resolution_layout.addWidget(QLabel("x"))
resolution_layout.addWidget(self.resolution_y_input)
resolution_layout.addWidget(QLabel("@"))
resolution_layout.addWidget(self.frame_rate_input)
resolution_layout.addWidget(QLabel("fps"))
output_settings_layout.addLayout(resolution_layout)
# add group to layout
main_layout.addWidget(self.output_settings_group)
resolution_layout.addStretch()
# Engine Group
self.engine_group = QGroupBox("Engine Settings")
fps_layout = QHBoxLayout()
self.fps_options_list = QComboBox()
self.fps_options_list.setFixedWidth(200)
self.fps_options_list.addItem("Original FPS")
for fps_option in COMMON_FRAME_RATES:
self.fps_options_list.addItem(fps_option)
self.fps_options_list.currentIndexChanged.connect(self._fps_preset_changed)
fps_layout.addWidget(self.fps_options_list)
self.fps_input = QDoubleSpinBox()
self.fps_input.setDecimals(3)
self.fps_input.setRange(1.0, 999.0)
self.fps_input.setValue(23.976)
self.fps_input.setFixedWidth(80)
fps_layout.addWidget(self.fps_input)
fps_layout.addWidget(QLabel("fps"))
fps_layout.addStretch()
resolution_group_layout.addLayout(fps_layout)
output_settings_layout.addStretch()
# ==================== Tab 3: Engine Settings ====================
self.engine_group = QWidget()
engine_group_layout = QVBoxLayout(self.engine_group)
engine_layout = QHBoxLayout()
engine_layout.addWidget(QLabel("Engine:"))
self.engine_type = QComboBox()
self.engine_type.currentIndexChanged.connect(self.engine_changed)
engine_layout.addWidget(self.engine_type)
# Version
engine_layout.addWidget(QLabel("Version:"))
engine_layout.addWidget(QLabel("Engine Version:"))
self.engine_version_combo = QComboBox()
self.engine_version_combo.addItem('latest')
engine_layout.addWidget(self.engine_version_combo)
engine_group_layout.addLayout(engine_layout)
# dynamic options
# Dynamic engine options
self.engine_options_layout = QVBoxLayout()
engine_group_layout.addLayout(self.engine_options_layout)
# Raw Args
raw_args_layout = QHBoxLayout(self.engine_group)
raw_args_layout = QHBoxLayout()
raw_args_layout.addWidget(QLabel("Raw Args:"))
self.raw_args = QLineEdit()
raw_args_layout.addWidget(self.raw_args)
@@ -208,24 +262,34 @@ class NewRenderJobForm(QWidget):
args_help_button.clicked.connect(self.args_help_button_clicked)
raw_args_layout.addWidget(args_help_button)
engine_group_layout.addLayout(raw_args_layout)
main_layout.addWidget(self.engine_group)
engine_group_layout.addStretch()
# Cameras Group
self.cameras_group = QGroupBox("Cameras")
# ==================== Tab 4: Cameras ====================
self.cameras_group = QWidget()
cameras_layout = QVBoxLayout(self.cameras_group)
self.cameras_list = QListWidget()
self.cameras_group.setHidden(True)
self.cameras_list.itemChanged.connect(self.update_job_count)
cameras_layout.addWidget(self.cameras_list)
main_layout.addWidget(self.cameras_group)
# Notes Group
self.notes_group = QGroupBox("Additional Notes")
# ==================== Tab 5: Misc / Notes ====================
self.notes_group = QWidget()
notes_layout = QVBoxLayout(self.notes_group)
self.notes_input = QPlainTextEdit()
notes_layout.addWidget(self.notes_input)
main_layout.addWidget(self.notes_group)
# Submit Button
# == Create Tabs
self.tabs.addTab(self.project_group, "Job Settings")
self.tabs.addTab(self.output_settings_group, "Output Settings")
self.tabs.addTab(self.engine_group, "Engine Settings")
self.tabs.addTab(self.cameras_group, "Cameras")
self.tabs.addTab(self.notes_group, "Notes")
self.update_server_list()
index = self.tabs.indexOf(self.cameras_group)
if index != -1:
self.tabs.setTabEnabled(index, False)
# ==================== Submit Section (outside tabs) ====================
self.submit_button = QPushButton("Submit Job")
self.submit_button.clicked.connect(self.submit_job)
main_layout.addWidget(self.submit_button)
@@ -240,17 +304,40 @@ class NewRenderJobForm(QWidget):
self.submit_progress_label.setHidden(True)
main_layout.addWidget(self.submit_progress_label)
# Initial engine state
self.toggle_engine_enablement(False)
self.tabs.setCurrentIndex(0)
def update_engine_info(self):
# get the engine info and add them all to the ui
self.engine_info = self.server_proxy.get_engine_info(response_type='full')
self.engine_type.addItems(self.engine_info.keys())
# select the best engine for the file type
engine = EngineManager.engine_for_project_path(self.project_path)
self.engine_type.setCurrentText(engine.name().lower())
# refresh ui
self.engine_changed()
def update_job_count(self, changed_item=None):
checked = 1
if self.cameras_group.isEnabled():
checked = 0
total = self.cameras_list.count()
for i in range(total):
item = self.cameras_list.item(i)
if item.checkState() == Qt.CheckState.Checked:
checked += 1
message = f"Submit {checked} Jobs" if checked > 1 else "Submit Job"
self.submit_button.setText(message)
self.submit_button.setEnabled(bool(checked))
def _resolution_preset_changed(self, index):
selected_res = COMMON_RESOLUTIONS.get(self.resolution_options_list.currentText())
if selected_res:
self.resolution_x_input.setValue(selected_res[0])
self.resolution_y_input.setValue(selected_res[1])
elif index == 0:
self.resolution_x_input.setValue(self.project_info.get('resolution_x'))
self.resolution_y_input.setValue(self.project_info.get('resolution_y'))
def _fps_preset_changed(self, index):
selected_fps = COMMON_FRAME_RATES.get(self.fps_options_list.currentText())
if selected_fps:
self.fps_input.setValue(selected_fps)
elif index == 0:
self.fps_input.setValue(self.project_info.get('fps'))
def engine_changed(self):
# load the version numbers
@@ -259,14 +346,35 @@ class NewRenderJobForm(QWidget):
self.engine_version_combo.addItem('latest')
self.file_format_combo.clear()
if current_engine:
engine_vers = [version_info['version'] for version_info in self.engine_info[current_engine]['versions']]
engine_info = self.server_proxy.get_engine(current_engine, 'full', timeout=10)
self.current_engine_options = engine_info.get('options', [])
if not engine_info:
raise FileNotFoundError(f"Cannot get information about engine '{current_engine}'")
engine_vers = [v['version'] for v in engine_info['versions']]
self.engine_version_combo.addItems(engine_vers)
self.file_format_combo.addItems(self.engine_info[current_engine]['supported_export_formats'])
self.file_format_combo.addItems(engine_info.get('supported_export_formats'))
def server_changed(self, hostname):
if not hostname:
return
self.server_proxy = RenderServerProxy(hostname)
if self.engine_type and self.engine_type.count():
try:
self.engine_changed()
except Exception as e:
logger.error(f"Error updating engine data for server '{hostname}': {e}")
def update_server_list(self):
current_hostname = self.server_input.currentText()
clients = ZeroconfServer.found_hostnames()
self.server_input.clear()
self.server_input.addItems(clients)
if current_hostname and current_hostname in clients:
self.server_input.setCurrentText(current_hostname)
elif clients:
self.server_changed(clients[0])
def browse_scene_file(self):
file_name, _ = QFileDialog.getOpenFileName(self, "Select Scene File")
@@ -280,35 +388,47 @@ class NewRenderJobForm(QWidget):
self.process_label.setHidden(False)
self.toggle_engine_enablement(False)
output_name, _ = os.path.splitext(os.path.basename(self.scene_file_input.text()))
output_name = output_name.replace(' ', '_')
self.render_name_input.setText(output_name)
output_name = Path(self.scene_file_input.text()).stem.replace(' ', '_')
self.job_name_input.setText(output_name)
file_name = self.scene_file_input.text()
# setup bg worker
self.worker_thread = GetProjectInfoWorker(window=self, project_path=file_name)
self.worker_thread.message_signal.connect(self.post_get_project_info_update)
self.worker_thread.error_signal.connect(self.show_error_message)
self.worker_thread.start()
def browse_output_path(self):
directory = QFileDialog.getExistingDirectory(self, "Select Output Directory")
if directory:
self.render_name_input.setText(directory)
self.job_name_input.setText(directory)
def args_help_button_clicked(self):
url = (f'http://{self.server_proxy.hostname}:{self.server_proxy.port}/api/engine/'
url = (f'http://{self.server_proxy.hostname}:{self.server_proxy.port}/api/engines/'
f'{self.engine_type.currentText()}/help')
self.engine_help_viewer = EngineHelpViewer(url)
self.engine_help_viewer.show()
def show_error_message(self, message):
msg = QMessageBox(self)
msg.setIcon(QMessageBox.Icon.Critical)
msg.setWindowTitle("Error")
msg.setText(message)
msg.exec()
# -------- Update --------
def post_get_project_info_update(self):
"""Called by the GetProjectInfoWorker - Do not call directly."""
try:
self.engine_type.addItems(self.installed_engines)
self.engine_type.setCurrentText(self.preferred_engine)
self.engine_changed()
# Set the best engine we can find
input_path = self.scene_file_input.text()
engine = EngineManager.engine_for_project_path(input_path)
engine = EngineManager.engine_class_for_project_path(input_path)
engine_index = self.engine_type.findText(engine.name().lower())
if engine_index >= 0:
@@ -326,12 +446,13 @@ class NewRenderJobForm(QWidget):
self.end_frame_input.setValue(self.project_info.get('frame_end'))
self.resolution_x_input.setValue(self.project_info.get('resolution_x'))
self.resolution_y_input.setValue(self.project_info.get('resolution_y'))
self.frame_rate_input.setValue(self.project_info.get('fps'))
self.fps_input.setValue(self.project_info.get('fps'))
# Cameras
self.cameras_list.clear()
index = self.tabs.indexOf(self.cameras_group)
if self.project_info.get('cameras'):
self.cameras_group.setHidden(False)
self.tabs.setTabEnabled(index, True)
found_active = False
for camera in self.project_info['cameras']:
# create the list items and make them checkable
@@ -344,13 +465,12 @@ class NewRenderJobForm(QWidget):
if not found_active:
self.cameras_list.item(0).setCheckState(Qt.CheckState.Checked)
else:
self.cameras_group.setHidden(True)
self.tabs.setTabEnabled(index, False)
self.update_job_count()
# Dynamic Engine Options
clear_layout(self.engine_options_layout) # clear old options
# dynamically populate option list
system_info = self.engine_info.get(engine.name(), {}).get('system_info', {})
self.current_engine_options = engine.ui_options(system_info=system_info)
for option in self.current_engine_options:
h_layout = QHBoxLayout()
label = QLabel(option['name'].replace('_', ' ').capitalize() + ':')
@@ -364,17 +484,18 @@ class NewRenderJobForm(QWidget):
text_box = QLineEdit()
h_layout.addWidget(text_box)
self.engine_options_layout.addLayout(h_layout)
except AttributeError:
pass
except AttributeError as e:
logger.error(f"AttributeError in post_get_project_info_update: {e}")
def toggle_engine_enablement(self, enabled=False):
"""Toggle on/off all the render settings"""
self.project_group.setHidden(not enabled)
self.output_settings_group.setHidden(not enabled)
self.engine_group.setHidden(not enabled)
self.notes_group.setHidden(not enabled)
if not enabled:
self.cameras_group.setHidden(True)
indexes = [self.tabs.indexOf(self.project_group),
self.tabs.indexOf(self.output_settings_group),
self.tabs.indexOf(self.engine_group),
self.tabs.indexOf(self.cameras_group),
self.tabs.indexOf(self.notes_group)]
for idx in indexes:
self.tabs.setTabEnabled(idx, enabled)
self.submit_button.setEnabled(enabled)
def after_job_submission(self, error_string):
@@ -449,19 +570,22 @@ class SubmitWorker(QThread):
try:
hostname = self.window.server_input.currentText()
resolution = (self.window.resolution_x_input.text(), self.window.resolution_y_input.text())
job_json = {'owner': psutil.Process().username() + '@' + socket.gethostname(),
'engine_name': self.window.engine_type.currentText().lower(),
'engine_version': self.window.engine_version_combo.currentText(),
'args': {'raw': self.window.raw_args.text(),
'export_format': self.window.file_format_combo.currentText()},
'output_path': self.window.render_name_input.text(),
'export_format': self.window.file_format_combo.currentText(),
'resolution': resolution,
'fps': self.window.fps_input.text(),},
'output_path': self.window.job_name_input.text(),
'start_frame': self.window.start_frame_input.value(),
'end_frame': self.window.end_frame_input.value(),
'priority': self.window.priority_input.currentIndex() + 1,
'notes': self.window.notes_input.toPlainText(),
'enable_split_jobs': self.window.enable_splitjobs.isChecked(),
'split_jobs_same_os': self.window.splitjobs_same_os.isChecked(),
'name': self.window.render_name_input.text()}
'name': self.window.job_name_input.text()}
# get the dynamic args
for i in range(self.window.engine_options_layout.count()):
@@ -477,7 +601,7 @@ class SubmitWorker(QThread):
# determine if any cameras are checked
selected_cameras = []
if self.window.cameras_list.count() and not self.window.cameras_group.isHidden():
if self.window.cameras_list.count() and self.window.cameras_group.isEnabled():
for index in range(self.window.cameras_list.count()):
item = self.window.cameras_list.item(index)
if item.checkState() == Qt.CheckState.Checked:
@@ -489,22 +613,23 @@ class SubmitWorker(QThread):
children_jobs = []
for cam in selected_cameras:
child_job_data = dict()
child_job_data['args'] = {}
child_job_data['args'] = dict(job_json['args'])
child_job_data['args']['camera'] = cam
child_job_data['name'] = job_json['name'].replace(' ', '-') + "_" + cam.replace(' ', '')
child_job_data['output_path'] = child_job_data['name']
children_jobs.append(child_job_data)
job_json['child_jobs'] = children_jobs
# presubmission tasks
engine = EngineManager.engine_with_name(self.window.engine_type.currentText().lower())
input_path = engine().perform_presubmission_tasks(input_path)
# presubmission tasks - use local installs
engine_class = EngineManager.engine_class_with_name(self.window.engine_type.currentText().lower())
latest_engine = EngineManager.get_latest_engine_instance(engine_class)
input_path = Path(latest_engine.perform_presubmission_tasks(input_path))
# submit
err_msg = ""
result = self.window.server_proxy.post_job_to_server(file_path=input_path, job_data=job_json,
callback=create_callback)
result = self.window.server_proxy.create_job(file_path=input_path, job_data=job_json,
callback=create_callback)
if not (result and result.ok):
err_msg = f"Error posting job to server: {result.message}"
err_msg = f"Error posting job to server: {result.text}"
self.message_signal.emit(err_msg)
@@ -516,6 +641,7 @@ class GetProjectInfoWorker(QThread):
"""Worker class called to retrieve information about a project file on a background thread and update the UI"""
message_signal = pyqtSignal()
error_signal = pyqtSignal(str)
def __init__(self, window, project_path):
super().__init__()
@@ -523,9 +649,19 @@ class GetProjectInfoWorker(QThread):
self.project_path = project_path
def run(self):
engine = EngineManager.engine_for_project_path(self.project_path)
self.window.project_info = engine().get_project_info(self.project_path)
self.message_signal.emit()
try:
# get the engine info and add them all to the ui
self.window.installed_engines = self.window.server_proxy.get_engine_names()
# select the best engine for the file type
self.window.preferred_engine = self.window.server_proxy.get_engine_for_filename(self.project_path)
# this should be the only time we use a local engine instead of using the proxy besides submitting
engine_class = EngineManager.engine_class_for_project_path(self.project_path)
engine = EngineManager.get_latest_engine_instance(engine_class)
self.window.project_info = engine.get_project_info(self.project_path)
self.message_signal.emit()
except Exception as e:
self.error_signal.emit(str(e))
def clear_layout(layout):
+4 -2
View File
@@ -93,7 +93,7 @@ class EngineBrowserWindow(QMainWindow):
def update_table(self):
def update_table_worker():
raw_server_data = RenderServerProxy(self.hostname).get_engine_info()
raw_server_data = RenderServerProxy(self.hostname).get_engines()
if not raw_server_data:
return
@@ -158,7 +158,9 @@ class EngineBrowserWindow(QMainWindow):
if reply is not QMessageBox.StandardButton.Yes:
return
result = RenderServerProxy(self.hostname).delete_engine(engine_info['engine'], engine_info['version'])
result = RenderServerProxy(self.hostname).delete_engine_download(
engine_info['engine'], engine_info['version'], engine_info.get('system_os'), engine_info.get('cpu'),
)
if result.ok:
self.update_table()
else:
+1 -1
View File
@@ -26,5 +26,5 @@ class EngineHelpViewer(QMainWindow):
self.fetch_help()
def fetch_help(self):
result = requests.get(self.help_path)
result = requests.get(self.help_path, timeout=10)
self.text_edit.setPlainText(result.text)
+1 -1
View File
@@ -26,5 +26,5 @@ class LogViewer(QMainWindow):
self.fetch_logs()
def fetch_logs(self):
result = requests.get(self.log_path)
result = requests.get(self.log_path, timeout=10)
self.text_edit.setPlainText(result.text)
+174 -129
View File
@@ -2,26 +2,25 @@
import ast
import datetime
import io
import json
import logging
import os
import sys
import threading
import time
from typing import List, Dict, Any, Optional
import PIL
import humanize
from PIL import Image
from PyQt6.QtCore import Qt, QByteArray, QBuffer, QIODevice, QThread
from PyQt6.QtCore import Qt, QByteArray, QBuffer, QIODevice, QThread, pyqtSignal
from PyQt6.QtGui import QPixmap, QImage, QFont, QIcon
from PyQt6.QtWidgets import QMainWindow, QWidget, QHBoxLayout, QListWidget, QTableWidget, QAbstractItemView, \
QTableWidgetItem, QLabel, QVBoxLayout, QHeaderView, QMessageBox, QGroupBox, QPushButton, QListWidgetItem, \
QFileDialog
from src.api.api_server import API_VERSION
from src.api.serverproxy_manager import ServerProxyManager
from src.render_queue import RenderQueue
from src.utilities.misc_helper import get_time_elapsed, resources_dir, is_localhost
from src.utilities.status_utils import RenderStatus
from src.utilities.zeroconf_server import ZeroconfServer
from src.ui.add_job_window import NewRenderJobForm
from src.ui.console_window import ConsoleWindow
from src.ui.engine_browser import EngineBrowserWindow
@@ -30,8 +29,10 @@ from src.ui.widgets.menubar import MenuBar
from src.ui.widgets.proportional_image_label import ProportionalImageLabel
from src.ui.widgets.statusbar import StatusBar
from src.ui.widgets.toolbar import ToolBar
from src.api.serverproxy_manager import ServerProxyManager
from src.utilities.misc_helper import launch_url, iso_datestring_to_formatted_datestring
from src.utilities.misc_helper import get_time_elapsed, resources_dir, is_localhost
from src.utilities.misc_helper import launch_url
from src.utilities.status_utils import RenderStatus
from src.utilities.zeroconf_server import ZeroconfServer
from src.version import APP_NAME
logger = logging.getLogger()
@@ -52,12 +53,12 @@ class MainWindow(QMainWindow):
super().__init__()
# Load the queue
self.job_list_view = None
self.server_info_ram = None
self.server_info_cpu = None
self.server_info_os = None
self.server_info_gpu = None
self.server_info_hostname = None
self.job_list_view: Optional[QTableWidget] = None
self.server_info_ram: Optional[str] = None
self.server_info_cpu: Optional[str] = None
self.server_info_os: Optional[str] = None
self.server_info_gpu: Optional[List[Dict[str, Any]]] = None
self.server_info_hostname: Optional[str] = None
self.engine_browser_window = None
self.server_info_group = None
self.current_hostname = None
@@ -99,8 +100,10 @@ class MainWindow(QMainWindow):
self.create_toolbars()
# start background update
self.bg_update_thread = QThread()
self.bg_update_thread.run = self.__background_update
self.found_servers = []
self.job_data = {}
self.bg_update_thread = BackgroundUpdater(window=self)
self.bg_update_thread.updated_signal.connect(self.update_ui_data)
self.bg_update_thread.start()
# Setup other windows
@@ -111,7 +114,12 @@ class MainWindow(QMainWindow):
# Pick default job
self.job_picked()
def setup_ui(self, main_layout):
def setup_ui(self, main_layout: QVBoxLayout) -> None:
"""Setup the main user interface layout.
Args:
main_layout: The main layout container for the UI widgets.
"""
# Servers
server_list_group = QGroupBox("Available Servers")
@@ -161,45 +169,41 @@ class MainWindow(QMainWindow):
self.job_list_view.verticalHeader().setVisible(False)
self.job_list_view.itemSelectionChanged.connect(self.job_picked)
self.job_list_view.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers)
self.refresh_job_headers()
# Image Layout
image_group = QGroupBox("Job Preview")
image_layout = QVBoxLayout(image_group)
image_layout.setContentsMargins(0, 0, 0, 0)
image_center_layout = QHBoxLayout()
image_center_layout.addWidget(self.image_label)
image_layout.addWidget(self.image_label)
# image_layout.addLayout(image_center_layout)
# Setup Job Headers
self.job_list_view.setHorizontalHeaderLabels(["ID", "Name", "Engine", "Priority", "Status",
"Time Elapsed", "Frames", "Date Created"])
self.job_list_view.setColumnHidden(0, True)
# Job Layout
job_list_group = QGroupBox("Render Jobs")
self.job_list_view.horizontalHeader().setSectionResizeMode(1, QHeaderView.ResizeMode.Stretch)
self.job_list_view.horizontalHeader().setSectionResizeMode(2, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(3, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(4, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(5, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(6, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(7, QHeaderView.ResizeMode.ResizeToContents)
# Job List Layout
job_list_group = QGroupBox("Job Preview")
job_list_layout = QVBoxLayout(job_list_group)
job_list_layout.setContentsMargins(0, 0, 0, 0)
image_layout.addWidget(self.job_list_view, stretch=True)
image_layout.addLayout(job_list_layout)
job_list_layout.addWidget(self.image_label)
job_list_layout.addWidget(self.job_list_view, stretch=True)
# Add them all to the window
main_layout.addLayout(info_layout)
right_layout = QVBoxLayout()
right_layout.setContentsMargins(0, 0, 0, 0)
right_layout.addWidget(image_group)
# right_layout.addWidget(job_list_group)
right_layout.addWidget(job_list_group)
main_layout.addLayout(right_layout)
def __background_update(self):
while True:
try:
self.update_servers()
self.fetch_jobs()
except RuntimeError:
pass
except Exception as e:
logger.error(f"Uncaught exception in background update: {e}")
time.sleep(0.5)
def closeEvent(self, event):
"""Handle window close event with job running confirmation.
Args:
event: The close event triggered by user.
"""
running_jobs = len(RenderQueue.running_jobs())
if running_jobs:
reply = QMessageBox.question(self, "Running Jobs",
@@ -212,7 +216,12 @@ class MainWindow(QMainWindow):
else:
event.ignore()
# -- Server Code -- #
# -- Server Code -- #
def refresh_job_list(self):
"""Refresh the job list display."""
self.job_list_view.clearContents()
self.bg_update_thread.needs_update = True
@property
def current_server_proxy(self):
@@ -220,27 +229,22 @@ class MainWindow(QMainWindow):
def server_picked(self):
"""Update the UI elements relevant to the server selection."""
try:
# Retrieve the new hostname selected by the user
new_hostname = self.server_list_view.currentItem().text()
current_item = self.server_list_view.currentItem()
if current_item is None:
return
# Check if the hostname has changed to avoid unnecessary updates
if new_hostname != self.current_hostname:
# Update the current hostname and clear the job list
self.current_hostname = new_hostname
self.job_list_view.setRowCount(0)
self.fetch_jobs(clear_table=True)
new_hostname = current_item.text()
if new_hostname == self.current_hostname:
return
# Select the first row if there are jobs listed
if self.job_list_view.rowCount():
self.job_list_view.selectRow(0)
self.current_hostname = new_hostname
self.job_list_view.setRowCount(0)
self.refresh_job_list()
# Update server information display
self.update_server_info_display(new_hostname)
if self.job_list_view.rowCount():
self.job_list_view.selectRow(0)
except AttributeError:
# Handle cases where the server list view might not be properly initialized
pass
self.update_server_info_display(new_hostname)
def update_server_info_display(self, hostname):
"""Updates the server information section of the UI."""
@@ -281,45 +285,46 @@ class MainWindow(QMainWindow):
self.server_info_ram.setText(memory_info)
self.server_info_gpu.setText(gpu_info)
def fetch_jobs(self, clear_table=False):
def update_ui_data(self):
"""Update UI data with current server and job information."""
self.update_servers()
if not self.current_server_proxy:
return
if clear_table:
self.job_list_view.clear()
self.refresh_job_headers()
server_job_data = self.job_data.get(self.current_server_proxy.hostname)
if server_job_data is None:
return
job_fetch = self.current_server_proxy.get_all_jobs(ignore_token=False)
if job_fetch:
num_jobs = len(job_fetch)
self.job_list_view.setRowCount(num_jobs)
num_jobs = len(server_job_data)
self.job_list_view.setRowCount(num_jobs)
for row, job in enumerate(job_fetch):
for row, job in enumerate(server_job_data):
display_status = job['status'] if job['status'] != RenderStatus.RUNNING.value else \
('%.0f%%' % (job['percent_complete'] * 100)) # if running, show percent, otherwise just show status
tags = (job['status'],)
start_time = datetime.datetime.fromisoformat(job['start_time']) if job['start_time'] else None
end_time = datetime.datetime.fromisoformat(job['end_time']) if job['end_time'] else None
display_status = job['status'] if job['status'] != RenderStatus.RUNNING.value else \
('%.0f%%' % (job['percent_complete'] * 100)) # if running, show percent, otherwise just show status
tags = (job['status'],)
start_time = datetime.datetime.fromisoformat(job['start_time']) if job['start_time'] else None
end_time = datetime.datetime.fromisoformat(job['end_time']) if job['end_time'] else None
time_elapsed = "" if (job['status'] != RenderStatus.RUNNING.value and not end_time) else \
get_time_elapsed(start_time, end_time)
time_elapsed = "" if (job['status'] != RenderStatus.RUNNING.value and not end_time) else \
get_time_elapsed(start_time, end_time)
name = job.get('name') or os.path.basename(job.get('input_path', ''))
engine_name = f"{job.get('renderer', '')}-{job.get('renderer_version')}"
priority = str(job.get('priority', ''))
total_frames = str(job.get('total_frames', ''))
date_created_string = iso_datestring_to_formatted_datestring(job['date_created'])
name = job.get('name') or os.path.basename(job.get('input_path', ''))
engine_name = f"{job.get('engine', '')}-{job.get('engine_version')}"
priority = str(job.get('priority', ''))
total_frames = str(job.get('total_frames', ''))
converted_time = datetime.datetime.fromisoformat(job['date_created'])
humanized_time = humanize.naturaltime(converted_time)
items = [QTableWidgetItem(job['id']), QTableWidgetItem(name), QTableWidgetItem(engine_name),
QTableWidgetItem(priority), QTableWidgetItem(display_status), QTableWidgetItem(time_elapsed),
QTableWidgetItem(total_frames), QTableWidgetItem(date_created_string)]
items = [QTableWidgetItem(job['id']), QTableWidgetItem(name), QTableWidgetItem(engine_name),
QTableWidgetItem(priority), QTableWidgetItem(display_status), QTableWidgetItem(time_elapsed),
QTableWidgetItem(total_frames), QTableWidgetItem(humanized_time)]
for col, item in enumerate(items):
self.job_list_view.setItem(row, col, item)
for col, item in enumerate(items):
self.job_list_view.setItem(row, col, item)
# -- Job Code -- #
# -- Job Code -- #
def job_picked(self):
def fetch_preview(job_id):
@@ -327,7 +332,7 @@ class MainWindow(QMainWindow):
default_image_path = "error.png"
before_fetch_hostname = self.current_server_proxy.hostname
response = self.current_server_proxy.request(f'job/{job_id}/thumbnail?size=big')
response = self.current_server_proxy.request(f'jobs/{job_id}/thumbnail?size=big')
if response.ok:
try:
with io.BytesIO(response.content) as image_data_stream:
@@ -385,6 +390,11 @@ class MainWindow(QMainWindow):
self.topbar.actions_call['Open Files'].setVisible(False)
def selected_job_ids(self):
"""Get list of selected job IDs from the job list.
Returns:
List[str]: List of selected job ID strings.
"""
try:
selected_rows = self.job_list_view.selectionModel().selectedRows()
job_ids = []
@@ -392,26 +402,20 @@ class MainWindow(QMainWindow):
id_item = self.job_list_view.item(selected_row.row(), 0)
job_ids.append(id_item.text())
return job_ids
except AttributeError:
except AttributeError as e:
logger.error(f"AttributeError in selected_job_ids: {e}")
return []
def refresh_job_headers(self):
self.job_list_view.setHorizontalHeaderLabels(["ID", "Name", "Engine", "Priority", "Status",
"Time Elapsed", "Frames", "Date Created"])
self.job_list_view.setColumnHidden(0, True)
self.job_list_view.horizontalHeader().setSectionResizeMode(1, QHeaderView.ResizeMode.Stretch)
self.job_list_view.horizontalHeader().setSectionResizeMode(2, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(3, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(4, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(5, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(6, QHeaderView.ResizeMode.ResizeToContents)
self.job_list_view.horizontalHeader().setSectionResizeMode(7, QHeaderView.ResizeMode.ResizeToContents)
# -- Image Code -- #
# -- Image Code -- #
def load_image_path(self, image_path):
# Load and set the image using QPixmap
"""Load and display an image from file path.
Args:
image_path: Path to the image file to load.
"""
# Load and set image using QPixmap
try:
pixmap = QPixmap(image_path)
if not pixmap:
@@ -444,28 +448,25 @@ class MainWindow(QMainWindow):
logger.error(f"Error loading image data: {e}")
def update_servers(self):
found_servers = list(set(ZeroconfServer.found_hostnames() + self.added_hostnames))
found_servers = [x for x in found_servers if ZeroconfServer.get_hostname_properties(x)['api_version'] == API_VERSION]
# Always make sure local hostname is first
if found_servers and not is_localhost(found_servers[0]):
for hostname in found_servers:
if self.found_servers and not is_localhost(self.found_servers[0]):
for hostname in self.found_servers:
if is_localhost(hostname):
found_servers.remove(hostname)
found_servers.insert(0, hostname)
self.found_servers.remove(hostname)
self.found_servers.insert(0, hostname)
break
old_count = self.server_list_view.count()
# Update proxys
for hostname in found_servers:
for hostname in self.found_servers:
ServerProxyManager.get_proxy_for_hostname(hostname) # setup background updates
# Add in all the missing servers
current_server_list = []
for i in range(self.server_list_view.count()):
current_server_list.append(self.server_list_view.item(i).text())
for hostname in found_servers:
for hostname in self.found_servers:
if hostname not in current_server_list:
properties = ZeroconfServer.get_hostname_properties(hostname)
image_path = os.path.join(resources_dir(), f"{properties.get('system_os', 'Monitor')}.png")
@@ -476,7 +477,7 @@ class MainWindow(QMainWindow):
servers_to_remove = []
for i in range(self.server_list_view.count()):
name = self.server_list_view.item(i).text()
if name not in found_servers:
if name not in self.found_servers:
servers_to_remove.append(name)
# remove any servers that shouldn't be shown any longer
@@ -491,6 +492,9 @@ class MainWindow(QMainWindow):
if not old_count and self.server_list_view.count():
self.server_list_view.setCurrentRow(0)
self.server_picked()
elif self.server_list_view.count() and self.server_list_view.currentItem() is None:
self.server_list_view.setCurrentRow(0)
self.server_picked()
def create_toolbars(self) -> None:
"""
@@ -523,7 +527,7 @@ class MainWindow(QMainWindow):
"New Job", f"{resources_directory}/AddProduct.png", self.new_job)
self.addToolBar(Qt.ToolBarArea.TopToolBarArea, self.topbar)
# -- Toolbar Buttons -- #
# -- Toolbar Buttons -- #
def open_console_window(self) -> None:
"""
@@ -538,28 +542,32 @@ class MainWindow(QMainWindow):
self.engine_browser_window.show()
def job_logs(self) -> None:
"""
Event handler for the "Logs" button.
"""Open log viewer for selected job.
Opens a log viewer window showing the logs for the currently selected job.
"""
selected_job_ids = self.selected_job_ids()
if selected_job_ids:
url = f'http://{self.current_server_proxy.hostname}:{self.current_server_proxy.port}/api/job/{selected_job_ids[0]}/logs'
url = (f'http://{self.current_server_proxy.hostname}:{self.current_server_proxy.port}'
f'/api/jobs/{selected_job_ids[0]}/logs')
self.log_viewer_window = LogViewer(url)
self.log_viewer_window.show()
def stop_job(self, event):
"""
Event handler for the Stop Job button
"""Stop selected render jobs with user confirmation.
Args:
event: The button click event.
"""
job_ids = self.selected_job_ids()
if not job_ids:
return
if len(job_ids) == 1:
job = next((job for job in self.current_server_proxy.get_all_jobs() if job.get('id') == job_ids[0]), None)
job = next((job for job in self.current_server_proxy.get_jobs() if job.get('id') == job_ids[0]), None)
if job:
display_name = job.get('name', os.path.basename(job.get('input_path', '')))
message = f"Are you sure you want to stop the job:\n{display_name}?"
message = f"Are you sure you want to stop job: {display_name}?"
else:
return # Job not found, handle this case as needed
else:
@@ -572,18 +580,20 @@ class MainWindow(QMainWindow):
if msg_box.exec() == QMessageBox.StandardButton.Yes:
for job_id in job_ids:
self.current_server_proxy.cancel_job(job_id, confirm=True)
self.fetch_jobs(clear_table=True)
self.refresh_job_list()
def delete_job(self, event):
"""
Event handler for the Delete Job button
"""Delete selected render jobs with user confirmation.
Args:
event: The button click event.
"""
job_ids = self.selected_job_ids()
if not job_ids:
return
if len(job_ids) == 1:
job = next((job for job in self.current_server_proxy.get_all_jobs() if job.get('id') == job_ids[0]), None)
job = next((job for job in self.current_server_proxy.get_jobs() if job.get('id') == job_ids[0]), None)
if job:
display_name = job.get('name', os.path.basename(job.get('input_path', '')))
message = f"Are you sure you want to delete the job:\n{display_name}?"
@@ -599,7 +609,7 @@ class MainWindow(QMainWindow):
if msg_box.exec() == QMessageBox.StandardButton.Yes:
for job_id in job_ids:
self.current_server_proxy.delete_job(job_id, confirm=True)
self.fetch_jobs(clear_table=True)
self.refresh_job_list()
def download_files(self, event):
@@ -608,8 +618,8 @@ class MainWindow(QMainWindow):
return
import webbrowser
download_url = (f"http://{self.current_server_proxy.hostname}:{self.current_server_proxy.port}"
f"/api/job/{job_ids[0]}/download_all")
download_url = (f'http://{self.current_server_proxy.hostname}:{self.current_server_proxy.port}'
f'/api/jobs/{job_ids[0]}/download_all')
webbrowser.open(download_url)
def open_files(self, event):
@@ -618,7 +628,7 @@ class MainWindow(QMainWindow):
return
for job_id in job_ids:
job_info = self.current_server_proxy.get_job_info(job_id)
job_info = self.current_server_proxy.get_job(job_id)
path = os.path.dirname(job_info['output_path'])
launch_url(path)
@@ -630,6 +640,41 @@ class MainWindow(QMainWindow):
self.new_job_window.show()
class BackgroundUpdater(QThread):
"""Worker class to fetch job and server information and update the UI"""
updated_signal = pyqtSignal()
error_signal = pyqtSignal(str)
def __init__(self, window):
super().__init__()
self.window = window
self.needs_update = True
def run(self):
"""Main background thread execution loop.
Continuously fetches server and job data, updating the main UI
every second or when updates are needed.
"""
try:
last_run = 0
while True:
now = time.monotonic()
if now - last_run >= 1.0 or self.needs_update:
self.window.found_servers = list(set(ZeroconfServer.found_hostnames() + self.window.added_hostnames))
self.window.found_servers = [x for x in self.window.found_servers if
ZeroconfServer.get_hostname_properties(x)['api_version'] == API_VERSION]
if self.window.current_server_proxy:
self.window.job_data[self.window.current_server_proxy.hostname] = \
self.window.current_server_proxy.get_jobs(ignore_token=False)
self.needs_update = False
self.updated_signal.emit()
time.sleep(0.05)
except Exception as e:
print(f"ERROR: {e}")
self.error_signal.emit(str(e))
if __name__ == "__main__":
# lazy load GUI frameworks
from PyQt6.QtWidgets import QApplication
+5 -7
View File
@@ -15,7 +15,7 @@ from PyQt6.QtWidgets import QApplication, QMainWindow, QListWidget, QListWidgetI
from src.api.server_proxy import RenderServerProxy
from src.engines.engine_manager import EngineManager
from src.utilities.config import Config
from src.utilities.misc_helper import launch_url, system_safe_path
from src.utilities.misc_helper import launch_url
from src.version import APP_AUTHOR, APP_NAME
settings = QSettings(APP_AUTHOR, APP_NAME)
@@ -37,7 +37,7 @@ class GetEngineInfoWorker(QThread):
self.parent = parent
def run(self):
data = RenderServerProxy(socket.gethostname()).get_engine_info()
data = RenderServerProxy(socket.gethostname()).get_engines()
self.done.emit(data)
class SettingsWindow(QMainWindow):
@@ -59,9 +59,7 @@ class SettingsWindow(QMainWindow):
self.check_for_new_engines_button = None
if not EngineManager.engines_path: # fix issue where sometimes path was not set
EngineManager.engines_path = system_safe_path(
os.path.join(os.path.join(os.path.expanduser(Config.upload_folder),
'engines')))
EngineManager.engines_path = Path(Config.upload_folder).expanduser() / "engines"
self.installed_engines_table = None
@@ -413,7 +411,7 @@ class SettingsWindow(QMainWindow):
msg_result = msg_box.exec()
messagebox_shown = True
if msg_result == QMessageBox.StandardButton.Yes:
EngineManager.download_engine(engine=engine.name(), version=result['version'], background=True,
EngineManager.download_engine(engine_name=engine.name(), version=result['version'], background=True,
ignore_system=ignore_system)
self.engine_download_progress_bar.setHidden(False)
self.engine_download_progress_bar.setValue(0)
@@ -551,4 +549,4 @@ if __name__ == "__main__":
app = QApplication([])
window = SettingsWindow()
window.show()
app.exec()
app.exec()
+43 -23
View File
@@ -1,10 +1,24 @@
import os
from pathlib import Path
from typing import Optional
import yaml
from src.utilities.misc_helper import current_system_os, copy_directory_contents
_CONFIG_ATTRS = [
'upload_folder', 'update_engines_on_launch', 'max_content_path',
'server_log_level', 'log_buffer_length', 'worker_process_timeout',
'flask_log_level', 'flask_debug_enable', 'queue_eval_seconds',
'port_number', 'enable_split_jobs', 'download_timeout_seconds',
]
class Config:
# Initialize class variables with default values
_default_instance: Optional['Config'] = None
# Class-level defaults — mutated by _sync_class() so existing
# callers (Config.upload_folder) continue to work during the
# migration to instance-based access.
upload_folder = "~/zordon-uploads/"
update_engines_on_launch = True
max_content_path = 100000000
@@ -18,33 +32,40 @@ class Config:
enable_split_jobs = True
download_timeout_seconds = 120
@classmethod
def load_config(cls, config_path):
def __init__(self) -> None:
for attr in _CONFIG_ATTRS:
setattr(self, attr, getattr(Config, attr))
def load(self, config_path: Path) -> None:
with open(config_path, 'r') as ymlfile:
cfg = yaml.safe_load(ymlfile)
cls.upload_folder = os.path.expanduser(cfg.get('upload_folder', cls.upload_folder))
cls.update_engines_on_launch = cfg.get('update_engines_on_launch', cls.update_engines_on_launch)
cls.max_content_path = cfg.get('max_content_path', cls.max_content_path)
cls.server_log_level = cfg.get('server_log_level', cls.server_log_level)
cls.log_buffer_length = cfg.get('log_buffer_length', cls.log_buffer_length)
cls.worker_process_timeout = cfg.get('worker_process_timeout', cls.worker_process_timeout)
cls.flask_log_level = cfg.get('flask_log_level', cls.flask_log_level)
cls.flask_debug_enable = cfg.get('flask_debug_enable', cls.flask_debug_enable)
cls.queue_eval_seconds = cfg.get('queue_eval_seconds', cls.queue_eval_seconds)
cls.port_number = cfg.get('port_number', cls.port_number)
cls.enable_split_jobs = cfg.get('enable_split_jobs', cls.enable_split_jobs)
cls.download_timeout_seconds = cfg.get('download_timeout_seconds', cls.download_timeout_seconds)
for attr in _CONFIG_ATTRS:
if attr in cfg:
setattr(self, attr, cfg[attr])
self.upload_folder = str(Path(self.upload_folder).expanduser())
@classmethod
def config_dir(cls):
def _sync_class(cls) -> None:
if cls._default_instance is not None:
for attr in _CONFIG_ATTRS:
setattr(cls, attr, getattr(cls._default_instance, attr))
@classmethod
def load_config(cls, config_path: Path) -> None:
instance = Config()
instance.load(config_path)
cls._default_instance = instance
cls._sync_class()
@classmethod
def config_dir(cls) -> Path:
# Set up the config path
if current_system_os() == 'macos':
local_config_path = os.path.expanduser('~/Library/Application Support/Zordon')
local_config_path = Path('~/Library/Application Support/Zordon').expanduser()
elif current_system_os() == 'windows':
local_config_path = os.path.join(os.environ['APPDATA'], 'Zordon')
local_config_path = Path(os.environ['APPDATA']) / 'Zordon'
else:
local_config_path = os.path.expanduser('~/.config/Zordon')
local_config_path = Path('~/.config/Zordon').expanduser()
return local_config_path
@classmethod
@@ -61,10 +82,9 @@ class Config:
# Determine the template path
resource_environment_path = os.environ.get('RESOURCEPATH')
if resource_environment_path:
template_path = os.path.join(resource_environment_path, 'config')
template_path = Path(resource_environment_path) / 'config'
else:
template_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), 'config')
template_path = Path(__file__).resolve().parents[2] / 'config'
# Copy contents from the template to the local configuration directory
copy_directory_contents(template_path, local_config_dir)
+89 -80
View File
@@ -9,11 +9,12 @@ import string
import subprocess
import sys
from datetime import datetime
from typing import Optional, List, Dict, Any
logger = logging.getLogger()
def launch_url(url):
def launch_url(url: str) -> None:
logger = logging.getLogger(__name__)
if shutil.which('xdg-open'):
@@ -37,7 +38,7 @@ def launch_url(url):
logger.error(f"Failed to launch URL: {url}. Error: {e}")
def file_exists_in_mounts(filepath):
def file_exists_in_mounts(filepath: str) -> Optional[str]:
"""
Check if a file exists in any mounted directory.
It searches for the file in common mount points like '/Volumes', '/mnt', and '/media'.
@@ -78,7 +79,7 @@ def file_exists_in_mounts(filepath):
return possible_mount_path
def get_time_elapsed(start_time=None, end_time=None):
def get_time_elapsed(start_time: Optional[datetime] = None, end_time: Optional[datetime] = None) -> str:
def strfdelta(tdelta, fmt='%H:%M:%S'):
days = tdelta.days
@@ -105,7 +106,7 @@ def get_time_elapsed(start_time=None, end_time=None):
return elapsed_time_string
def get_file_size_human(file_path):
def get_file_size_human(file_path: str) -> str:
size_in_bytes = os.path.getsize(file_path)
# Convert size to a human-readable format
@@ -121,26 +122,19 @@ def get_file_size_human(file_path):
return f"{size_in_bytes / 1024 ** 4:.2f} TB"
# Convert path to the appropriate format for the current platform
def system_safe_path(path):
if platform.system().lower() == "windows":
return os.path.normpath(path)
return path.replace("\\", "/")
def current_system_os():
def current_system_os() -> str:
return platform.system().lower().replace('darwin', 'macos')
def current_system_os_version():
return platform.mac_ver()[0] if current_system_os() == 'macos' else platform.release().lower()
def current_system_os_version() -> str:
return platform.release()
def current_system_cpu():
# convert all x86 64 to "x64"
return platform.machine().lower().replace('amd64', 'x64').replace('x86_64', 'x64')
def current_system_cpu() -> str:
return platform.machine().lower().replace('amd64', 'x64')
def current_system_cpu_brand():
def current_system_cpu_brand() -> str:
"""Fast cross-platform CPU brand string"""
if sys.platform.startswith('darwin'): # macOS
try:
@@ -175,28 +169,21 @@ def current_system_cpu_brand():
# Ultimate fallback
return platform.processor() or 'Unknown CPU'
def resources_dir():
resource_environment_path = os.environ.get('RESOURCEPATH', None)
if resource_environment_path: # running inside resource bundle
return os.path.join(resource_environment_path, 'resources')
else:
return os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), 'resources')
def resources_dir() -> str:
return os.path.join(os.path.dirname(__file__), '..', '..', 'resources')
def copy_directory_contents(src_dir, dst_dir):
"""
Copy the contents of the source directory (src_dir) to the destination directory (dst_dir).
"""
def copy_directory_contents(src_dir: str, dst_dir: str) -> None:
for item in os.listdir(src_dir):
src_path = os.path.join(src_dir, item)
dst_path = os.path.join(dst_dir, item)
if os.path.isdir(src_path):
shutil.copytree(src_path, dst_path, dirs_exist_ok=True)
shutil.copytree(src_path, dst_path)
else:
shutil.copy2(src_path, dst_path)
def check_for_updates(repo_name, repo_owner, app_name, current_version):
def check_for_updates(repo_name: str, repo_owner: str, app_name: str, current_version: str) -> Optional[Dict[str, Any]]:
def get_github_releases(owner, repo):
import requests
url = f"https://api.github.com/repos/{owner}/{repo}/releases"
@@ -223,82 +210,50 @@ def check_for_updates(repo_name, repo_owner, app_name, current_version):
return latest_version
return None
def is_localhost(comparison_hostname):
# this is necessary because socket.gethostname() does not always include '.local' - This is a sanitized comparison
try:
comparison_hostname = comparison_hostname.lower().replace('.local', '')
local_hostname = socket.gethostname().lower().replace('.local', '')
return comparison_hostname == local_hostname
except AttributeError:
return False
def is_localhost(comparison_hostname: str) -> bool:
return comparison_hostname in ['localhost', '127.0.0.1', socket.gethostname()]
def num_to_alphanumeric(num):
# List of possible alphanumeric characters
characters = string.ascii_letters + string.digits
# Make sure number is positive
num = abs(num)
# Convert number to alphanumeric
result = ""
while num > 0:
num, remainder = divmod(num, len(characters))
result += characters[remainder]
return result[::-1] # Reverse the result to get the correct alphanumeric string
def num_to_alphanumeric(num: int) -> str:
return string.ascii_letters[num % 26] + str(num // 26)
def iso_datestring_to_formatted_datestring(iso_date_string):
from dateutil import parser
import pytz
# Parse the ISO date string into a datetime object and convert timezones
date = parser.isoparse(iso_date_string).astimezone(pytz.UTC)
local_timezone = datetime.now().astimezone().tzinfo
date_local = date.astimezone(local_timezone)
# Format the date to the desired readable yet sortable format with 12-hour time
formatted_date = date_local.strftime('%Y-%m-%d %I:%M %p')
return formatted_date
def get_gpu_info():
def get_gpu_info() -> List[Dict[str, Any]]:
"""Cross-platform GPU information retrieval"""
def get_windows_gpu_info():
"""Get GPU info on Windows"""
try:
result = subprocess.run(
['wmic', 'path', 'win32_videocontroller', 'get', 'name,AdapterRAM', '/format:list'],
['wmic', 'path', 'win32_videocontroller', 'get', 'name,AdapterRAM', '/format:list'],
capture_output=True, text=True, timeout=5
)
# Virtual adapters to exclude
virtual_adapters = [
'virtual', 'rdp', 'hyper-v', 'microsoft basic', 'basic display',
'vga compatible', 'dummy', 'nvfbc', 'nvencode'
]
gpus = []
current_gpu = None
for line in result.stdout.strip().split('\n'):
line = line.strip()
if not line:
continue
if line.startswith('Name='):
if current_gpu and current_gpu.get('name'):
gpus.append(current_gpu)
gpu_name = line.replace('Name=', '').strip()
# Skip virtual adapters
if any(virtual in gpu_name.lower() for virtual in virtual_adapters):
current_gpu = None
else:
current_gpu = {'name': gpu_name, 'memory': 'Integrated'}
elif line.startswith('AdapterRAM=') and current_gpu:
vram_bytes_str = line.replace('AdapterRAM=', '').strip()
if vram_bytes_str and vram_bytes_str != '0':
@@ -307,10 +262,10 @@ def get_gpu_info():
current_gpu['memory'] = round(vram_gb, 2)
except:
pass
if current_gpu and current_gpu.get('name'):
gpus.append(current_gpu)
return gpus if gpus else [{'name': 'Unknown GPU', 'memory': 'Unknown'}]
except Exception as e:
logger.error(f"Failed to get Windows GPU info: {e}")
@@ -326,7 +281,7 @@ def get_gpu_info():
result = subprocess.run(['system_profiler', 'SPDisplaysDataType', '-detailLevel', 'mini', '-json'],
capture_output=True, text=True, timeout=5)
data = json.loads(result.stdout)
gpus = []
displays = data.get('SPDisplaysDataType', [])
for display in displays:
@@ -339,7 +294,7 @@ def get_gpu_info():
except Exception as e:
print(f"Failed to get macOS GPU info: {e}")
return [{'name': 'Unknown GPU', 'memory': 'Unknown'}]
def get_linux_gpu_info():
gpus = []
try:
@@ -361,7 +316,7 @@ def get_gpu_info():
vendor = "AMD"
elif "intel" in name.lower():
vendor = "Intel"
gpus.append({
"name": name,
"vendor": vendor,
@@ -377,10 +332,64 @@ def get_gpu_info():
return gpus
system = platform.system()
if system == 'Darwin': # macOS
return get_macos_gpu_info()
elif system == 'Windows':
return get_windows_gpu_info()
else: # Assume Linux or other
return get_linux_gpu_info()
COMMON_RESOLUTIONS = {
# SD
"SD_480p": (640, 480),
"NTSC_DVD": (720, 480),
"PAL_DVD": (720, 576),
# HD
"HD_720p": (1280, 720),
"HD_900p": (1600, 900),
"HD_1080p": (1920, 1080),
# Cinema / Film
"2K_DCI": (2048, 1080),
"4K_DCI": (4096, 2160),
# UHD / Consumer
"UHD_4K": (3840, 2160),
"UHD_5K": (5120, 2880),
"UHD_8K": (7680, 4320),
# Ultrawide / Aspect Variants
"UW_1080p": (2560, 1080),
"UW_1440p": (3440, 1440),
"UW_5K": (5120, 2160),
# Mobile / Social
"VERTICAL_1080x1920": (1080, 1920),
"SQUARE_1080": (1080, 1080),
# Classic / Legacy
"VGA": (640, 480),
"SVGA": (800, 600),
"XGA": (1024, 768),
"WXGA": (1280, 800),
}
COMMON_FRAME_RATES = {
"23.976 (NTSC Film)": 23.976,
"24 (Cinema)": 24.0,
"25 (PAL)": 25.0,
"29.97 (NTSC)": 29.97,
"30": 30.0,
"48 (HFR Film)": 48.0,
"50 (PAL HFR)": 50.0,
"59.94": 59.94,
"60": 60.0,
"72": 72.0,
"90 (VR)": 90.0,
"120": 120.0,
"144 (Gaming)": 144.0,
"240 (HFR)": 240.0,
}
+2 -2
View File
@@ -17,7 +17,7 @@ def download_missing_frames_from_subjob(local_job, subjob_id, subjob_hostname):
try:
local_files = [os.path.basename(x) for x in local_job.file_list()]
subjob_proxy = RenderServerProxy(subjob_hostname)
subjob_files = subjob_proxy.get_job_files_list(job_id=subjob_id) or []
subjob_files = subjob_proxy.get_job_files(job_id=subjob_id) or []
for subjob_filename in subjob_files:
if subjob_filename not in local_files:
@@ -127,7 +127,7 @@ def distribute_server_work(start_frame, end_frame, available_servers, method='ev
def fetch_benchmark(server):
try:
benchmark = requests.get(f'http://{server["hostname"]}:{ZeroconfServer.server_port}'
f'/api/cpu_benchmark').text
f'/api/cpu_benchmark', timeout=15).text
server['cpu_benchmark'] = benchmark
logger.debug(f'Benchmark for {server["hostname"]}: {benchmark}')
except requests.exceptions.RequestException as e:
+102 -62
View File
@@ -1,5 +1,6 @@
import logging
import socket
from typing import Dict, List, Optional
from pubsub import pub
from zeroconf import Zeroconf, ServiceInfo, ServiceBrowser, ServiceStateChange, NonUniqueNameException, \
@@ -9,105 +10,144 @@ logger = logging.getLogger()
class ZeroconfServer:
service_type = None
server_name = None
server_port = None
server_ip = None
zeroconf = Zeroconf()
service_info = None
client_cache = {}
properties = {}
_default_instance: Optional['ZeroconfServer'] = None
@classmethod
def configure(cls, service_type, server_name, server_port):
cls.service_type = service_type
cls.server_name = server_name
cls.server_port = server_port
try: # Stop any previously running instances
service_type: Optional[str] = None
server_name: Optional[str] = None
server_port: Optional[int] = None
properties: Dict = {}
def __init__(self) -> None:
self.service_type: Optional[str] = None
self.server_name: Optional[str] = None
self.server_port: Optional[int] = None
self.server_ip: Optional[str] = None
self.zeroconf: Zeroconf = Zeroconf()
self.service_info: Optional[ServiceInfo] = None
self.client_cache: Dict = {}
self.properties: Dict = {}
def _configure(self, service_type: str, server_name: str, server_port: int) -> None:
self.service_type = service_type
self.server_name = server_name
self.server_port = server_port
try:
socket.gethostbyname(socket.gethostname())
except socket.gaierror:
cls.stop()
self.stop()
@classmethod
def start(cls, listen_only=False):
if not cls.service_type:
def _start(self, listen_only: bool = False) -> None:
if not self.service_type:
raise RuntimeError("The 'configure' method must be run before starting the zeroconf server")
elif not listen_only:
logger.debug(f"Starting zeroconf service")
cls._register_service()
if not listen_only:
logger.debug("Starting zeroconf service")
self._register_service()
else:
logger.debug(f"Starting zeroconf service - Listen only mode")
cls._browse_services()
logger.debug("Starting zeroconf service - Listen only mode")
self._browse_services()
@classmethod
def stop(cls):
def _stop(self) -> None:
logger.debug("Stopping zeroconf service")
cls._unregister_service()
cls.zeroconf.close()
self._unregister_service()
if self.zeroconf:
self.zeroconf.close()
@classmethod
def _register_service(cls):
def _register_service(self) -> None:
try:
cls.server_ip = socket.gethostbyname(socket.gethostname())
self.server_ip = socket.gethostbyname(socket.gethostname())
info = ServiceInfo(
cls.service_type,
f"{cls.server_name}.{cls.service_type}",
addresses=[socket.inet_aton(cls.server_ip)],
port=cls.server_port,
properties=cls.properties,
self.service_type,
f"{self.server_name}.{self.service_type}",
addresses=[socket.inet_aton(self.server_ip)],
port=self.server_port,
properties=self.properties,
)
cls.service_info = info
cls.zeroconf.register_service(info)
logger.info(f"Registered zeroconf service: {cls.service_info.name}")
self.service_info = info
self.zeroconf.register_service(info)
logger.info(f"Registered zeroconf service: {self.service_info.name}")
except (NonUniqueNameException, socket.gaierror) as e:
logger.error(f"Error establishing zeroconf: {e}")
@classmethod
def _unregister_service(cls):
if cls.service_info:
cls.zeroconf.unregister_service(cls.service_info)
logger.info(f"Unregistered zeroconf service: {cls.service_info.name}")
cls.service_info = None
def _unregister_service(self) -> None:
if self.service_info:
self.zeroconf.unregister_service(self.service_info)
logger.info(f"Unregistered zeroconf service: {self.service_info.name}")
self.service_info = None
@classmethod
def _browse_services(cls):
browser = ServiceBrowser(cls.zeroconf, cls.service_type, [cls._on_service_discovered])
browser.is_alive()
def _browse_services(self) -> None:
ServiceBrowser(self.zeroconf, self.service_type, [self._on_service_discovered])
@classmethod
def _on_service_discovered(cls, zeroconf, service_type, name, state_change):
def _on_service_discovered(self, zeroconf, service_type, name, state_change) -> None:
try:
info = zeroconf.get_service_info(service_type, name)
hostname = name.split(f'.{cls.service_type}')[0]
hostname = name.split(f'.{self.service_type}')[0]
logger.debug(f"Zeroconf: {hostname} {state_change}")
if service_type == cls.service_type:
if state_change == ServiceStateChange.Added or state_change == ServiceStateChange.Updated:
cls.client_cache[hostname] = info
if service_type == self.service_type:
if state_change in (ServiceStateChange.Added, ServiceStateChange.Updated):
self.client_cache[hostname] = info
else:
cls.client_cache.pop(hostname)
self.client_cache.pop(hostname, None)
pub.sendMessage('zeroconf_state_change', hostname=hostname, state_change=state_change)
except NotRunningException:
pass
@classmethod
def found_hostnames(cls):
def _sync_class(cls) -> None:
if cls._default_instance is not None:
inst = cls._default_instance
cls.service_type = inst.service_type
cls.server_name = inst.server_name
cls.server_port = inst.server_port
cls.server_ip = inst.server_ip
cls.properties = inst.properties
def _found_hostnames(self) -> List[str]:
local_hostname = socket.gethostname()
def sort_key(hostname):
# Return 0 if it's the local hostname so it comes first, else return 1
return False if hostname == local_hostname else True
# Sort the list with the local hostname first
sorted_hostnames = sorted(cls.client_cache.keys(), key=sort_key)
sorted_hostnames = sorted(self.client_cache.keys(), key=sort_key)
return sorted_hostnames
def _get_hostname_properties(self, hostname: str) -> Dict:
server_info = self.client_cache.get(hostname)
if server_info is None:
return {}
decoded_server_info = {key.decode('utf-8'): value.decode('utf-8') for key, value in server_info.properties.items()}
return decoded_server_info
# --- Forwarders for backward compatibility ---
@classmethod
def configure(cls, service_type, server_name, server_port):
if cls._default_instance is not None:
cls._default_instance._configure(service_type, server_name, server_port)
cls._sync_class()
@classmethod
def start(cls, listen_only=False):
if cls._default_instance is not None:
cls._default_instance._start(listen_only)
@classmethod
def stop(cls):
if cls._default_instance is not None:
cls._default_instance._stop()
@classmethod
def found_hostnames(cls):
if cls._default_instance is not None:
return cls._default_instance._found_hostnames()
return []
@classmethod
def get_hostname_properties(cls, hostname):
server_info = cls.client_cache.get(hostname).properties
decoded_server_info = {key.decode('utf-8'): value.decode('utf-8') for key, value in server_info.items()}
return decoded_server_info
if cls._default_instance is not None:
return cls._default_instance._get_hostname_properties(hostname)
return {}
# Example usage:
+2 -2
View File
@@ -1,8 +1,8 @@
APP_NAME = "Zordon"
APP_VERSION = "0.0.1"
APP_VERSION = "0.8.0"
APP_AUTHOR = "Brett Williams"
APP_DESCRIPTION = "Distributed Render Farm Tools"
APP_COPYRIGHT_YEAR = "2024"
APP_COPYRIGHT_YEAR = "2026"
APP_LICENSE = "MIT License"
APP_REPO_NAME = APP_NAME
APP_REPO_OWNER = "blw1138"
View File
+134
View File
@@ -0,0 +1,134 @@
from unittest.mock import patch
from pathlib import Path
import pytest
from src.render_queue import RenderQueue
from src.engines.engine_manager import EngineManager
from src.api.preview_manager import PreviewManager
from src.utilities.config import Config
from src.distributed_job_manager import DistributedJobManager
from src.utilities.zeroconf_server import ZeroconfServer
# ---------------------------------------------------------------------------
# Global pubsub patch real pubsub can fire callbacks across tests.
# Each service module does `from pubsub import pub` at import time, so we
# must patch each module-level reference individually.
# ---------------------------------------------------------------------------
_PUBSUB_TARGETS = [
'pubsub.pub',
'src.render_queue.pub',
'src.distributed_job_manager.pub',
'src.utilities.zeroconf_server.pub',
'src.engines.core.base_worker.pub',
]
@pytest.fixture(autouse=True)
def _patch_pubsub():
mocks = {}
patchers = []
for target in _PUBSUB_TARGETS:
p = patch(target)
patchers.append(p)
mocks[target] = p.start()
yield mocks.get('pubsub.pub')
for p in reversed(patchers):
p.stop()
# ---------------------------------------------------------------------------
# Temp directory for file-system-dependent tests
# ---------------------------------------------------------------------------
@pytest.fixture
def tmp_workspace(tmp_path: Path) -> Path:
ws = tmp_path / 'zordon_test'
ws.mkdir()
return ws
# ---------------------------------------------------------------------------
# Config fixture
# ---------------------------------------------------------------------------
@pytest.fixture
def config_instance(tmp_workspace: Path) -> Config:
orig = Config._default_instance
cfg = Config()
cfg.upload_folder = str(tmp_workspace / 'uploads')
Config._default_instance = cfg
Config._sync_class()
yield cfg
Config._default_instance = orig
Config._sync_class()
# ---------------------------------------------------------------------------
# EngineManager fixture
# ---------------------------------------------------------------------------
@pytest.fixture
def engine_manager_instance(tmp_workspace: Path) -> EngineManager:
orig = EngineManager._default_instance
em = EngineManager()
em.engines_path = str(tmp_workspace / 'engines')
EngineManager._default_instance = em
EngineManager._sync_class()
yield em
EngineManager._default_instance = orig
EngineManager._sync_class()
# ---------------------------------------------------------------------------
# RenderQueue fixture
# ---------------------------------------------------------------------------
@pytest.fixture
def render_queue_instance() -> RenderQueue:
orig = RenderQueue._default_instance
rq = RenderQueue()
RenderQueue._default_instance = rq
RenderQueue._sync_class()
yield rq
RenderQueue._default_instance = orig
RenderQueue._sync_class()
# ---------------------------------------------------------------------------
# DistributedJobManager fixture
# ---------------------------------------------------------------------------
@pytest.fixture
def distributed_job_manager_instance() -> DistributedJobManager:
orig = DistributedJobManager._default_instance
djm = DistributedJobManager()
DistributedJobManager._default_instance = djm
DistributedJobManager._sync_class()
yield djm
DistributedJobManager._default_instance = orig
DistributedJobManager._sync_class()
# ---------------------------------------------------------------------------
# PreviewManager fixture
# ---------------------------------------------------------------------------
@pytest.fixture
def preview_manager_instance(tmp_workspace: Path) -> PreviewManager:
orig = PreviewManager._default_instance
pm = PreviewManager()
pm.storage_path = str(tmp_workspace / 'previews')
PreviewManager._default_instance = pm
PreviewManager._sync_class()
yield pm
PreviewManager._default_instance = orig
PreviewManager._sync_class()
# ---------------------------------------------------------------------------
# ZeroconfServer fixture
# ---------------------------------------------------------------------------
@pytest.fixture
def zeroconf_server_instance() -> ZeroconfServer:
orig = ZeroconfServer._default_instance
zs = ZeroconfServer()
ZeroconfServer._default_instance = zs
ZeroconfServer._sync_class()
yield zs
ZeroconfServer._default_instance = orig
ZeroconfServer._sync_class()
+85
View File
@@ -0,0 +1,85 @@
import logging
import os
import time
import unittest
from pathlib import Path
from src.api.server_proxy import RenderServerProxy
logger = logging.getLogger(__name__)
SERVER_HOST = os.environ.get('ZORDON_TEST_HOST', '127.0.0.1')
SERVER_PORT = os.environ.get('ZORDON_TEST_PORT', '8080')
@unittest.skipIf(os.environ.get('ZORDON_SKIP_INTEGRATION'),
'set ZORDON_SKIP_INTEGRATION to skip integration tests')
class SubmissionTestCase(unittest.TestCase):
"""Integration tests requiring a running Zordon server.
Start the server: python server.py
Run tests: ZORDON_TEST_HOST=127.0.0.1 python -m pytest tests/job_creation_tests.py
Override host/port via ZORDON_TEST_HOST and ZORDON_TEST_PORT env vars.
"""
render_server = None
test_job_id = None
@classmethod
def setUpClass(cls):
cls.render_server = RenderServerProxy(SERVER_HOST, SERVER_PORT)
@classmethod
def tearDownClass(cls):
cls.render_server = None
def test_connection_ok(self):
self.assertTrue(self.render_server.is_online(),
msg=f'Server not reachable at {SERVER_HOST}:{SERVER_PORT}')
def test_submit_job(self):
sample_file_path = Path(__file__).parent / 'resources' / 'batman_sample.blend'
self.assertTrue(sample_file_path.exists(),
msg=f'Test file not found: {sample_file_path}')
sample_job = {
'name': 'sample_blender_job',
'renderer': 'blender',
'start_frame': 1,
'end_frame': 5,
'args': {'engine': 'CYCLES'},
'enable_split_jobs': False,
}
response = self.render_server.create_job(
file_path=sample_file_path, job_data=sample_job)
self.assertIsNotNone(response, msg='No response from server')
self.assertTrue(response.ok, msg=f'Server returned {response.status_code}')
logger.info('Submitted to server ok!')
job_data = response.json()
self.__class__.test_job_id = job_data[0]['id']
def test_wait_for_job_to_complete(self):
if not self.__class__.test_job_id:
self.skipTest('No job was submitted in test_submit_job')
file_count = 0
while True:
update_response = self.render_server.get_job(self.__class__.test_job_id)
if update_response:
print(f"Status: {update_response['status']}")
if update_response['status'] not in [
'not_started', 'running', 'waiting_for_subjobs', 'configuring'
]:
self.assertEqual(update_response['status'], 'completed',
msg=f"Job ended with status: {update_response['status']}")
break
if update_response['file_count'] > file_count:
file_count = update_response['file_count']
print(f"File count is now {file_count}")
time.sleep(1)
Binary file not shown.
+172
View File
@@ -0,0 +1,172 @@
from pathlib import Path
from unittest.mock import patch
from src.utilities.config import Config, _CONFIG_ATTRS
class TestConfigDefaults:
"""Default attribute values."""
def test_default_upload_folder(self):
assert Config.upload_folder == '~/zordon-uploads/'
def test_default_port(self):
assert Config.port_number == 8080
def test_default_server_log_level(self):
assert Config.server_log_level == 'debug'
def test_default_enable_split_jobs(self):
assert Config.enable_split_jobs is True
def test_default_worker_timeout(self):
assert Config.worker_process_timeout == 120
class TestConfigInstance:
"""Instance creation and attribute initialisation."""
def test_init_copies_class_attrs(self):
cfg = Config()
assert cfg.upload_folder == '~/zordon-uploads/'
assert cfg.port_number == 8080
def test_init_has_all_attrs(self):
cfg = Config()
for attr in _CONFIG_ATTRS:
assert hasattr(cfg, attr), f'missing attr: {attr}'
class TestConfigLoad:
"""Loading configuration from YAML."""
def test_load_sets_attributes(self, tmp_path):
config_file = tmp_path / 'config.yaml'
config_file.write_text('port_number: 9090\nserver_log_level: info\n')
cfg = Config()
cfg.load(config_file)
assert cfg.port_number == 9090
assert cfg.server_log_level == 'info'
def test_load_expands_upload_folder(self, tmp_path):
config_file = tmp_path / 'config.yaml'
config_file.write_text('upload_folder: ~/custom-uploads/\n')
cfg = Config()
cfg.load(config_file)
# expanduser strips trailing slash
assert cfg.upload_folder.endswith('/custom-uploads')
def test_load_ignores_unknown_keys(self, tmp_path):
config_file = tmp_path / 'config.yaml'
config_file.write_text('nonexistent_key: value\nport_number: 7070\n')
cfg = Config()
cfg.load(config_file)
assert cfg.port_number == 7070
def test_load_raises_on_missing_file(self):
cfg = Config()
try:
cfg.load(Path('/nonexistent/path.yaml'))
assert False, 'expected FileNotFoundError'
except FileNotFoundError:
pass
class TestConfigSyncClass:
"""_sync_class propagates instance attrs to class level."""
def test_sync_class_propagates_attrs(self):
orig = Config._default_instance
try:
cfg = Config()
cfg.port_number = 7777
Config._default_instance = cfg
Config._sync_class()
assert Config.port_number == 7777
finally:
Config._default_instance = orig
Config._sync_class()
def test_sync_class_noop_when_no_instance(self):
orig = Config._default_instance
try:
Config._default_instance = None
Config._sync_class()
finally:
Config._default_instance = orig
Config._sync_class()
class TestConfigLoadConfig:
"""Classmethod load_config — full bootstrap."""
def test_load_config_sets_default_instance(self, tmp_path):
orig = Config._default_instance
try:
config_file = tmp_path / 'config.yaml'
config_file.write_text('port_number: 9999\n')
Config._default_instance = None
Config.load_config(config_file)
assert Config._default_instance is not None
assert Config.port_number == 9999
finally:
Config._default_instance = orig
Config._sync_class()
class TestConfigDir:
"""config_dir() returns OS-specific path."""
@patch('src.utilities.config.current_system_os')
def test_config_dir_macos(self, mock_os):
mock_os.return_value = 'macos'
result = Config.config_dir()
assert 'Library/Application Support/Zordon' in str(result)
@patch('src.utilities.config.current_system_os')
def test_config_dir_windows(self, mock_os):
mock_os.return_value = 'windows'
with patch.dict('os.environ', {'APPDATA': 'C:\\Users\\Test\\AppData\\Roaming'}):
result = Config.config_dir()
assert 'Zordon' in str(result)
@patch('src.utilities.config.current_system_os')
def test_config_dir_linux(self, mock_os):
mock_os.return_value = 'linux'
result = Config.config_dir()
assert '.config/Zordon' in str(result)
class TestSetupConfigDir:
"""setup_config_dir creates dir and copies template."""
@patch('src.utilities.config.copy_directory_contents')
@patch('src.utilities.config.os.makedirs')
@patch('src.utilities.config.os.path.exists')
def test_creates_dir_when_missing(self, mock_exists, mock_makedirs, mock_copy):
mock_exists.return_value = False
Config.setup_config_dir()
mock_makedirs.assert_called_once()
@patch('src.utilities.config.copy_directory_contents')
@patch('src.utilities.config.os.makedirs')
@patch('src.utilities.config.os.path.exists')
def test_skips_when_dir_exists(self, mock_exists, mock_makedirs, mock_copy):
mock_exists.return_value = True
Config.setup_config_dir()
mock_makedirs.assert_not_called()
mock_copy.assert_not_called()
+203
View File
@@ -0,0 +1,203 @@
from unittest.mock import MagicMock, patch
import src.distributed_job_manager as djm_module
from src.distributed_job_manager import DistributedJobManager
from src.utilities.status_utils import RenderStatus
class TestSubscribeToListener:
"""PubSub subscription."""
def test_subscribes_to_status_change(self, distributed_job_manager_instance):
distributed_job_manager_instance._subscribe_to_listener()
# Check via the module-level reference (the one _subscribe_to_listener uses)
djm_module.pub.subscribe.assert_any_call(
distributed_job_manager_instance._local_job_status_changed,
'status_change',
)
djm_module.pub.subscribe.assert_any_call(
distributed_job_manager_instance._local_job_frame_complete,
'frame_complete',
)
class TestCreateRenderJob:
"""Creating a render job."""
@patch('src.distributed_job_manager.os.makedirs')
@patch('src.distributed_job_manager.EngineManager.create_worker')
def test_creates_worker_and_adds_to_queue(
self, mock_create_worker, mock_makedirs, distributed_job_manager_instance,
config_instance, tmp_path,
):
worker = MagicMock()
worker.total_frames = 10
worker.parent = None
worker.id = 'job-1'
mock_create_worker.return_value = worker
project_path = tmp_path / 'test_project.blend'
project_path.write_text('fake')
attrs = {
'engine_name': 'blender',
'args': {'engine': 'CYCLES'},
'name': 'Test Job',
'start_frame': 1,
'end_frame': 10,
'priority': 2,
'enable_split_jobs': False,
}
with patch('src.distributed_job_manager.RenderQueue.add_to_render_queue') as mock_add:
with patch('src.distributed_job_manager.PreviewManager.update_previews_for_job'):
result = DistributedJobManager.create_render_job(attrs, project_path)
assert result == worker
assert worker.status == RenderStatus.NOT_STARTED
assert mock_create_worker.call_args.kwargs['output_path'] == project_path.parent.parent / 'output' / 'test_project'
mock_add.assert_called_once_with(worker, force_start=False)
@patch('src.distributed_job_manager.os.makedirs')
@patch('src.distributed_job_manager.EngineManager.create_worker')
def test_uses_requested_output_path(
self, mock_create_worker, mock_makedirs, distributed_job_manager_instance,
config_instance, tmp_path,
):
worker = MagicMock()
worker.total_frames = 10
worker.parent = None
mock_create_worker.return_value = worker
project_path = tmp_path / 'test_project.blend'
project_path.write_text('fake')
attrs = {
'engine_name': 'blender',
'args': {},
'name': 'Camera Job',
'output_path': 'test_project_Camera-001',
'enable_split_jobs': False,
}
with patch('src.distributed_job_manager.RenderQueue.add_to_render_queue'):
with patch('src.distributed_job_manager.PreviewManager.update_previews_for_job'):
DistributedJobManager.create_render_job(attrs, project_path)
assert mock_create_worker.call_args.kwargs['output_path'] == (
project_path.parent.parent / 'output' / 'test_project_Camera-001'
)
@patch('src.distributed_job_manager.os.makedirs')
@patch('src.distributed_job_manager.EngineManager.create_worker')
def test_uses_output_subdir_when_requested(
self, mock_create_worker, mock_makedirs, distributed_job_manager_instance,
config_instance, tmp_path,
):
worker = MagicMock()
worker.total_frames = 10
worker.parent = None
mock_create_worker.return_value = worker
project_path = tmp_path / 'test_project.blend'
project_path.write_text('fake')
attrs = {
'engine_name': 'blender',
'args': {},
'name': 'Camera Job',
'output_path': 'test_project_Camera-001',
'__use_output_subdir': True,
'enable_split_jobs': False,
}
with patch('src.distributed_job_manager.RenderQueue.add_to_render_queue'):
with patch('src.distributed_job_manager.PreviewManager.update_previews_for_job'):
DistributedJobManager.create_render_job(attrs, project_path)
expected_output_dir = project_path.parent.parent / 'output' / 'test_project_Camera-001'
assert mock_create_worker.call_args.kwargs['output_path'] == expected_output_dir / 'test_project_Camera-001'
mock_makedirs.assert_called_with(expected_output_dir, exist_ok=True)
@patch('src.distributed_job_manager.os.makedirs')
@patch('src.distributed_job_manager.EngineManager.create_worker')
def test_split_jobs_enabled_calls_split_async(
self, mock_create_worker, mock_makedirs, distributed_job_manager_instance,
config_instance, tmp_path,
):
worker = MagicMock()
worker.total_frames = 10
worker.parent = None
mock_create_worker.return_value = worker
project_path = tmp_path / 'test_project.blend'
project_path.write_text('fake')
attrs = {
'engine_name': 'blender',
'args': {},
'name': 'Split Job',
'start_frame': 1,
'end_frame': 10,
'priority': 2,
'enable_split_jobs': True,
}
# The forwarder passes system_os=None by default
with patch.object(distributed_job_manager_instance, '_split_into_subjobs_async') as mock_split:
with patch('src.distributed_job_manager.RenderQueue.add_to_render_queue'):
with patch('src.distributed_job_manager.PreviewManager.update_previews_for_job'):
DistributedJobManager.create_render_job(attrs, project_path)
mock_split.assert_called_once_with(worker, attrs, project_path, None)
class TestHandleSubjobUpdate:
"""Processing subjob update notifications."""
def test_updates_child_info(self, distributed_job_manager_instance):
parent_job = MagicMock()
parent_job.children = {}
subjob_data = {
'id': 'sub-1',
'hostname': 'worker-1',
'status': 'completed',
'percent_complete': 1.0,
'file_count': 5,
}
with patch('src.distributed_job_manager.download_missing_frames_from_subjob',
return_value=True):
DistributedJobManager.handle_subjob_update_notification(parent_job, subjob_data)
assert 'sub-1@worker-1' in parent_job.children
assert parent_job.children['sub-1@worker-1']['download_status'] == 'completed'
class TestFindAvailableServers:
"""Discovering remote servers."""
@patch('src.distributed_job_manager.ZeroconfServer.found_hostnames')
@patch('src.distributed_job_manager.ZeroconfServer.get_hostname_properties')
@patch('src.distributed_job_manager.RenderServerProxy')
def test_finds_matching_server(
self, mock_proxy_class, mock_get_props, mock_found_hostnames,
):
mock_found_hostnames.return_value = ['server-1.local']
from src.api.api_server import API_VERSION
mock_get_props.return_value = {'api_version': API_VERSION, 'system_os': 'macos'}
mock_proxy = MagicMock()
mock_proxy.get_engine_availability.return_value = {
'available': True,
'hostname': 'server-1.local',
}
mock_proxy_class.return_value = mock_proxy
result = DistributedJobManager.find_available_servers('blender')
assert len(result) == 1
assert result[0]['hostname'] == 'server-1.local'
mock_proxy.get_engine_availability.assert_called_once_with('blender')
+239
View File
@@ -0,0 +1,239 @@
from unittest.mock import MagicMock, patch
import pytest
from src.engines.engine_manager import EngineManager, EngineDownloadWorker
class TestEngineManagerSyncClass:
"""_sync_class propagates instance attrs to class level."""
def test_sync_class_propagates_engines_path(self, engine_manager_instance):
assert EngineManager.engines_path == engine_manager_instance.engines_path
def test_sync_class_noop_when_no_instance(self):
orig = EngineManager._default_instance
try:
EngineManager._default_instance = None
EngineManager.engines_path = 'original'
EngineManager._sync_class()
assert EngineManager.engines_path == 'original'
finally:
EngineManager._default_instance = orig
EngineManager._sync_class()
def test_supported_engines_returns_list(self):
engines = EngineManager.supported_engines()
assert len(engines) >= 2
class TestEngineClassMapping:
"""Mapping file extensions and names to engine classes."""
def test_engine_class_with_name_finds_blender(self, engine_manager_instance):
cls = EngineManager.engine_class_with_name('blender')
assert cls is not None
assert cls.__name__ == 'Blender'
def test_engine_class_with_name_finds_ffmpeg(self, engine_manager_instance):
cls = EngineManager.engine_class_with_name('ffmpeg')
assert cls is not None
assert cls.__name__ == 'FFMPEG'
def test_engine_class_with_name_returns_none_for_unknown(self, engine_manager_instance):
cls = EngineManager.engine_class_with_name('nonexistent')
assert cls is None
def test_engine_class_with_name_case_insensitive(self, engine_manager_instance):
cls = EngineManager.engine_class_with_name('BLENDER')
assert cls is not None
assert cls.__name__ == 'Blender'
def test_engine_class_for_project_path_no_engines_path(self, engine_manager_instance):
engine_manager_instance.engines_path = None
with pytest.raises(FileNotFoundError):
EngineManager.engine_class_for_project_path('test.blend')
class TestGetInstalledEngineData:
"""Parsing directory listings for managed engines."""
def test_get_installed_engine_data_no_path(self, engine_manager_instance):
engine_manager_instance.engines_path = None
with pytest.raises(FileNotFoundError):
EngineManager.get_installed_engine_data()
def test_get_installed_engine_data_empty_dir(self, engine_manager_instance, tmp_path):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
result = EngineManager.get_installed_engine_data(ignore_system=True)
assert result == []
@patch('src.engines.engine_manager.os.listdir')
@patch('src.engines.engine_manager.os.path.isdir')
def test_parse_managed_engine_directory(
self, mock_isdir, mock_listdir, engine_manager_instance, tmp_path,
):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
mock_listdir.return_value = ['blender-3.6.0-macos-arm64']
mock_isdir.return_value = True
result = EngineManager.get_installed_engine_data(ignore_system=True)
assert len(result) == 1
assert result[0]['engine'] == 'blender'
assert result[0]['version'] == '3.6.0'
assert result[0]['system_os'] == 'macos'
assert result[0]['cpu'] == 'arm64'
assert result[0]['type'] == 'managed'
@patch('src.engines.engine_manager.os.listdir')
@patch('src.engines.engine_manager.os.path.isdir')
def test_filter_by_engine_name(
self, mock_isdir, mock_listdir, engine_manager_instance, tmp_path,
):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
mock_listdir.return_value = ['blender-3.6.0-macos-arm64', 'ffmpeg-6.0-macos-arm64']
mock_isdir.return_value = True
blender_only = EngineManager.get_installed_engine_data(
filter_name='blender', ignore_system=True)
assert len(blender_only) == 1
assert blender_only[0]['engine'] == 'blender'
class TestNewestInstalledEngineData:
"""Filtering by system and CPU."""
@patch('src.engines.engine_manager.current_system_os')
@patch('src.engines.engine_manager.current_system_cpu')
def test_newest_filters_by_system_and_cpu(
self, mock_cpu, mock_os, engine_manager_instance, tmp_path,
):
mock_os.return_value = 'macos'
mock_cpu.return_value = 'arm64'
engine_manager_instance.engines_path = str(tmp_path / 'engines')
dirs = [
'blender-3.6.0-macos-arm64',
'blender-4.0.0-linux-x86_64',
'blender-4.1.0-macos-arm64',
]
with (patch('src.engines.engine_manager.os.listdir', return_value=dirs),
patch('src.engines.engine_manager.os.path.isdir', return_value=True)):
result = EngineManager.newest_installed_engine_data(
'blender', ignore_system=True)
assert result['version'] == '4.1.0'
assert result['system_os'] == 'macos'
assert result['cpu'] == 'arm64'
def test_newest_returns_empty_on_no_match(self, engine_manager_instance, tmp_path):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
result = EngineManager.newest_installed_engine_data('nonexistent')
assert result == []
class TestIsVersionInstalled:
"""Checking whether a specific version is installed."""
@patch('src.engines.engine_manager.current_system_os')
@patch('src.engines.engine_manager.current_system_cpu')
def test_finds_matching_version(
self, mock_cpu, mock_os, engine_manager_instance, tmp_path,
):
mock_os.return_value = 'macos'
mock_cpu.return_value = 'arm64'
engine_manager_instance.engines_path = str(tmp_path / 'engines')
dirs = ['blender-3.6.0-macos-arm64', 'blender-4.1.0-macos-arm64']
with (patch('src.engines.engine_manager.os.listdir', return_value=dirs),
patch('src.engines.engine_manager.os.path.isdir', return_value=True)):
result = EngineManager.is_version_installed('blender', '3.6.0')
assert result is not False
@patch('src.engines.engine_manager.current_system_os')
@patch('src.engines.engine_manager.current_system_cpu')
def test_returns_false_on_no_match(
self, mock_cpu, mock_os, engine_manager_instance, tmp_path,
):
mock_os.return_value = 'macos'
mock_cpu.return_value = 'arm64'
engine_manager_instance.engines_path = str(tmp_path / 'engines')
result = EngineManager.is_version_installed('blender', '99.0.0')
assert result is False
class TestDeleteEngineDownload:
"""Deleting a managed engine directory."""
@patch('src.engines.engine_manager.shutil.rmtree')
@patch('src.engines.engine_manager.current_system_os', return_value='macos')
@patch('src.engines.engine_manager.current_system_cpu', return_value='arm64')
def test_delete_managed_engine(
self, mock_cpu, mock_os, mock_rmtree, engine_manager_instance, tmp_path,
):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
engines_dir = tmp_path / 'engines' / 'blender-3.6.0-macos-arm64'
engines_dir.mkdir(parents=True)
(engines_dir / 'Blender').write_text('fake binary')
result = EngineManager.delete_engine_download('blender', '3.6.0')
assert result is True
mock_rmtree.assert_called_once_with(str(engines_dir), ignore_errors=False)
def test_delete_nonexistent_engine(self, engine_manager_instance, tmp_path):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
result = EngineManager.delete_engine_download('nonexistent', '1.0.0')
assert result is False
class TestActiveDownloads:
"""Background download tracking."""
def test_active_downloads_empty_initially(self, engine_manager_instance):
assert EngineManager.active_downloads() == []
def test_download_tasks_tracked(self, engine_manager_instance):
task = MagicMock(spec=EngineDownloadWorker)
task.is_alive.return_value = True
task.name = 'blender-4.0.0-macos-arm64'
engine_manager_instance.download_tasks.append(task)
active = EngineManager.active_downloads()
assert len(active) == 1
class TestCreateWorker:
"""Creating worker instances for render jobs."""
def test_create_worker_raises_when_no_engines_installed(self, engine_manager_instance, tmp_path):
engine_manager_instance.engines_path = str(tmp_path / 'engines')
with patch.object(engine_manager_instance, '_get_installed_engine_data', return_value=[]):
with pytest.raises(FileNotFoundError, match='Cannot find any installed'):
EngineManager.create_worker(
'blender',
input_path=tmp_path / 'test.blend',
output_path=tmp_path / 'output',
)
def test_create_worker_raises_for_unknown_engine(self, engine_manager_instance):
with pytest.raises(AttributeError):
EngineManager.create_worker(
'nonexistent',
input_path='/tmp/test.blend',
output_path='/tmp/output',
)
class TestDownloadableEngines:
"""Engines with a downloader."""
def test_downloadable_engines_returns_list(self, engine_manager_instance):
engines = EngineManager.downloadable_engines()
assert isinstance(engines, list)
+169
View File
@@ -0,0 +1,169 @@
from unittest.mock import MagicMock, patch
from src.api.preview_manager import PreviewManager
def make_mock_job(job_id='test-job-1', input_path='/tmp/test.blend',
file_list=None, **kwargs):
job = MagicMock()
job.id = job_id
job.input_path = input_path
job.file_list.return_value = file_list or []
for k, v in kwargs.items():
setattr(job, k, v)
return job
class TestPreviewManagerDefaults:
"""Default state."""
def test_storage_path_none_initially(self, preview_manager_instance):
assert preview_manager_instance.storage_path is not None
def test_running_jobs_empty_initially(self, preview_manager_instance):
assert preview_manager_instance._running_jobs == {}
class TestGeneratePreviewWorker:
"""Core preview generation logic."""
def test_skips_when_no_supported_files(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job(file_list=[str(tmp_path / 'output.txt')])
with patch.object(preview_manager_instance, '_generate_job_preview_worker') as mock_gen:
preview_manager_instance._update_previews_for_job(job)
mock_gen.assert_called_once_with(job, False)
def test_uses_input_path_when_no_file_list(self, preview_manager_instance, tmp_path):
"""When file_list is empty, falls back to input_path."""
job = make_mock_job(input_path=str(tmp_path / 'output.mp4'))
with patch.object(preview_manager_instance, '_generate_job_preview_worker') as mock_gen:
preview_manager_instance._update_previews_for_job(job)
mock_gen.assert_called_once()
def test_generate_preview_checks_existing_files(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job(input_path=str(tmp_path / 'test.jpg'))
with patch('src.api.preview_manager.save_first_frame') as mock_save:
with patch('src.api.preview_manager.os.path.exists', return_value=False):
preview_manager_instance._generate_job_preview_worker(job)
# No file_list → falls back to input_path → label is "input"
expected_img = str(tmp_path / f'{job.id}-input-480.jpg')
mock_save.assert_called_once_with(
source_path=job.input_path,
dest_path=expected_img,
max_width=480,
)
class TestUpdatePreviewsForJob:
"""Dispatch of preview generation."""
def test_starts_new_thread(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job(file_list=[str(tmp_path / 'test.mp4')])
with patch('src.api.preview_manager.threading.Thread') as mock_thread:
thread_instance = MagicMock()
mock_thread.return_value = thread_instance
preview_manager_instance._update_previews_for_job(job)
mock_thread.assert_called_once()
thread_instance.start.assert_called_once()
assert preview_manager_instance._running_jobs[job.id] == thread_instance
def test_reuses_existing_thread(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job()
existing_thread = MagicMock()
existing_thread.is_alive.return_value = True
preview_manager_instance._running_jobs[job.id] = existing_thread
with patch('src.api.preview_manager.threading.Thread') as mock_thread:
preview_manager_instance._update_previews_for_job(job)
mock_thread.assert_not_called()
def test_join_when_wait_requested(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job()
with patch('src.api.preview_manager.threading.Thread') as mock_thread:
thread_instance = MagicMock()
mock_thread.return_value = thread_instance
preview_manager_instance._update_previews_for_job(
job, wait_until_completion=True, timeout=30,
)
thread_instance.join.assert_called_once_with(timeout=30)
class TestGetPreviewsForJob:
"""Reading preview files."""
def test_returns_empty_when_no_previews(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job()
result = preview_manager_instance._get_previews_for_job(job)
assert result == {}
def test_returns_preview_files(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job(job_id='abc')
# Create a fake preview file
(tmp_path / 'abc-output-480.jpg').write_text('preview')
result = preview_manager_instance._get_previews_for_job(job)
assert 'output' in result
assert len(result['output']) == 1
assert result['output'][0]['kind'] == 'image'
class TestDeletePreviewsForJob:
"""Cleaning up preview files."""
def test_deletes_existing_previews(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job(job_id='abc')
preview_file = tmp_path / 'abc-output-480.jpg'
preview_file.write_text('preview')
preview_manager_instance._delete_previews_for_job(job)
assert not preview_file.exists()
def test_no_error_when_no_previews(self, preview_manager_instance, tmp_path):
preview_manager_instance.storage_path = str(tmp_path)
job = make_mock_job()
preview_manager_instance._delete_previews_for_job(job)
class TestForwarders:
"""Classmethod forwarders delegate to instance."""
def test_update_previews_for_job_forwarder(self, preview_manager_instance):
job = make_mock_job()
with patch.object(preview_manager_instance, '_update_previews_for_job') as mock_method:
PreviewManager.update_previews_for_job(job)
mock_method.assert_called_once_with(job, False, False, None)
def test_get_previews_for_job_forwarder(self, preview_manager_instance):
job = make_mock_job()
with patch.object(preview_manager_instance, '_get_previews_for_job',
return_value={'input': []}) as mock_method:
result = PreviewManager.get_previews_for_job(job)
assert result == {'input': []}
mock_method.assert_called_once_with(job)
def test_delete_previews_for_job_forwarder(self, preview_manager_instance):
job = make_mock_job()
with patch.object(preview_manager_instance, '_delete_previews_for_job') as mock_method:
PreviewManager.delete_previews_for_job(job)
mock_method.assert_called_once_with(job)
+307
View File
@@ -0,0 +1,307 @@
from unittest.mock import MagicMock, patch
from datetime import datetime
import pytest
from src.render_queue import RenderQueue, JobNotFoundError
from src.utilities.status_utils import RenderStatus
def make_mock_job(job_id='test-1', status=RenderStatus.NOT_STARTED,
engine_name='blender', priority=2, **kwargs):
job = MagicMock()
job.id = job_id
job.status = status
job.engine_name = engine_name
job.priority = priority
job.name = kwargs.get('name', 'Test Job')
job.scheduled_start = kwargs.get('scheduled_start', None)
for k, v in kwargs.items():
setattr(job, k, v)
return job
class TestRenderQueueDefaults:
"""Default state."""
def test_init_sets_empty_queue(self, render_queue_instance):
assert render_queue_instance.job_queue == []
def test_init_sets_max_instances(self, render_queue_instance):
assert render_queue_instance.maximum_renderer_instances == {'blender': 1, 'aerender': 1, 'ffmpeg': 4}
def test_init_is_not_running(self, render_queue_instance):
assert render_queue_instance.is_running is False
class TestAllJobs:
"""Returning all jobs."""
def test_all_jobs_empty_initially(self, render_queue_instance):
assert RenderQueue.all_jobs() == []
def test_all_jobs_returns_job_list(self, render_queue_instance):
job = make_mock_job()
render_queue_instance.job_queue.append(job)
assert len(RenderQueue.all_jobs()) == 1
assert RenderQueue.all_jobs()[0] == job
class TestJobsWithStatus:
"""Filtering jobs by status."""
def test_jobs_with_status_returns_matching(self, render_queue_instance):
running = make_mock_job('job-1', RenderStatus.RUNNING)
pending = make_mock_job('job-2', RenderStatus.NOT_STARTED)
render_queue_instance.job_queue.extend([running, pending])
result = RenderQueue.jobs_with_status(RenderStatus.RUNNING)
assert len(result) == 1
assert result[0].id == 'job-1'
def test_jobs_with_status_empty_when_no_match(self, render_queue_instance):
job = make_mock_job('job-1', RenderStatus.COMPLETED)
render_queue_instance.job_queue.append(job)
result = RenderQueue.jobs_with_status(RenderStatus.RUNNING)
assert result == []
def test_jobs_with_status_sorts_by_priority(self, render_queue_instance):
high = make_mock_job('job-1', RenderStatus.NOT_STARTED, priority=1)
low = make_mock_job('job-2', RenderStatus.NOT_STARTED, priority=5)
render_queue_instance.job_queue.extend([low, high])
result = RenderQueue.jobs_with_status(RenderStatus.NOT_STARTED, priority_sorted=True)
assert result[0].id == 'job-1'
class TestRunningAndPending:
"""Convenience methods for running/pending."""
def test_running_jobs(self, render_queue_instance):
running = make_mock_job('job-1', RenderStatus.RUNNING)
pending = make_mock_job('job-2', RenderStatus.NOT_STARTED)
render_queue_instance.job_queue.extend([running, pending])
result = RenderQueue.running_jobs()
assert len(result) == 1
assert result[0].id == 'job-1'
def test_pending_jobs_includes_not_started_and_scheduled(self, render_queue_instance):
ns = make_mock_job('job-1', RenderStatus.NOT_STARTED)
sched = make_mock_job('job-2', RenderStatus.SCHEDULED)
running = make_mock_job('job-3', RenderStatus.RUNNING)
render_queue_instance.job_queue.extend([ns, sched, running])
result = RenderQueue.pending_jobs()
assert len(result) == 2
class TestJobWithId:
"""Lookup by job ID."""
def test_finds_job(self, render_queue_instance):
job = make_mock_job('abc-123')
render_queue_instance.job_queue.append(job)
found = RenderQueue.job_with_id('abc-123')
assert found == job
def test_raises_when_not_found(self, render_queue_instance):
with pytest.raises(JobNotFoundError, match='abc-123'):
RenderQueue.job_with_id('abc-123')
def test_returns_none_when_not_found_with_none_ok(self, render_queue_instance):
result = RenderQueue.job_with_id('abc-123', none_ok=True)
assert result is None
class TestJobCounts:
"""Counting jobs by status."""
def test_job_counts_returns_all_statuses(self, render_queue_instance):
render_queue_instance.job_queue.extend([
make_mock_job('j1', RenderStatus.RUNNING),
make_mock_job('j2', RenderStatus.COMPLETED),
make_mock_job('j3', RenderStatus.NOT_STARTED),
])
counts = RenderQueue.job_counts()
assert counts[RenderStatus.RUNNING.value] == 1
assert counts[RenderStatus.COMPLETED.value] == 1
assert counts[RenderStatus.NOT_STARTED.value] == 1
assert counts[RenderStatus.ERROR.value] == 0
assert counts[RenderStatus.CANCELLED.value] == 0
class TestIsAvailableForJob:
"""Renderer availability checks."""
def test_available_when_no_running_jobs(self, render_queue_instance):
assert RenderQueue.is_available_for_job('blender') is True
def test_not_available_when_maxed_out(self, render_queue_instance):
render_queue_instance.job_queue.append(
make_mock_job('j1', RenderStatus.RUNNING, engine_name='blender')
)
assert RenderQueue.is_available_for_job('blender') is False
def test_available_for_different_renderer(self, render_queue_instance):
render_queue_instance.job_queue.append(
make_mock_job('j1', RenderStatus.RUNNING, engine_name='blender')
)
assert RenderQueue.is_available_for_job('ffmpeg') is True
def test_blocked_by_higher_priority(self, render_queue_instance):
render_queue_instance.job_queue.append(
make_mock_job('j1', RenderStatus.RUNNING, engine_name='blender', priority=0)
)
assert RenderQueue.is_available_for_job('blender', priority=2) is False
def test_ffmpeg_allows_multiple_instances(self, render_queue_instance):
for i in range(4):
render_queue_instance.job_queue.append(
make_mock_job(f'j{i}', RenderStatus.RUNNING, engine_name='ffmpeg')
)
assert RenderQueue.is_available_for_job('ffmpeg') is False
class TestAddToRenderQueue:
"""Adding jobs."""
def test_add_job_appends_to_queue(self, render_queue_instance):
render_queue_instance.session = MagicMock()
job = make_mock_job()
RenderQueue.add_to_render_queue(job)
assert job in render_queue_instance.job_queue
def test_add_job_saves_state(self, render_queue_instance):
render_queue_instance.session = MagicMock()
job = make_mock_job()
RenderQueue.add_to_render_queue(job)
render_queue_instance.session.add.assert_called_once_with(job)
def test_add_job_force_start_calls_start(self, render_queue_instance):
render_queue_instance.session = MagicMock()
job = make_mock_job(status=RenderStatus.NOT_STARTED)
with patch.object(render_queue_instance, '_start_job') as mock_start:
RenderQueue.add_to_render_queue(job, force_start=True)
mock_start.assert_called_once_with(job)
def test_add_job_force_start_skips_completed(self, render_queue_instance):
render_queue_instance.session = MagicMock()
job = make_mock_job(status=RenderStatus.COMPLETED)
with patch.object(render_queue_instance, '_start_job') as mock_start:
RenderQueue.add_to_render_queue(job, force_start=True)
mock_start.assert_not_called()
def test_add_job_evaluates_queue_when_running(self, render_queue_instance):
render_queue_instance.session = MagicMock()
render_queue_instance.is_running = True
job = make_mock_job()
with patch.object(render_queue_instance, '_evaluate_queue') as mock_eval:
RenderQueue.add_to_render_queue(job)
mock_eval.assert_called_once()
class TestStartCancelDelete:
"""Job lifecycle."""
def test_start_job_calls_job_start(self, render_queue_instance):
job = make_mock_job()
with patch.object(render_queue_instance, '_save_state'):
RenderQueue.start_job(job)
job.start.assert_called_once()
def test_cancel_job_calls_job_stop(self, render_queue_instance):
job = make_mock_job()
job.stop.return_value = None
job.status = RenderStatus.CANCELLED
result = RenderQueue.cancel_job(job)
assert result is True
job.stop.assert_called_once()
def test_delete_job_removes_from_queue(self, render_queue_instance):
job = make_mock_job()
render_queue_instance.job_queue.append(job)
render_queue_instance.session = MagicMock()
result = RenderQueue.delete_job(job)
assert result is True
assert job not in render_queue_instance.job_queue
render_queue_instance.session.delete.assert_called_once_with(job)
class TestStartStop:
"""Starting and stopping the queue evaluation loop."""
def test_start_sets_running_and_evaluates(self, render_queue_instance):
with patch.object(render_queue_instance, '_evaluate_queue') as mock_eval:
RenderQueue.start()
assert render_queue_instance.is_running is True
mock_eval.assert_called_once()
def test_stop_clears_running(self, render_queue_instance):
render_queue_instance.is_running = True
RenderQueue.stop()
assert render_queue_instance.is_running is False
class TestEvaluateQueue:
"""Queue evaluation dispatches jobs."""
def test_evaluate_starts_not_started_jobs(self, render_queue_instance):
job = make_mock_job()
render_queue_instance.job_queue.append(job)
with patch.object(render_queue_instance, '_start_job') as mock_start:
with patch.object(render_queue_instance, '_save_state'):
RenderQueue.evaluate_queue()
mock_start.assert_called_once_with(job)
def test_evaluate_respects_max_instances(self, render_queue_instance):
# One already running, only 1 slot for blender
render_queue_instance.job_queue.append(
make_mock_job('running-1', RenderStatus.RUNNING, engine_name='blender')
)
waiting = make_mock_job('waiting-1', RenderStatus.NOT_STARTED, engine_name='blender')
render_queue_instance.job_queue.append(waiting)
with patch.object(render_queue_instance, '_start_job') as mock_start:
with patch.object(render_queue_instance, '_save_state'):
RenderQueue.evaluate_queue()
mock_start.assert_not_called()
def test_evaluate_starts_scheduled_jobs(self, render_queue_instance):
past = datetime(2020, 1, 1)
job = make_mock_job(status=RenderStatus.SCHEDULED, scheduled_start=past)
render_queue_instance.job_queue.append(job)
with patch.object(render_queue_instance, '_start_job') as mock_start:
with patch.object(render_queue_instance, '_save_state'):
RenderQueue.evaluate_queue()
mock_start.assert_called_once_with(job)
class TestClearHistory:
"""Clearing completed/cancelled/error jobs."""
def test_clear_history_removes_finished_jobs(self, render_queue_instance):
render_queue_instance.session = MagicMock()
jobs = [
make_mock_job('c', RenderStatus.COMPLETED),
make_mock_job('e', RenderStatus.ERROR),
make_mock_job('x', RenderStatus.CANCELLED),
make_mock_job('r', RenderStatus.RUNNING),
]
render_queue_instance.job_queue.extend(jobs)
RenderQueue.clear_history()
assert len(render_queue_instance.job_queue) == 1
assert render_queue_instance.job_queue[0].id == 'r'
+159
View File
@@ -0,0 +1,159 @@
from unittest.mock import MagicMock, patch
import pytest
from src.utilities.zeroconf_server import ZeroconfServer
class TestConfigure:
"""Configuring service parameters."""
def test_configure_sets_attributes(self, zeroconf_server_instance):
with patch('socket.gethostbyname', return_value='192.168.1.1'):
ZeroconfServer.configure('_zordon._tcp.local.', 'test-server', 8080)
assert zeroconf_server_instance.service_type == '_zordon._tcp.local.'
assert zeroconf_server_instance.server_name == 'test-server'
assert zeroconf_server_instance.server_port == 8080
def test_configure_calls_sync_class(self, zeroconf_server_instance):
with patch('socket.gethostbyname', return_value='192.168.1.1'):
ZeroconfServer.configure('_zordon._tcp.local.', 'test-server', 8080)
assert ZeroconfServer.service_type == '_zordon._tcp.local.'
assert ZeroconfServer.server_port == 8080
def test_configure_stops_on_gaierror(self, zeroconf_server_instance):
import socket
with patch('socket.gethostbyname', side_effect=socket.gaierror):
with patch.object(zeroconf_server_instance, '_stop') as mock_stop:
ZeroconfServer.configure('_zordon._tcp.local.', 'test-server', 8080)
mock_stop.assert_called_once()
class TestSyncClass:
"""_sync_class propagates instance attrs to class level."""
def test_sync_class_propagates_all_attrs(self, zeroconf_server_instance):
zeroconf_server_instance.service_type = '_test._tcp.'
zeroconf_server_instance.server_name = 'foo'
zeroconf_server_instance.server_port = 9999
zeroconf_server_instance.server_ip = '10.0.0.1'
zeroconf_server_instance.properties = {'key': 'val'}
ZeroconfServer._sync_class()
assert ZeroconfServer.service_type == '_test._tcp.'
assert ZeroconfServer.server_name == 'foo'
assert ZeroconfServer.server_port == 9999
assert ZeroconfServer.server_ip == '10.0.0.1'
assert ZeroconfServer.properties == {'key': 'val'}
class TestStartStop:
"""Service lifecycle."""
def test_start_raises_without_configure(self, zeroconf_server_instance):
with pytest.raises(RuntimeError, match='configure'):
ZeroconfServer.start()
@patch('src.utilities.zeroconf_server.ServiceBrowser')
def test_start_listen_only_skips_register(self, mock_browser, zeroconf_server_instance):
with patch('socket.gethostbyname', return_value='192.168.1.1'):
ZeroconfServer.configure('_zordon._tcp.local.', 'test', 8080)
with patch.object(zeroconf_server_instance, '_register_service') as mock_register:
ZeroconfServer.start(listen_only=True)
mock_register.assert_not_called()
@patch('src.utilities.zeroconf_server.ServiceBrowser')
def test_start_registers_service(self, mock_browser, zeroconf_server_instance):
with patch('socket.gethostbyname', return_value='192.168.1.1'):
ZeroconfServer.configure('_zordon._tcp.local.', 'test', 8080)
with patch.object(zeroconf_server_instance, '_register_service') as mock_register:
ZeroconfServer.start(listen_only=False)
mock_register.assert_called_once()
def test_stop_unregisters_and_closes(self, zeroconf_server_instance):
zeroconf_server_instance.service_info = MagicMock()
with patch.object(zeroconf_server_instance, '_unregister_service') as mock_unreg:
with patch.object(zeroconf_server_instance.zeroconf, 'close') as mock_close:
ZeroconfServer.stop()
mock_unreg.assert_called_once()
mock_close.assert_called_once()
class TestRegisterService:
"""Service registration with Zeroconf."""
@patch('src.utilities.zeroconf_server.ServiceInfo')
@patch('socket.gethostbyname')
def test_registers_service_info(self, mock_gethostbyname, mock_service_info,
zeroconf_server_instance):
mock_gethostbyname.return_value = '192.168.1.1'
zeroconf_server_instance.service_type = '_zordon._tcp.local.'
zeroconf_server_instance.server_name = 'test'
zeroconf_server_instance.server_port = 8080
zeroconf_server_instance.properties = {}
# Replace real Zeroconf with a mock so we don't actually register
zeroconf_server_instance.zeroconf = MagicMock()
mock_info = MagicMock()
mock_service_info.return_value = mock_info
with patch('socket.inet_aton', return_value=b'\xc0\xa8\x01\x01'):
zeroconf_server_instance._register_service()
zeroconf_server_instance.zeroconf.register_service.assert_called_once_with(mock_info)
assert zeroconf_server_instance.service_info == mock_info
class TestFoundHostnames:
"""Discovery cache."""
def test_found_hostnames_empty_initially(self, zeroconf_server_instance):
result = zeroconf_server_instance._found_hostnames()
assert result == []
@patch('socket.gethostname', return_value='my-machine')
def test_hostnames_sorted_with_local_first(self, mock_hostname, zeroconf_server_instance):
zeroconf_server_instance.client_cache = {
'other-machine': MagicMock(),
'my-machine': MagicMock(),
}
result = zeroconf_server_instance._found_hostnames()
# sort_key returns False (0) for local → sorted first
assert result[0] == 'my-machine'
def test_get_hostname_properties_returns_decoded(self, zeroconf_server_instance):
info = MagicMock()
info.properties = {b'key': b'value', b'num': b'42'}
zeroconf_server_instance.client_cache['server-1'] = info
result = zeroconf_server_instance._get_hostname_properties('server-1')
assert result == {'key': 'value', 'num': '42'}
def test_get_hostname_properties_returns_empty_for_unknown(self, zeroconf_server_instance):
result = zeroconf_server_instance._get_hostname_properties('unknown')
assert result == {}
class TestForwarders:
"""Classmethod forwarders delegate to instance."""
def test_found_hostnames_forwarder(self, zeroconf_server_instance):
zeroconf_server_instance.client_cache['svr'] = MagicMock()
result = ZeroconfServer.found_hostnames()
assert 'svr' in result
def test_get_hostname_properties_forwarder(self, zeroconf_server_instance):
info = MagicMock()
info.properties = {}
zeroconf_server_instance.client_cache['svr'] = info
result = ZeroconfServer.get_hostname_properties('svr')
assert result == {}