aboutsummaryrefslogtreecommitdiff
path: root/docs/compute.md
diff options
context:
space:
mode:
authorStefan Boberg <[email protected]>2026-03-23 12:23:19 +0100
committerGitHub Enterprise <[email protected]>2026-03-23 12:23:19 +0100
commit26aa50677403e4c5ad053b221bc7264fe1d249f2 (patch)
treede196528390e8875b0551d52071038120d969f73 /docs/compute.md
parentProcess management improvements (#881) (diff)
downloadzen-26aa50677403e4c5ad053b221bc7264fe1d249f2.tar.xz
zen-26aa50677403e4c5ad053b221bc7264fe1d249f2.zip
Documentation updates (#882)
Restructured the docs folder in preparation for more docs. Improved the contents a bit.
Diffstat (limited to 'docs/compute.md')
-rw-r--r--docs/compute.md92
1 files changed, 71 insertions, 21 deletions
diff --git a/docs/compute.md b/docs/compute.md
index df8a22870..06453c975 100644
--- a/docs/compute.md
+++ b/docs/compute.md
@@ -1,29 +1,33 @@
-# DDC compute interface design documentation
+# Zen Compute
-This is a work in progress
+> **Note:** Compute interfaces are a work in progress and are not yet included
+> in official binary releases.
-## General architecture
+Zen server implements a compute interface for managing distributed processing
+via the UE5 DDC 2.0 Build API.
-The Zen server compute interfaces implement a basic model for distributing compute processes.
-Clients can implement [Functions](#functions) in [worker executables](#workers) and dispatch
-[actions](#actions) to them via a message based interface.
+## Compute Model
-The API requires users to describe the actions and the workers explicitly fully up front and the
-work is described and submitted as singular objects to the compute service. The model somewhat
-resembles Lambda and other stateless compute services but is more tightly constrained to allow
-for optimizations and to integrate tightly with the storage components in Zen server.
+The compute interface implements a "pure function" model for distributing work,
+similar in spirit to serverless compute paradigms like AWS Lambda.
-This is in contrast with Unreal Build Accelerator in where the worker (remote process)
-and the inputs are discovered on-the-fly as the worker progresses and inputs and results
-are communicated via relatively high-frequency RPCs.
+Clients implement transformation [Functions](#functions) in
+[worker executables](#workers) and dispatch [Actions](#actions) to them via a
+message-based interface.
-### Actions
+Actions and workers must be described explicitly and fully up front — work is
+submitted as self-contained objects to the compute service. This is more
+constrained than general-purpose serverless platforms, which allows for
+optimizations and tight integration with Zen's storage model.
-An action is described by an action descriptor, which is a compact binary object which
-contains a self-contained description of the inputs and the function which should be applied
-to generate an output.
-#### Sample Action Descriptor
+## Actions
+
+An action is the unit of work in the compute model. It is described by an action
+descriptor — a Compact Binary object containing a self-contained description of
+the inputs and the function to apply to produce an output.
+
+### Sample Action Descriptor
```
work item 4857714dee2383b50b2e7d72afd79848ab5d13f8 (2 attachments):
@@ -39,7 +43,7 @@ Inputs:
RawSize: 3334
```
-### Functions
+## Functions
Functions are identified by a name, and a version specification. For
matching purposes there's also a build system version specification.
@@ -56,7 +60,7 @@ function version build system
CompileShaderJobs 83027356-2cf7-41ca-aba5-c81ab0ff2129 17fe280d-ccd8-4be8-a9d1-89c944a70969 69cb9bb50e9600b5bd5e5ca4ba0f9187b118069a
```
-### Workers
+## Workers
A worker is an executable which accepts some command line options which are used to pass the
information required to execute an action. There are two modes, one legacy mode which is
@@ -87,7 +91,7 @@ communication with the invoking program (the 'build system'). To be able to evol
interface, each worker also indicates the version of the build system using the
`BuildSystemVersion` attribute.
-#### Sample Worker Descriptor
+### Sample Worker Descriptor
```
worker 69cb9bb50e9600b5bd5e5ca4ba0f9187b118069a:
@@ -201,3 +205,49 @@ worker resolution.
worker.
`/compute/queues/{oidtoken}/jobs/{lsn}` - GET action result by LSN, scoped to the queue
+
+## Relationship to Unreal Build Accelerator
+
+Zen Compute is designed to complement Unreal Build Accelerator (UBA), not
+replace it. The two systems target different workload characteristics:
+
+- **Zen Compute** — suited to workloads where the inputs and function are fully
+ known before execution begins. All data is declared up front in the action
+ descriptor, and the worker runs as a self-contained transformation. This
+ enables content-addressed caching of results and efficient scheduling.
+
+- **UBA** — suited to workloads where inputs are discovered dynamically as the
+ process runs. The remote process and its dependencies are resolved on the fly,
+ with inputs and results exchanged via high-frequency RPCs throughout execution.
+
+In practice, Zen Compute handles workloads like shader compilation where the
+inputs are well-defined, while UBA handles more complex build processes with
+dynamic dependency graphs.
+
+## Execution Flow
+
+```mermaid
+sequenceDiagram
+ participant C as Client
+ participant G as Zen Server
+ participant Q as Runner
+ participant W as Worker
+
+ C->>G: POST /jobs
+ G-->>C: 202 Accepted (job_id)
+ G->>Q: enqueue(action)
+ Q-->>G: job_id
+
+ C->>G: GET /jobs/job_id
+ G-->>C: 202 Accepted (job_id)
+
+ Q->>W: spawn()
+ Q-->>W: action
+ W->>W: process
+ W->>Q: complete(job_id)
+
+ C->>G: GET /jobs/job_id
+ G->>Q: status(job_id)
+ Q-->>G: done
+ G-->>C: 200 OK (result)
+```