1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
|
# Zen Storage Service
Zen is a high-performance *local storage* service for Unreal Engine 5. It manages
*secondary* data such as cooker output, derived data caches (DDC), project
metadata, and build artifacts. In addition to storage, Zen can schedule certain
data transformations (shader compilation, texture compression, etc.) for
distributed execution.
Zen is primarily deployed as a daemon on user machines for local caching of
cooker artifacts, but can also serve as a high-performance shared instance for
build farms and teams — complementing and offloading Unreal Cloud DDC.
## Storage Services
### DDC (Derived Data Cache) / Structured Cache
The primary workload alongside the Project Store. Caches derived data produced
by the Unreal Engine cooker — cooked assets, shader bytecode, texture compilations,
and other intermediate artifacts. Entries are keyed by a hash derived from a set of
inputs, and organized into namespaces and buckets. See [Cache](cache.md).
### Content-Addressable Storage (CAS)
The lowest storage layer. Data is stored and retrieved by its content hash
(IoHash). CAS provides automatic deduplication — identical content always
produces the same hash. Higher-level services reference CAS objects via
attachments.
### Project Store
The project store stores cooked asset data and associated metadata required
for dependency tracking by the cooker.
Entries have configurable expiration and are garbage collected accordingly.
See [Projects](projects.md)
### Build Store
Storage for build artifacts (staged or packaged builds) with configurable
retention. Build store entries are subject to garbage collection when they
exceed their maximum duration.
This is an optional feature, and it is disabled by default.
### Object Store
General-purpose key-value storage organized by namespace, with support for
associated metadata. Similar in spirit to S3 in that it presents a generic
key-value store interface.
Primarily intended to be used for certain developer workflows, such as when
working with UE5 Individal Asset Streaming/Download. This is optional and
disabled by default.
## Compute Services
Zen can optionally act as a remote executor for UE5's Derived Data Build
interface, distributing fine-grained actions across multiple workers with
low-latency scheduling.
See [Compute](compute.md) for details.
## Supporting Services
### Sessions
Tracks client connections from Unreal Editor instances, build agents, and other
tools. Each session carries an application name, mode, metadata, and a log
stream visible in the web dashboard.
### Garbage Collection
Automatic management of storage lifecycle. Full GC scans all references, removes
expired data, prunes unreferenced CAS content, and compacts fragmented storage.
Lightweight GC runs more frequently with a focus on expiration only.
## Server Modes
Zenserver supports several operational modes, selected at launch:
### Storage Server
The primary mode, running all storage and cache services.
### Hub
A coordination service for managing multiple storage instances on the
same host.
```mermaid
graph LR
Client[UE5 Client] <--> Zen[Zen Server]
Client2[UE5 Client II] <--> Zen2[Zen Server]
subgraph User 1
Client
end
subgraph User 2
Client2
end
subgraph Hub[Hub Instance]
Zen
Zen2[Zen Server]
HubServer[Hub Server]
end
Orchestrator --> HubServer
```
### Compute
Action processing endpoint for distributed Derived Data Build workloads.
```mermaid
graph LR
Client[UE5 Client] <--> Zen[Zen Server]
Zen <--> Zen2[Compute]
Zen <--> Zen3[Compute]
Zen <--> Zen4[Compute]
```
### Proxy
A relay server that forwards requests to upstream storage instances, with
optional stream parsing and recording. Primarily intended for use during
development to aid in visualising component interactions in advanced setups.
```mermaid
graph LR
Client[UE5 Client] <--> Proxy <--> Zen[Zen Server]
Proxy <--> Zen2[Zen Server]
```
## Architecture
```mermaid
graph TD
Client[UE5 Client] --> DDC
Client --> PRJ
Client --> OBJ
subgraph Zen[Zen Server]
DDC[DDC Cache]
PRJ[Project Store]
OBJ[Object Store]
CAS[Content-Addressable Storage]
end
DDC --> CAS
PRJ --> CAS
DDC -.-> Upstream[Upstream Cache]
```
## Web Dashboard
Zenserver includes an embedded web dashboard providing real-time visibility into:
- **Storage** — disk usage, directory breakdown, cache namespaces, and garbage
collection status and history
- **Network** — HTTP request rates, latency percentiles, and bandwidth by
service
- **Sessions** — active and ended sessions with live log streaming
- **Info** — server configuration, build info, and runtime state
## CLI
The `zen` command-line utility provides commands for interacting with a running
zenserver:
- **Server management** — `serve`, `up`, `admin`
- **Cache operations** — `cache`, `projectstore`, `workspaces`
- **Storage operations** — `copy`, `dedup`, `vfs`, `wipe`
- **Monitoring** — `status`, `info`, `top`, `trace`, `bench`
- **Diagnostics** — `ui` (launch web dashboard), `version`
## HTTP Server
Zenserver uses platform-optimized HTTP serving:
- **Windows** — http.sys kernel-mode driver for maximum throughput (default),
with ASIO as a fallback
- **Linux / macOS** — ASIO-based asynchronous I/O server
The server implementation can be overridden with `--http=asio` on any platform.
|