1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
|
# Storage
## DDC (Derived Data Cache)
A cache for derived data produced by the Unreal Engine cooker. DDC entries are
keyed by a hash (typically generated from the input data combined with code
versions and parameters) and typically contain cooked assets, shader bytecode, or
other build artifacts. DDC is one of the primary workloads served by zen.
### Cache Namespace
A logical partition within the cache service. Each namespace (e.g.
`ns_ue.ddc`) has its own set of buckets and storage accounting. Namespaces
allow different types of cached data to be managed independently.
### Cache Bucket
A subdivision within a cache namespace. Buckets provide fine-grained grouping of
cache entries, typically corresponding to a specific type of derived data (e.g.
`bulkdatalist`, `animationsequence`).
## Project Store
A storage service for per-project data such as asset metadata and editor state.
Like the build store, project store entries have configurable expiration and are
garbage collected accordingly.
## CAS (Content-Addressable Storage)
The lowest storage layer in zen. Data is stored and retrieved by its content hash
(IoHash) also known as Content Id (aka 'cid'). CAS provides automatic deduplication
since identical content always produces the same hash. CAS content is garbage
collected based on what is referenced from higher-level services.
Content is immutable once written to CAS. Note that zenserver always stores data
in `CompressedBuffer` format, and the Content Id is derived from the *decompressed*
data.
### IoHash
A 20-byte (160-bit) content hash used to identify data in CAS. IoHash is the
fundamental addressing unit for content-addressable operations.
### Attachment
A reference from a higher-level store (build store, project store, or cache) to
a CAS object. Attachments are what keep CAS objects alive during garbage
collection — unreferenced CAS content is eligible for removal.
## Build Store
A storage service for build artifacts. Build store entries have configurable
expiration and are subject to garbage collection when they exceed their maximum
retention duration.
## Object Store
A general-purpose object storage service for arbitrary key-value data. Objects
are organized by namespace and can have associated metadata.
# Garbage Collection
## Full GC
A complete garbage collection pass that scans all referencers and reference
stores, removes expired data, prunes unreferenced CAS content, and optionally
compacts storage blocks. Full GC runs at a configured interval (typically hours).
## Lightweight GC
A faster, less thorough garbage collection pass focused on removing expired
entries without performing full reference scanning or compaction. Runs more
frequently than full GC.
## Referencer
A component that holds references to CAS content (e.g. the cache service,
project store, or build store). During GC, each referencer is scanned to
determine which CAS objects are still in use.
## Reference Store
The CAS-side counterpart to a referencer. During GC, reference stores are scanned
to identify and remove CAS objects that are no longer referenced by any
referencer.
## Compaction
A GC phase that reclaims fragmented disk space by rewriting storage blocks. A
block is compacted when its usage falls below the configured threshold
percentage. Compaction frees disk space but is more expensive than simple
deletion.
## Write Block
A period during GC where write operations to the affected stores are temporarily
blocked to ensure consistency. The write block duration is tracked as a GC
performance metric.
# Network
## http.sys
The Windows kernel-mode HTTP server driver used by zen on Windows for maximum
throughput. Requires either administrator privileges or a URL reservation to bind
to network interfaces. Can be overridden with `--http=asio` to use the
user-mode ASIO server instead.
## ASIO Server
An asynchronous I/O based HTTP server implementation available on all platforms.
This is the default on Linux and macOS, and can be used on Windows as an
alternative to http.sys.
# Sessions
## Session
A logical connection from a client application (e.g. Unreal Editor, a build
agent) to the zen server. Each session has an ID, application name, mode, and
tracks activity timestamps. Sessions can carry metadata and log entries.
## Session Log
A per-session log stream visible in the sessions browser UI. The server's own
session log captures zen's internal log output, while external sessions can
post log entries via the API.
# General
## zen CLI
The command-line client utility (`zen`) for interacting with a running
zenserver. Provides commands for cache management, server status, diagnostics,
and more.
## zenserver
The main server binary that integrates all zen storage services, the HTTP
server, telemetry, and the web dashboard. It can be launched in a number
of modes (storage, compute, proxy, hub) depending on the desired functionality.
|