# Zen Storage Service

Zen is a high-performance *local storage* service for Unreal Engine 5. It manages
*secondary* data such as cooker output, derived data caches (DDC), project
metadata, and build artifacts. In addition to storage, Zen can schedule certain
data transformations (shader compilation, texture compression, etc.) for
distributed execution.

Zen is primarily deployed as a daemon on user machines for local caching of
cooker artifacts, but can also serve as a high-performance shared instance for
build farms and teams — complementing and offloading Unreal Cloud DDC.

## Storage Services

### DDC (Derived Data Cache) / Structured Cache

The primary workload alongside the Project Store. Caches derived data produced 
by the Unreal Engine cooker — cooked assets, shader bytecode, texture compilations, 
and other intermediate artifacts. Entries are keyed by a hash derived from a set of
inputs, and organized into namespaces and buckets. See [Cache](cache.md).

### Content-Addressable Storage (CAS)

The lowest storage layer. Data is stored and retrieved by its content hash
(IoHash). CAS provides automatic deduplication — identical content always
produces the same hash. Higher-level services reference CAS objects via
attachments.

### Project Store

The project store stores cooked asset data and associated metadata required
for dependency tracking by the cooker.

Entries have configurable expiration and are garbage collected accordingly.

See [Projects](projects.md)

### Build Store

Storage for build artifacts (staged or packaged builds) with configurable 
retention. Build store entries are subject to garbage collection when they 
exceed their maximum duration.

This is an optional feature, and it is disabled by default.

### Object Store

General-purpose key-value storage organized by namespace, with support for
associated metadata. Similar in spirit to S3 in that it presents a generic
key-value store interface.

Primarily intended to be used for certain developer workflows, such as when 
working with UE5 Individal Asset Streaming/Download. This is optional and
disabled by default.

## Compute Services

Zen can optionally act as a remote executor for UE5's Derived Data Build
interface, distributing fine-grained actions across multiple workers with 
low-latency scheduling.

See [Compute](compute.md) for details.

## Supporting Services

### Sessions

Tracks client connections from Unreal Editor instances, build agents, and other
tools. Each session carries an application name, mode, metadata, and a log
stream visible in the web dashboard.

### Garbage Collection

Automatic management of storage lifecycle. Full GC scans all references, removes
expired data, prunes unreferenced CAS content, and compacts fragmented storage.
Lightweight GC runs more frequently with a focus on expiration only.

## Server Modes

Zenserver supports several operational modes, selected at launch:

### Storage Server

The primary mode, running all storage and cache services.

### Hub

A coordination service for managing multiple storage instances on the
same host.

```mermaid
graph LR
    Client[UE5 Client] <--> Zen[Zen Server]
    Client2[UE5 Client II] <--> Zen2[Zen Server]

    subgraph User 1
      Client
    end

    subgraph User 2
      Client2
    end

    subgraph Hub[Hub Instance]
      Zen
      Zen2[Zen Server]
      HubServer[Hub Server] 
    end

    Orchestrator --> HubServer
```

### Compute

Action processing endpoint for distributed Derived Data Build workloads.

```mermaid
graph LR
    Client[UE5 Client] <--> Zen[Zen Server]
    Zen <--> Zen2[Compute]
    Zen <--> Zen3[Compute]
    Zen <--> Zen4[Compute]
```

### Proxy

A relay server that forwards requests to upstream storage instances, with
optional stream parsing and recording. Primarily intended for use during
development to aid in visualising component interactions in advanced setups.

```mermaid
graph LR
    Client[UE5 Client] <--> Proxy <--> Zen[Zen Server]
    Proxy <--> Zen2[Zen Server]
```

## Architecture

```mermaid
graph TD
    Client[UE5 Client] --> DDC
    Client --> PRJ
    Client --> OBJ

    subgraph Zen[Zen Server]
        DDC[DDC Cache]
        PRJ[Project Store]
        OBJ[Object Store]
        CAS[Content-Addressable Storage]
    end

    DDC --> CAS
    PRJ --> CAS

    DDC -.-> Upstream[Upstream Cache]
```

## Web Dashboard

Zenserver includes an embedded web dashboard providing real-time visibility into:

- **Storage** — disk usage, directory breakdown, cache namespaces, and garbage
  collection status and history
- **Network** — HTTP request rates, latency percentiles, and bandwidth by
  service
- **Sessions** — active and ended sessions with live log streaming
- **Info** — server configuration, build info, and runtime state

## CLI

The `zen` command-line utility provides commands for interacting with a running
zenserver:

- **Server management** — `serve`, `up`, `admin`
- **Cache operations** — `cache`, `projectstore`, `workspaces`
- **Storage operations** — `copy`, `dedup`, `vfs`, `wipe`
- **Monitoring** — `status`, `info`, `top`, `trace`, `bench`
- **Diagnostics** — `ui` (launch web dashboard), `version`

## HTTP Server

Zenserver uses platform-optimized HTTP serving:

- **Windows** — http.sys kernel-mode driver for maximum throughput (default),
  with ASIO as a fallback
- **Linux / macOS** — ASIO-based asynchronous I/O server

The server implementation can be overridden with `--http=asio` on any platform.