path: root/README.md

# Unreal Zen Storage Service (aka "Zen Server")

This is the implementation of the local storage service for UE5. It is intended to be deployed on
user machines either as a daemon or launched ad hoc as required during of editor/cooker/game startup. The engine
will use this service to manage local storage of secondary data such as the output from the cooker and local caches
used to accelerate data transformations (aka DDC).

Unreal Zen Storage Service can also be deployed as a shared instance for use as a shared cache (DDC). This is recommended
for high load environments such as a build farm since we can handle significantly higher request rates compared to
plain file share (SMB) and Cloud DDC.

We also supports upstream  connectivity to cloud storage services as well as other Zen server instances. 
However, due to how upstream/downstream propagation is handled this is not currently recommended as performance 
will be better if you use a hierarchical DDC graph in UE. 

# Build Instructions

## Building on Windows

Windows is our primary development platform and is generally the easiest to debug and profile on. For
debugging we recommend Visual Studio 2022 or later, and for profiling you may want to use a high 
frequency sampling profiler such as Superluminal Performance.

### Windows Setup

To build the code you will need Visual Studio 2022 (we use c++20 features), git and vcpkg.

* Install Visual Studio 2022
  * `winget install Microsoft.VisualStudio.2022.Professional --silent --override "--wait --quiet --add ProductLang En-us --add Microsoft.VisualStudio.Workload.NativeDesktop --includeRecommended"`
* Install [git](https://git-scm.com/download/win)
  * You can also use `winget install git.git` if you have winget installed
  * You may want to install the github CLI to manage credentials etc - `winget install github.cli`
* Install [xmake](https://xmake.io/#/getting_started)
  * You can also use `winget install xmake` if you have winget installed
* Install vcpkg (see below)

#### Installing vcpkg

We use vcpkg to manage some libraries. Right now it's not set up on a project local
basis and requires manual bootstrap so you will need to do the following at least once:

* open up a command line window
  * create a `git`/`github` directory somewhere for you to clone repos into
  * issue `git clone https://github.com/microsoft/vcpkg.git` and build it using the `bootstrap-vcpkg.bat` script
* optional: add the `vcpkg` directory you cloned to your PATH to allow invoking vcpkg on the command line
* issue `vcpkg integrate install` to make sure xmake and Visual Studio can locate the vcpkg install

Now you are ready to start building!

### Building with Visual Studio

We currently require a C++ compiler from Visual Studio 2022 or later. The following steps need to be run 
from a Command Prompt window or a Terminal instance

* Install Visual Studio 2022
* clone the `zen` repository if you haven't already. This requires a valid github login and you need to 
  be part of the EpicGames organization
  * run `git clone https://github.com/EpicGames/zen.git` or `gh repo clone EpicGames/zen` (you may want 
    to use `gh` to log in to github as well)
* run `xmake sln` or `xmake project -k vsxmake2022 -a x64 -y` to generate a Visual Studio solution file
* open the `vsxmake2022\zen.sln` VS solution (`start vsxmake2022\zen.sln`)
  * Note: if you want full connectivity with the http.sys server implementation you currently need to run 
    Visual Studio in ADMIN mode since http.sys requires elevation to be able to listen on a non-local network socket.
    You can start Visual Studio in admin mode by holding CTRL-SHIFT when launching Visual Studio. Alternatively
    you can add an URL reservation (see below)
  * you can now build and run `zenserver` as usual from Visual Studio. Note that the test suite is compiled into 
    the Debug configuration only at this time, to keep Release lean
  * third-party dependencies will be built the first time via the `vcpkg` integration. This is not as
    fast as it could be (it does not go wide) but should only happen on the first build and will leverage
    a local build cache

### Building with xmake on the command line

* configure xmake: `xmake config -m debug|release -a x64`
* build zenserver: `xmake build zenserver`
  * build all targets at once: `xmake`

Compiled binaries are located in the `build` directory

#### Installing pre-commit (optional)

This is necessary to run pre-commit locally, which is useful in particular to run clang-format prior to
commit.

* Make sure python3 is installed. Version 3.11 or later should work
  * You can install using `winget install python3`
* Run `pip install pre-commit==3.2.0` (later versions may or may not work)
* If you want the pre-commit steps to be run at commit time you can run `pre-commit install` within your local repo.

## Building on Linux

The following instructions have been collated using Ubuntu 20.04.

Zen makes heavy use of C++20 features which at the time of writing has limited
toolchain and C++ library support. A minimum compiler version of GCC-11 or
Clang-12 is required, along with GNU's libstdc+++-11 or newer. Note that it is
not currently possible to use anything other than the system's default C++
library (e.g. LLVM's libc++) due to the way that Zen's third party dependencies
are managed via `vcpkg`.

The first step is to acquire a suitable compiler and C++ library. On Ubuntu 20.04
GCC-11 is not available in the standard package repositories so Ubuntu's tool-
chain test respository needs to be added to Apt (note that this is _not required_
for Ubuntu 21.04 onwards);

```
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
```

Now GCC-11 can be installed via Apt. This will also install a suitable version of
the C++ library.

```
sudo apt install -y --no-install-recommends g++-11
g++-11 --version
```

The easiest way to tell `xmake` to use the correct compiler version is to set
the `CXX` environment variable.

```
export CXX=g++-11
```

Next we need the `xmake` build system. For this we will download and install
`xmake` as a `.deb` package. Check the project's release page for more up to
date `.deb` files.

```
wget https://github.com/xmake-io/xmake/releases/download/v2.8.2/xmake-v2.8.2.amd64.deb
sudo dpkg -i xmake-v2.8.2.amd64.deb
xmake --version
```

To build some of the packages from vcpkg you may need some additional tools.

```
sudo apt-get install build-essential zip pkg-config
```

Some of Zen's third party dependencies are provided by Microsoft's `vcpkg`
C++ library manager. After cloning the project there is an initialisation step.

```
git clone https://github.com/microsoft/vcpkg.git ~/zen/vcpkg
~/zen/vcpkg/bootstrap-vcpkg.sh
~/zen/vcpkg/vcpkg --version
```

`xmake` uses an environment variable to find `vcpkg`. Alternatively this can be
done by including `VCPKG_ROOT=...` on the command line when invoking `xmake`.

```
export VCPKG_ROOT=~/zen/vcpkg
```

Clone the Zen project and tell `xmake` to use the correct GCC version.

```
git clone https://github.com/EpicGames/zen.git ~/zen/main
cd ~/zen/main
```

Now we are ready to build Zen.  The `-y` skips `xmake` from prompting about
updating `vcpkg` packages.

```
xmake config -y --mode=debug
xmake build
```

Note that the command above to set the build variant to debug is optional. Tests
are only built in debug.The `xmake` flags `-vD` can be useful to diagnose
`xmake` issues.

### Linux distribution compatibility

Builds for Linux have a dependency on a modern version of the C++ library that
supports Zen's use of C++20 and the GCC-11 toolchain used (assuming the above
build instructions are adhered to). However there is no guarantee that the this
dependency is met when building on one Linux install and running on another. For
example, at the time of writing the LTS version of Ubuntu and Debian do not have
GCC-11's version of libstdc++.so.6 available in their package repositories.

To solve this, `xmake bundle` will bundle the required C++ .so and zenserver
binary together using AppImage. This can briefly be summarised as create a
squashfs file system containing zenserver and any C++ shared objects list with
`lld`. This file system is then concatenated with AppImage's runtime. This is
implemented in `scripts/bundle_linux.sh`. More details can be found here;

https://github.com/AppImage/AppImageKit

## Building on Mac

Building on Mac is very similar to Linux; install xmake, clone vcpkg and Zen and
then build the project. You will also need to install Xcode or Xcode command line
tools.

```
## vcpkg
git clone https://github.com/microsoft/vcpkg.git ~/zen/vcpkg
~/zen/vcpkg/bootstrap-vcpkg.sh
~/zen/vcpkg/vcpkg --version

## xmake
curl -L https://github.com/xmake-io/xmake/releases/download/v2.8.2/xmake-v2.8.2.macosx.pkg > xmake-v2.8.2.macosx.pkg
sudo installer -pkg xmake-v2.8.2.macosx.pkg -target /

## Zen
git clone https://github.com/EpicGames/zen.git ~/zen/main
cd ~/zen/main

## Build
export VCPKG_ROOT=~/zen/vcpkg
xmake config -y --mode=debug --arch=x86_64
xmake build
```

# Implementation Notes

* The implementation currently depends only on a few libraries including the C++ standard library
* It uses exceptions for certain types of unexpected error conditions

# Contributing Code

See [CODING.md](docs/CODING.md)

# Debugging

When debugging multi-process scenarios such as when running `zenserver-test`, the [Microsoft Child Process Debugging Power Tool](https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool) 
is incredibly handy. When that is installed you may enable auto-attach to child processes via the Debug -> Other Debug Targets menu in Visual Studio.

# Testing

* There are some test projects
  * `zencore-test` exercises unit tests in the zencore project
  * `zenserver-test` exercises the zen server itself (functional tests)

The tests are implemented using [doctest](https://github.com/onqtam/doctest), which is similar to Catch in usage.

# Adding a http.sys URL reservation (Windows only)

Registering a handler for an HTTP endpoint requires either process elevation (i.e running Zen as admin) or a one-time URL reservation. An URL reservation can be added by issuing a command like

`netsh http add urlacl url=http://*:8558/ user=stefan.boberg` (enable for a specific user)

or 

`netsh http add urlacl url=http://*:8558/ sddl=D:(A;;GX;;;S-1-1-0)` (enable for any authenticated user)