# Unreal Zen Storage Service (aka "Zen Server") This is the implementation of the local storage service for UE5. It is intended to be deployed on user machines either as a daemon or launched ad hoc as required during of editor/cooker/game startup. The engine will use this service to manage local storage of secondary data such as the output from the cooker and local caches used to accelerate data transformations (aka DDC). Unreal Zen Storage Service can also be deployed as a shared instance for use as a shared cache (DDC). This is recommended for high load environments such as a build farm since we can handle significantly higher request rates compared to plain file share (SMB) and Cloud DDC. For a shared instance we strongly recommend using Windows OS and the http.sys path since this is the highest throughput option as of this writing. # Build Instructions ## Building on Windows Windows is our primary development platform and is generally the easiest to debug and profile on. For debugging we recommend Visual Studio 2022 or later, and for profiling you may want to use a high frequency sampling profiler such as Superluminal Performance. ### Windows Setup To build the code you will need Visual Studio 2022 (we use c++20 features), and git. * Install Visual Studio 2022 * `winget install Microsoft.VisualStudio.2022.Professional --silent --override "--wait --quiet --add ProductLang En-us --add Microsoft.VisualStudio.Workload.NativeDesktop --includeRecommended"` * Install [git](https://git-scm.com/download/win) * You can also use `winget install git.git` if you have winget installed * You may want to install the github CLI to manage credentials etc - `winget install github.cli` * Install [xmake](https://xmake.io/#/getting_started) * You can also use `winget install xmake` if you have winget installed ### Building with Visual Studio We currently require a C++ compiler from Visual Studio 2022 or later. The following steps need to be run from a Command Prompt window or a Terminal instance * Install Visual Studio 2022 * clone the `zen` repository if you haven't already. This requires a valid github login and you need to be part of the EpicGames organization * run `git clone https://github.com/EpicGames/zen.git` or `gh repo clone EpicGames/zen` (you may want to use `gh auth login` to log in to github as well) * run `xmake sln` or `xmake project -k vsxmake2022 -a x64 -y` to generate a Visual Studio solution file * open the `vsxmake2022\zen.sln` VS solution (`start vsxmake2022\zen.sln`) * Note: if you want full connectivity with the http.sys server implementation you currently need to run Visual Studio in ADMIN mode since http.sys requires elevation to be able to listen on a non-local network socket. You can start Visual Studio in admin mode by holding CTRL-SHIFT when launching Visual Studio. Alternatively you can add an URL reservation (see below) * you can now build and run `zenserver` as usual from Visual Studio. Note that the test suite is compiled into the Debug configuration only at this time, to keep Release lean ### Building with xmake on the command line * configure xmake: `xmake config -m debug` * build zenserver: `xmake build zenserver` * build all targets at once: `xmake` Compiled binaries end up in the `build` directory. xmake uses the `.xmake` directory to store project specific state. For more tips on working with xmake, see [xmake notes](docs/xmake.md) ### Installing pre-commit (optional but recommended) This is necessary to run pre-commit locally, which is useful in particular to run clang-format prior to commit. * Make sure python3 is installed. Version 3.11 or later should work * You can install using `winget install python3` * Run `pip install pre-commit==3.2.0` (later versions may or may not work) * If you want the pre-commit steps to be run at commit time (which is recommended) you can run `pre-commit install` within your local repo. Once you have pre-commit installed, it's convenient to use the provided xmake task to trigger it manually to apply clang-format rules. You can do this by running `xmake precommit` on the command line. ## Building on Linux The following instructions have been collated using Ubuntu 20.04. Zen makes use of C++20 features which at the time of writing has limited toolchain and C++ library support. A minimum compiler version of GCC-11 or Clang-12 is required, along with GNU's libstdc+++-11 or newer. The first step is to acquire a suitable compiler and C++ library. On Ubuntu 20.04 GCC-11 is not available in the standard package repositories so Ubuntu's tool- chain test respository needs to be added to Apt (note that this is _not required_ for Ubuntu 21.04 onwards); ``` sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test ``` Now GCC-11 can be installed via Apt. This will also install a suitable version of the C++ library. ``` sudo apt install -y --no-install-recommends g++-11 g++-11 --version ``` The easiest way to tell `xmake` to use the correct compiler version is to set the `CXX` environment variable. ``` export CXX=g++-11 ``` Install [xmake](https://xmake.io/#/getting_started). ``` curl -fsSL https://xmake.io/shget.text | bash xmake --version ``` Clone the Zen project and tell `xmake` to use the correct GCC version. ``` git clone https://github.com/EpicGames/zen.git ~/zen/main cd ~/zen/main ``` Now we are ready to build Zen. The `-y` skips `xmake` from prompting about building packages. ``` xmake config -y --mode=debug xmake build ``` Note that the command above to set the build variant to debug is optional. Tests are only built in debug.The `xmake` flags `-vD` can be useful to diagnose `xmake` issues. ## Building on Mac Building on Mac is very similar to Linux; install xmake, clone Zen and then build the project. You will also need to install Xcode or Xcode command line tools. ### [xmake](https://xmake.io/#/getting_started) ``` curl -fsSL https://xmake.io/shget.text | bash xmake --version ``` ### Zen ``` git clone https://github.com/EpicGames/zen.git ~/zen/main cd ~/zen/main ``` ### Build ``` xmake config -y --mode=debug xmake build ``` ### Multi-user scenarios Complications can arise in situations where hardware is shared between two or more users. To mitigate this a local version of `brew` can be used as follows; ``` git clone https://github.com/Homebrew/brew.git ~/zen/brew export PATH=~/zen/brew/bin:$PATH brew tap brew install xmake brew install pkgconfig ``` # Implementation Notes * The implementation currently depends only on a few libraries including the C++ standard library * In a few places we also use EASTL, mostly to eliminate memory allocations by using the excellent `eastl::fixed_vector` et al containers which contain an embedded memory buffer which is used until capacity is exceeded at which point it switches to the heap * It uses exceptions for certain types of unexpected error conditions. Exceptions should not be used for other flow control. # Contributing Code See [CODING.md](docs/CODING.md) # Debugging When debugging multi-process scenarios such as when running `zenserver-test`, the [Microsoft Child Process Debugging Power Tool](https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool) is incredibly handy. When that is installed you may enable auto-attach to child processes via the Debug -> Other Debug Targets menu in Visual Studio. # Testing * There are some test projects * `zencore-test` exercises unit tests in the zencore project * `zenserver-test` exercises the zen server itself (functional tests) The tests are implemented using [doctest](https://github.com/onqtam/doctest), which is similar to Catch in usage. # Adding a http.sys URL reservation (Windows only) Registering a handler for an HTTP endpoint requires either process elevation (i.e running Zen as admin) or a one-time URL reservation. An URL reservation can be added by issuing a command like `netsh http add urlacl url=http://*:8558/ user=stefan.boberg` (enable for a specific user) or `netsh http add urlacl url=http://*:8558/ sddl=D:(A;;GX;;;S-1-1-0)` (enable for any authenticated user)