Documentation updates (#882)

Restructured the docs folder in preparation for more docs. Improved the contents a bit.

6 files changed, 262 insertions, 0 deletions

diff --git a/docs/dev/CODING.md b/docs/dev/CODING.md
new file mode 100644
index 000000000..8924c8107
--- /dev/null
+++ b/docs/dev/CODING.md
@@ -0,0 +1,30 @@
+# Naming Conventions
+
+The naming conventions for Zen are intended to remain close to the Unreal Engine coding style, with some minor exceptions wrt prefixes.
+
+* Classes/Structs - `PascalCase`
+* Functions - `PascalCase()`
+* Class member variables - `m_PascalCase`
+* Global variables - `g_PascalCase`
+* Static variables - `s_PascalCase`
+* Thread local variables - `t_PascalCase`
+
+Those who are familiar with the UE coding standards will note that we do not require or encourage `F` prefixes on struct or classes, and we expect class members to have a `m_` member prefix.
+
+# Code formatting
+
+To ensure consistent formatting we rely on `clang-format` to automatically format source code. This leads to consistent formatting which should lead to fewer surprises and more straightforward merging.
+
+Formatting is ensured by using [pre-commit](https://pre-commit.com/)
+
+- [Ensure you have a Python install](https://www.python.org/downloads/) and that Python has been added to the PATH environment variable
+- [Install pre-commit](https://pre-commit.com/#installation) so it is available in PATH
+- Run pre-commit manually on staged files `pre-commit run`
+- Run pre-commit manually on all files `pre-commit run --all-files`. There is also a `xmake precommit` shortcut which may be easier to remember
+- Install git commit hooks `pre-commit install`, which will automatically run before every commit.
+
+# Standard Library / Containers
+
+Unlike UE5, use of `std` containers etc is acceptable though in some cases you may also consider using `eastl` equivalents which can
+sometimes enable more efficient runtime. In particular, `eastl::fixed_vector` et al can eliminate memory allocations by pre-allocating
+backing memory for common (small) sizes internally while still allowing larger sizes by overflowing to the heap.
diff --git a/docs/dev/Deploy.md b/docs/dev/Deploy.md
new file mode 100644
index 000000000..2a5e9be0f
--- /dev/null
+++ b/docs/dev/Deploy.md
@@ -0,0 +1,31 @@
+# Deploying a build to UE
+
+## Build
+
+Builds are produced via GitHub CI (Action Runners) and uploaded as artifacts
+to GitHub. See [create_release.yml](.github/workflows/create_release.yml) for
+details on what happens during the build.
+
+A new release is created when the [VERSION.txt](VERSION.txt) file changes.
+
+## Deploying into the UE tree
+
+After a successful release, you need to integrate the resulting binaries into
+the UE tree. There's a helper script for this in [scripts/download_artifacts.py](scripts/download_artifacts.py)
+
+You'll need a p4 workspace mapped to the stream or branch you want to deploy
+into, and a clone of the zen git repo. Since we use `gh` to download the 
+artifacts you'll need to use `gh auth login` and `gh repo clone` to clone
+the repo. With the current directory set to the root of the cloned git repo, 
+run the following command to download the artifacts corresponding to the 
+VERSION.txt in the repo by default.
+
+```bash
+> python scripts/download_artifacts.py <UE root path>
+```
+
+The script should go ahead and for each platform downloads the artifacts 
+using `gh artifact download` to a temporary directory, sync the corresponding
+files in Engine/Binaries/<platform> to #head and opens them for edit and
+copies the downloaded binaries into place, leaving the changes in your default
+changelist.
diff --git a/docs/dev/Github_runner_setup.md b/docs/dev/Github_runner_setup.md
new file mode 100644
index 000000000..42b2b1a01
--- /dev/null
+++ b/docs/dev/Github_runner_setup.md
@@ -0,0 +1,21 @@
+# Github action runner setup
+
+## Windows
+
+A Hyper-V "Quick Create" of a `Windows 11 dev environment` can give you a head start when setting up an instance for testing purposes. The following
+instructions should be sufficient to set up a completely clean Windows install.
+
+* For Windows Server, some gymnastics are required to get `winget` installed
+  * run `irm winget.pro | iex` in an Administrator Powershell window to automate this process (see https://github.com/asheroto/winget-install for other options)
+* Install git using `winget install git.git` (alternate: https://git-scm.com/download/win)
+* Install VS Pro using winget by following the relevant instructions in [README.md](/README.md) (note: ONLY Visual Studio)
+* Enable scripting using `Set-ExecutionPolicy -ExecutionPolicy Unrestricted` in an admin PowerShell instance
+  * In the same admin PowerShell instance, disable firewall with `netsh advfirewall set currentprofile state off` to allow tests to run without firewall popups (TODO: research better options here as this does not seem ideal)
+* See https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners for basic instructions on setting up the runner
+* Remember to close the Terminal window after installing the above and open a new window to invoke `run.cmd`
+
+## Linux
+
+* Install docker (`sudo apt-get install docker.io`)
+  * Make sure runner user is allowed to use docker (add user to docker group using `sudo usermod -aG docker ${USER}` note that you will need to log out before running the runner script as this change is not applied directly)
+* Install development tools and utilities (`sudo apt install git build-essential curl zip unzip tar pkg-config`)
diff --git a/docs/dev/VersioningCompatibility.md b/docs/dev/VersioningCompatibility.md
new file mode 100644
index 000000000..f4a283653
--- /dev/null
+++ b/docs/dev/VersioningCompatibility.md
@@ -0,0 +1,17 @@
+# Zen Storage Server Versioning
+
+Any API changes made in the Zen Storage Server must be backwards compatible. That is to say, a newer 
+version of the server must be compatible with older versions of client code (typically, the Unreal Engine
+editor or runtime). This means you can use any newer version of the server with some version of the engine.
+For example, using Zen Server 5.5.0 with an engine version of 5.4.3 would be perfectly valid and in some
+cases this may be desirable to get improvements in performance or functionality from newer versions of
+the server without upgrading the engine.
+
+Newer clients may not be backwards compatible however, which means that you will generally not be able to 
+use an older server version with more recent UE versions.
+
+The server is versioned such that you can easily correlate the server version with the engine version. Thus
+in the Unreal Engine v5.4.0 release the server will be tagged with 5.4.0 which indicates that it is 
+compatible with the 5.4.0 (and earlier) runtime. As updates are made, new versions may be released and these
+will have a higher minor version number (5.4.1, 5.4.2 etc) but these will not necessarily line up with point
+releases of the engine since there may be more or fewer updates to the server than to the engine.
diff --git a/docs/dev/WindowsOnLinux.md b/docs/dev/WindowsOnLinux.md
new file mode 100644
index 000000000..540447cb2
--- /dev/null
+++ b/docs/dev/WindowsOnLinux.md
@@ -0,0 +1,109 @@
+# Cross-Compiling Windows Binaries on Linux
+
+This document describes how to build `zenserver.exe` and `zen.exe` for Windows
+from a Linux host using clang-cl, lld-link, and the Windows SDK fetched via
+[xwin](https://github.com/Jake-Shadle/xwin).
+
+## Prerequisites
+
+- **LLVM/Clang 17+** with the following tools (typically in `/usr/lib/llvm-<ver>/bin/`):
+  - `clang` (used as `clang-cl` via `--driver-mode=cl`)
+  - `lld-link`
+  - `llvm-lib`
+  - `llvm-rc`
+  - `llvm-ml`
+  - `llvm-mt`
+
+  On Ubuntu/Debian:
+  ```bash
+  sudo apt install clang-18 lld-18 llvm-18
+  ```
+
+- **xmake** (build system) — see [xmake.io](https://xmake.io) for installation.
+
+## Step 1: Fetch the Windows SDK
+
+Run the provided script to download the Windows SDK and MSVC CRT headers/libs:
+
+```bash
+./scripts/win_cross/get_win_sdk.sh
+```
+
+This downloads the SDK to `~/.xwin-sdk` by default. Override the location with
+an argument or the `$XWIN_SDK_DIR` environment variable:
+
+```bash
+./scripts/win_cross/get_win_sdk.sh /path/to/sdk
+```
+
+The script also creates:
+- Tool wrapper scripts (clang-cl, lld-link, llvm-lib, etc.) in the SDK `bin/` directory
+- An MSVC-compatible directory layout that xmake's built-in clang-cl toolchain can discover
+- Debug CRT lib symlinks so cmake package builds succeed
+
+## Step 2: Configure and Build
+
+```bash
+xmake config -y -p windows -a x64 \
+  --toolchain=clang-cl \
+  --sdk=$HOME/.xwin-sdk \
+  --mrc=$HOME/.xwin-sdk/bin/x64/rc.exe \
+  -m release \
+  --zensentry=n \
+  --zenmimalloc=n
+
+xmake
+```
+
+### Notes
+
+- **`--sdk` must use an absolute path** — `$HOME/.xwin-sdk` works, but
+  `~/.xwin-sdk` does not (xmake doesn't expand `~`).
+- **`--mrc`** points to the resource compiler wrapper. Without it, xmake can't
+  auto-detect `llvm-rc` (it doesn't support `--version`).
+- **`--zensentry=no`** is recommended — crashpad (used by sentry-native) is
+  difficult to cross-compile.
+- **`--zenmimalloc=no`** is recommended initially to reduce the package surface.
+- **LTO is automatically disabled** for cross-compilation builds.
+
+## Step 3: Verify
+
+```bash
+file build/windows/x64/release/zenserver.exe
+# PE32+ executable (console) x86-64, for MS Windows, 12 sections
+
+file build/windows/x64/release/zen.exe
+# PE32+ executable (console) x86-64, for MS Windows, 12 sections
+```
+
+## Running with Wine (optional)
+
+The resulting binaries can be run under Wine for testing:
+
+```bash
+wine build/windows/x64/release/zenserver.exe --help
+```
+
+## Troubleshooting
+
+### Library not found (case sensitivity)
+
+The Windows SDK ships headers and libs with specific casing (e.g. `DbgHelp.h`,
+`ws2_32.lib`). Linux filesystems are case-sensitive, so `#include <Dbghelp.h>`
+won't find `DbgHelp.h`. The codebase uses lowercase includes where possible. If
+you encounter a missing header/lib, check the actual casing in the SDK directory.
+
+### `add_ldflags(...) is ignored`
+
+xmake's auto flag checker may reject MSVC linker flags when using clang-cl. Add
+`{force = true}` to the `add_ldflags()` call.
+
+### Stale build state
+
+If you hit unexpected errors after changing toolchain settings, clean everything:
+
+```bash
+rm -rf .xmake build ~/.xmake
+```
+
+Then reconfigure and rebuild.
diff --git a/docs/dev/xmake.md b/docs/dev/xmake.md
new file mode 100644
index 000000000..a529107b8
--- /dev/null
+++ b/docs/dev/xmake.md
@@ -0,0 +1,54 @@
+# xmake notes and tips and tricks
+
+We use xmake to build code in this tree. We also use xmake to handle some of our third-party dependencies which are
+not in the tree.
+
+This document is intended as a basic guide on how to accomplish some common things when working with the Zen codebase.
+
+The official documentation for xmake is located here: https://xmake.io/ it covers most features but isn't
+necessarily rich in examples on how to accomplish things on different platforms and environments.
+
+# Build basics
+
+xmake is what I'd call a "stateful" build system in that there is a 'configuration' step which you
+will generally want to run before actually building the code. This allows you to specify which
+mode you want to build (in our case, "debug" or "release"), and also allows you to perform additional
+configuration of options such as whether we should include support for Sentry crash reporting, use 
+of 'mimalloc' for memory allocations etc etc
+
+Configure xmake for building 'debug' (which includes building all third-party packages) and 
+triggering a build, issue these commands in the shell:
+
+```
+dev/zen> xmake config --mode=debug
+dev/zen> xmake
+```
+
+If all goes well, you will be able to run the compiled programs using:
+
+```
+dev/zen> xmake run zenserver
+dev/zen> xmake run zen
+```
+
+# Cleaning out *all* build state
+
+You may run into build issues at some point due to bad on-disk state. For instance your workstation
+could crash at an inopportune moment leaving things in an inconsistent state, or you may run into bugs
+in compilers or the build system itself.
+
+When faced with this it's good to be able to wipe out all state which influences the build. Since xmake
+uses a number of different locations to store state it's not entirely obvious at first how to accomplish 
+this.
+
+## Windows
+
+```
+dev\zen> rmdir /s /q .xmake build %LOCALAPPDATA%\.xmake %TEMP%\.xmake
+```
+
+## Linux / MacOS
+
+```
+dev/zen> rm -rf .xmake build ~/.xmake
+```