diff options
Diffstat (limited to 'CLAUDE.md')
| -rw-r--r-- | CLAUDE.md | 314 |
1 files changed, 314 insertions, 0 deletions
diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..32c4b3d40 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,314 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +Unreal Zen Storage Service (zenserver) is a high-performance local storage service for UE5, managing secondary data such as cooker output and local caches (DDC - Derived Data Cache). Can be deployed as a daemon on user machines or as a shared cache instance for build farms. + +## Build System + +This project uses **xmake** as its build system. xmake is stateful - run configuration before building. + +### Common Build Commands + +```bash +# Configure for debug (includes tests) +xmake config -y -m debug -a x64 + +# Configure for release +xmake config -y -m release -a x64 + +# Build everything +xmake -y + +# Build specific target +xmake build -y zenserver +xmake build -y zen + +# Run targets +xmake run zenserver +xmake run zen + +# Generate IDE project files +xmake sln # Windows: Visual Studio 2022 +xmake project -k xcode # macOS: Xcode +``` + +### Build Configuration Options + +- `--zensentry=yes|no` - Enable/disable Sentry crash reporting (default: yes) +- `--zenmimalloc=yes|no` - Use mimalloc for memory management (default: yes, disabled with ASAN) +- `--zenrpmalloc=yes|no` - Use rpmalloc for memory management (default: yes) +- `--httpsys=yes|no` - Enable http.sys server (Windows only, default: yes) +- `--zencompute=yes|no` - Enable compute services endpoint (default: yes) +- `--zenmemtrack=yes|no` - Enable UE Memory Trace support (Windows only, default: yes) +- `--zentrace=yes|no` - Enable UE Trace support (default: yes) + +### Testing + +```bash +# Tests are only built in debug mode +xmake config -m debug +xmake + +# Run all tests +xmake test --run=all + +# Run specific test suites +xmake test --run=core # zencore-test +xmake test --run=store # zenstore-test +xmake test --run=server # zenserver test command +xmake test --run=integration # zenserver-test +xmake test --run=http # zenhttp-test +xmake test --run=util # zenutil-test +xmake test --run=remotestore # zenremotestore-test + +# Run tests with JUnit reporting +xmake test --run=all --junit +``` + +Tests use the **doctest** framework. To run server tests: `xmake run zenserver test` + +### Test Suite Naming Convention + +All `TEST_CASE` blocks must be wrapped in a `TEST_SUITE_BEGIN`/`TEST_SUITE_END` pair (not the `TEST_SUITE()` wrapper macro, to avoid reformatting). Suite names follow the pattern `{module}.{filename}` where the `zen` prefix is stripped from the module name: + +| Module | File | Suite name | +|---|---|---| +| `zencore` | `filesystem.cpp` | `"core.filesystem"` | +| `zenstore` | `blockstore.cpp` | `"store.blockstore"` | +| `zenhttp` | `httpclient.cpp` | `"http.httpclient"` | +| `zenutil` | `wildcard.cpp` | `"util.wildcard"` | +| `zennet` | `statsdclient.cpp` | `"net.statsdclient"` | +| `zentelemetry` | `stats.cpp` | `"telemetry.stats"` | +| `zenremotestore` | `chunkblock.cpp` | `"remotestore.chunkblock"` | +| `zenserver-test` | `cache-tests.cpp` | `"server.cache"` | + +Each test executable defaults to running only the suites for its own module (e.g. `zencore-test` defaults to `core.*`). This default is derived automatically from the executable name by `RunTestMain` and can be overridden on the command line: + +```bash +# Run only filesystem tests within zencore-test +zencore-test --test-suite=core.filesystem + +# Run all core tests (default behaviour, no flag needed) +zencore-test +``` + +### Code Formatting + +```bash +# Format all files (requires pre-commit) +xmake precommit + +# Or use pre-commit directly +pre-commit run --all-files +``` + +Install pre-commit hooks with `pre-commit install` to auto-format on commit. + +### Cleaning Build State + +When encountering build issues, clean all build state: + +**Windows:** +```cmd +rmdir /s /q .xmake build %LOCALAPPDATA%\.xmake %TEMP%\.xmake +``` + +**Linux/macOS:** +```bash +rm -rf .xmake build ~/.xmake +``` + +## Architecture + +### Module Structure + +The codebase is organized into layered modules with clear dependencies: + +**Core Libraries (Foundation Layer):** +- `zenbase` - Platform abstraction primitives +- `zencore` - Core utilities, memory management, logging, filesystem, crypto, threading +- `zenutil` - Higher-level utilities and common patterns + +**Protocol & Network Layer:** +- `zenhttp` - HTTP server implementations (http.sys on Windows, ASIO-based on Linux/Mac) +- `zennet` - Network utilities and protocols +- `transports` - Transport SDK and implementations + +**Storage Layer:** +- `zenstore` - Core storage engine and key-value store +- `zenvfs` - Virtual filesystem implementation +- `zenremotestore` - Remote storage client + +**Service Layer:** +- `zenserver` - Main server binary integrating all services +- `zencompute` - Compute services endpoint +- `zentelemetry` - Telemetry and metrics collection + +**Client Tools:** +- `zen` - CLI client utility for interacting with zenserver + - Commands are in `src/zen/cmds/` (e.g., `cache_cmd.h`, `status_cmd.h`, `ui_cmd.h`) + - Each command inherits from base command classes and registers with the CLI dispatcher + +**Test Projects:** +- `*-test` modules contain tests for their corresponding module (only built in debug mode) +- `zenserver-test` contains integration tests + +### Key Architectural Patterns + +**HTTP Server Implementations:** +- Windows: Uses http.sys kernel driver for highest throughput (requires elevation or URL reservation) +- Linux/macOS: ASIO-based HTTP server (also available on Windows) +- Server selection is compile-time based on platform but can be overridden on the command line via `--http=asio/httpsys/etc` + +**Storage Architecture:** +- zenserver manages multiple storage services (build store, cache, project store, object store) +- Content-addressable storage with deduplication on the lowest storage layer but this is mostly an internal implementation detail. + - Clients generally do not request content via content address. Instead they interface with the higher level storage services + - CAS content is garbage collected based on what is referenced from of higher level services + +**Frontend:** +- Web UI bundled as ZIP in `src/zenserver/frontend/*.zip` +- Dashboards for hub, orchestrator, and compute services are located in `src/zenserver/frontent/html/` + - These are the files which end up being bundled into the front-end zip mentioned above +- Update with `xmake updatefrontend` after modifying HTML/JS, and check in the resulting zip + +**Memory Management:** +- Can use mimalloc or rpmalloc for performance +- UE-style LLM (Low-Level Memory) tracking available on Windows +- Memory tracing support integrated with UE trace system +- Avoid using `std::string` when building (concatenating etc) temporary strings. Use `ExtendableStringBuilder<N>` + instead with an appropriate size N to avoid heap allocations in the common case. + +**Tracing:** +- UE Trace integration for profiling and diagnostics +- Trace files use .utrace format +- Trace analysis tools in zentrace module (if available) +- Traces can be visualized using Unreal Insights, available in the UE5 code base + +## Coding Standards + +**Naming Conventions (UE-inspired with modifications):** +- Classes/Structs: `PascalCase` +- Functions/Methods: `PascalCase()` +- Function parameters/local variables: `PascalCase` +- Member variables: `m_PascalCase` +- Global variables: `g_PascalCase` +- Static variables: `s_PascalCase` +- Thread-local variables: `t_PascalCase` + +**Note:** Unlike UE, no `F` prefix on structs/classes is required nor encouraged. Also, no `b` prefix on booleans. + +**Code Style:** +- C++20 standard +- clang-format enforced via pre-commit hooks +- Use `std` containers; `eastl::fixed_vector` etc. for performance-critical paths where the number of elements + in the container is typically small or when the container is temporary and suitable for stack allocation. +- For shared ownership, prefer intrusive reference counting via `TRefCounted<T>` (or `RefCounted`) with `Ref<T>` + over `std::shared_ptr`. Since the ownership policy is known at type definition time, the overhead of + `std::shared_ptr` (separate control block, etc.) is unnecessary. +- Exceptions used only for unexpected errors, not flow control + - Some `std` exception types like `runtime_error` are also available in the `zen` namespace and offer + convenience of formatting the exception message by allowing fmt-style formatting without using + `fmt::format` which can reduce clutter. To use these, please `#include <zencore/except_fmt.h>` +- Logging is done via `ZEN_DEBUG(...)`, `ZEN_INFO(...)`, `ZEN_WARN(...)`, `ZEN_ERROR(...)` macros + - Prefer these over the older `ZEN_LOG_INFO`, `ZEN_LOG_WARN`, etc. variants which require an explicit log category parameter + - Logging macros use `fmt::vformat` internally for message formatting. The first argument is the format string, + which must be a string literal (turned into `std::string_view` in the macros) + - Logging channels can be overridden on a scope-by-scope basis (per-class or even per-function) by implementing + a `LoggerRef Log()` function +- Avoid repetition. If the same pattern (e.g. data structure lookups, locking + find) appears multiple times, + factor it into a helper function and call that instead. +- Avoid `auto` when the type is not explicitly visible on the right-hand side of the assignment. Spell out the type instead. +- `if`/`else` should use braces even for single-statement branches + +**Includes:** +- Wrap third-party includes with `ZEN_THIRD_PARTY_INCLUDES_START` / `ZEN_THIRD_PARTY_INCLUDES_END` + - This helps avoid spurious compiler warnings due to the use of strict warnings-as-error policy + +**Platform Macros:** +- `ZEN_PLATFORM_WINDOWS`, `ZEN_PLATFORM_LINUX`, `ZEN_PLATFORM_MAC` +- Use `is_plat("windows")` etc. in xmake.lua files + +## Security Considerations + +**XSS in Web UI:** +Sanitize all dynamic data before inserting into innerHTML. Use `escapeHtml()` helper functions in JavaScript. + +**http.sys URL Reservation:** +On Windows, zenserver requires elevation or a URL reservation to bind http.sys to network interfaces: +```cmd +netsh http add urlacl url=http://*:8558/ user=<username> +``` + +**POST endpoint content types** +Endpoints accepting POST operations should only accept JSON or compact binary format payloads. + +## Platform-Specific Notes + +**Windows:** +- Primary development platform +- Visual Studio 2022+ required (C++20 features) +- Static CRT linking by default (`/MT` in release, `/MTd` in debug) +- Recommended profilers: Superluminal Performance, Visual Studio profiler + +**Linux:** +- Requires GCC-11+ or Clang-12+ with libstdc++-11+ +- Set `CXX=g++-11` before running xmake if needed +- UE cross-compile toolchain support via `UE_TOOLCHAIN_DIR` environment variable + +**macOS:** +- Requires Xcode or Xcode command-line tools +- Supports both x86_64 and arm64 architectures + +## Dependencies + +Key third-party libraries (managed by xmake): +- EASTL - High-performance containers +- ASIO - Async I/O +- spdlog/fmt - Logging and formatting +- doctest - Testing framework +- cxxopts - Command-line parsing +- json11 - JSON parsing +- lz4 - Compression +- xxhash - Hashing +- OpenSSL (Linux/Mac) - Crypto operations +- Sentry - Crash reporting (optional) + +## Adding New Commands to `zen` CLI + +1. Create `src/zen/cmds/mycommand_cmd.h` and `src/zen/cmds/mycommand_cmd.cpp` +2. Inherit from appropriate base command class +3. Implement `Execute()` method +4. Add `#include "cmds/mycommand_cmd.h"` to `src/zen/zen.cpp` +5. Register command in `zen.cpp` command dispatcher. These should be ordered in alphabetical identifier order for consistent merging + +## Adding New Modules + +1. Create `src/mymodule/` directory structure +2. Add `include/` subdirectory for public headers +3. Create `xmake.lua` defining the target +4. Add `includes("src/mymodule")` to root `xmake.lua` +5. Add dependencies via `add_deps()` in the module's xmake.lua + +## Debugging + +**Visual Studio:** +- Use Debug configuration (includes test code) +- For http.sys debugging, run Visual Studio as Administrator or add URL reservation +- Enable child process debugging with Microsoft Child Process Debugging Power Tool + +**Multi-process Scenarios:** +When debugging zenserver-test or other multi-process scenarios, use child process debugging tools to attach to spawned processes automatically. + +## Bundle Creation + +```bash +# Create deployable ZIP bundle +xmake bundle + +# Update frontend ZIP after HTML changes +xmake updatefrontend +``` |