aboutsummaryrefslogtreecommitdiff
path: root/xmake.lua
Commit message (Collapse)AuthorAgeFilesLines
* Clamp GC remaining time display and improve sanitizer docsStefan Boberg35 hours1-2/+17
| | | | | | | | | | Fix the GC status log showing absurd "Full GC in 492128002h" values caused by clock skew or cross-platform gc_state timestamp mismatches. Clamp remaining time to the configured interval in both the scheduler loop and the status report. Also expand sanitizer comments in xmake.lua with usage instructions for TSAN suppressions and a note about MSAN limitations.
* Move test task implementation from xmake.lua to scripts/test.luaStefan Boberg36 hours1-397/+2
| | | | | | Extract the ~400-line inline test runner into a dedicated script, matching the pattern used by the bundle task. The task definition in xmake.lua now just declares the menu and delegates to scripts.test.
* Made CPR optional, html generated at build time (#840)Stefan Boberg2 days1-11/+10
| | | | | | | - Fix potential crash on startup caused by logging macros being invoked before the logging system is initialized (null logger dereference in `ZenServerState::Sweep()`). `LoggerRef::ShouldLog` now guards against a null logger pointer. - Make CPR an optional dependency (`--zencpr` build option, enabled by default) so builds can proceed without it - Make zenvfs Windows-only (platform-specific target) - Generate the frontend zip at build time from source HTML files instead of checking in a binary blob which would accumulate with every single update
* Add clang-cl build supportStefan Boberg3 days1-0/+24
| | | | | | | | | | - Add clang-cl warning suppressions in xmake.lua matching Linux/macOS set - Guard /experimental:c11atomics with {tools="cl"} for MSVC-only - Fix long long / int64_t redefinition in string.h for clang-cl - Fix unclosed namespace in callstacktrace.cpp #else branch - Fix missing override in httpplugin.cpp - Reorder WorkerPool fields to match designated initializer order - Use INVALID_SOCKET instead of SOCKET_ERROR for SOCKET comparisons
* Switch httpclient default back-end over to libcurl (#832)Stefan Boberg3 days1-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Switches the default HTTP client to the libcurl-based backend and follows up with a series of correctness fixes and code quality improvements to `CurlHttpClient`. **Backend switch & build fixes:** - Switch default HTTP client to libcurl-based backend - Suppress `[[nodiscard]]` warning when building fmt - Miscellaneous bugfixes in HttpClient/libcurl - Pass `-y` to `xmake config` in `xmake test` task **Boilerplate reduction:** - Add `Session::SetHeaders()` for RAII ownership of `curl_slist`, eliminating manual `curl_slist_free_all` calls from every verb method - Add `Session::PerformWithResponseCallbacks()` to absorb the repeated 12-line write+header callback setup block - Extract `ParseHeaderLine()` shared helper, replacing 4 duplicate header-parsing implementations - Extract `BuildHeaderMap()` and `ApplyContentTypeFromHeaders()` helpers to deduplicate header-to-map conversion and Content-Type scanning - Unify the two `DoWithRetry` overloads (PayloadFile variant now delegates to the Validate variant) **Correctness fixes:** - `TransactPackage`: both phases now use `PerformWithResponseCallbacks()`, fixing missing abort support and a dead header collection loop - `TransactPackage`: error path now routes through `CommonResponse`, preserving curl error codes and messages for the caller - `ValidatePayload`: merged 3 separate header-scan loops into a single pass **Performance improvements:** - Replace `fmt::format` with `ExtendableStringBuilder` in `BuildHeaderList` and `BuildUrlWithParameters`, eliminating heap allocations in the common case - Replace `curl_easy_escape`/`curl_free` with inline URL percent-encoding using `AsciiSet` - Remove wasteful `CommonResponse(...)` construction in retry logging, formatting directly from `CurlResult` fields
* Transparent proxy mode (#823)Stefan Boberg4 days1-4/+18
| | | | | | | | | | | | | | | | | Adds a **transparent TCP proxy mode** to zenserver (activated via `zenserver proxy`), allowing it to sit between clients and upstream Zen servers to inspect and monitor HTTP/1.x traffic in real time. Primarily useful during development, to be able to observe multi-server/client interactions in one place. - **Dedicated proxy port** -- Proxy mode defaults to port 8118 with its own data directory to avoid collisions with a normal zenserver instance. - **TCP proxy core** (`src/zenserver/proxy/`) -- A new transparent TCP proxy that forwards connections to upstream targets, with support for both TCP/IP and Unix socket listeners. Multi-threaded I/O for connection handling. Supports Unix domain sockets for both upstream/downstream. - **HTTP traffic inspection** -- Parses HTTP/1.x request/response streams inline to extract method, path, status, content length, and WebSocket upgrades without breaking the proxied data. - **Proxy dashboard** -- A web UI showing live connection stats, per-target request counts, active connections, bytes transferred, and client IP/session ID rollups. - **Server mode display** -- Dashboard banner now shows the running server mode (Zen Proxy, Zen Compute, etc.). Supporting changes included in this branch: - **Wildcard log level matching** -- Log levels can now be set per-category using wildcard patterns (e.g. `proxy.*=debug`). - **`zen down --all`** -- New flag to shut down all running zenserver instances; also used by the new `xmake kill` task. - Minor test stability fixes (flaky hash collisions, per-thread RNG seeds). - Support ZEN_MALLOC environment variable for default allocator selection and switch default to rpmalloc - Fixed sentry-native build to allow LTO on Windows
* oidctoken tool package (#810)Stefan Boberg11 days1-0/+1
| | | | | * added OidcToken binary to the build process. The binary is mirrored from p4 and is placed next to the output of the build process. It is also placed in the release zip archives. * also fixed issue with Linux symbol stripping which was introduced in toolchain changes yesterday
* compute orchestration (#763)Stefan Boberg12 days1-1/+24
| | | | | | | | | | - Added local process runners for Linux/Wine, Mac with some sandboxing support - Horde & Nomad provisioning for development and testing - Client session queues with lifecycle management (active/draining/cancelled), automatic retry with configurable limits, and manual reschedule API - Improved web UI for orchestrator, compute, and hub dashboards with WebSocket push updates - Some security hardening - Improved scalability and `zen exec` command Still experimental - compute support is disabled by default
* native xmake toolchain definition for UE-clang (#805)Stefan Boberg12 days1-10/+32
| | | | | | | | | | | | | | | This change is meant to provide a smoother experience when working on Linux. After this change, the toolchain setup process is now simply ```bash $ scripts/ue_build_linux/get_ue_toolchain.sh ``` and then at config time the toolchain is automatically detected if you downloaded it to the default location or have the `UE_TOOLCHAIN_DIR` environment variable set ```bash xmake config --mode=debug ``` Compared to the old script-based approach this configures the toolchain more precisely, avoiding leakage into unrelated build processes such as when a package manager decides to build something like Ninja locally etc.
* Add test suites (#799)Stefan Boberg2026-03-021-29/+84
| | | | | | | | | | | | | Makes all test cases part of a test suite. Test suites are named after the module and the name of the file containing the implementation of the test. * This allows for better and more predictable filtering of which test cases to run which should also be able to reduce the time CI spends in tests since it can filter on the tests for that particular module. Also improves `xmake test` behaviour: * instead of an explicit list of projects just enumerate the test projects which are available based on build system state * also introduces logic to avoid running `xmake config` unnecessarily which would invalidate the existing build and do lots of unnecessary work since dependencies were invalidated by the updated config * also invokes build only for the chosen test targets As a bonus, also adds `xmake sln --open` which allows opening IDE after generation of solution/xmake project is done.
* added `--verbose` option to zenserver-test and `xmake test` (#798)Stefan Boberg2026-03-011-1/+6
| | | | | | * when `--verbose` is specified to zenserver-test, all child process output (typically, zenserver instances) is piped through to stdout. you can also pass `--verbose` to `xmake test` to accomplish the same thing. * this PR also consolidates all test runner `main` function logic (such as from zencore-test, zenhttp-test etc) into central implementation in zencore for consistency and ease of maintenance * also added extended utf8-tests including a fix to `Utf8ToWide()`
* test running / reporting improvements (#797)Stefan Boberg2026-02-281-50/+112
| | | | | | | | | | | | | | | | | | | **CI/CD improvements (validate.yml):** - Add test reporter (`ue-foundation/test-reporter@v2`) for all three platforms, rendering JUnit test results directly in PR check runs - Add "Trust workspace" step on Windows to fix git safe.directory ownership issue with self-hosted runners - Clean stale report files before each test run to prevent false failures from leftover XML - Broaden `paths-ignore` to skip builds for non-code changes (`*.md`, `LICENSE`, `.gitignore`, `docs/**`) **Test improvements:** - Convert `CHECK` to `REQUIRE` in several test suites (projectstore, integration, http) for fail-fast behavior - Mark some tests with `doctest::skip()` for selective execution - Skip httpclient transport tests pending investigation - Add `--noskip` option to `xmake test` task - Add `--repeat=<N>` option to `xmake test` task, to run tests repeatedly N times or until there is a failure **xmake test output improvements:** - Add totals row to test summary table - Right-justify numeric columns in summary table
* Add test summary table and failure reporting to xmake test (#794)Stefan Boberg2026-02-271-49/+255
| | | | | | | | | | - Add a summary table printed after all test suites complete, showing per-suite test case counts, assertion counts, timings and pass/fail status. - Add failure reporting: individual failing test cases are listed at the end with their file path and line number for easy navigation. - Made zenserver instances spawned by a hub not create new console windows for a better background testing experience - The TestListener in testing.cpp now writes a machine-readable summary file (via `ZEN_TEST_SUMMARY_FILE` env var) containing aggregate counts and per-test-case failure details. This runs as a doctest listener alongside any active reporter, so it works with both console and JUnit modes. - Tests now run in a deterministic order defined by a single ordered list that also serves as the test name/target mapping, replacing the previous unordered table + separate order list. - The `--run` option now accepts comma-separated values (e.g. `--run=core,http,util`) and validates each name, reporting unknown test names early. - Fix platform detection in `xmake test`: the config command now passes `-p` explicitly, fixing "mingw" misdetection when running from Git Bash on Windows. - Add missing "util" entry to the help text for `--run`.
* adding HttpClient tests (#785)Stefan Boberg2026-02-261-2/+2
| | | | | | | | | | | | | | | | | | | | Add comprehensive `HttpClient` test suite. Covers: - **HTTP verbs** -- GET, POST, PUT, DELETE, HEAD dispatch correctly - **GET/POST/PUT/Upload/Download** -- payload round-trips (IoBuffer, CbObject, CompositeBuffer), content types, large payloads, file-spill downloads - **Status codes** -- 2xx/4xx/5xx classification, exact code matching - **Response API** -- IsSuccess, AsText, AsObject, ToText, ErrorMessage, ThrowError - **Error handling** -- connection refused, request timeout, nonexistent endpoints - **Session management** -- default ID, SetSessionId, reset to zero - **Authentication** -- token provider, expired tokens, bearer verification - **Content type detection** -- text, JSON, binary, CbObject - **Request metadata** -- elapsed time, upload/download byte counts - **Retry logic** -- retry after transient 503s, no-retry baseline - **Latency measurement** -- MeasureLatency against live and unreachable servers - **KeyValueMap** -- construction from pairs, string_views, initializer lists - **Transport-level faults (GET)** -- connection reset/close before response, partial headers, truncated body, mid-body reset, stalled response timeout, retry after RST - **Transport-level faults (POST)** -- server reset/close before consuming body, mid-body reset, early 503 without consuming upload, stalled upload timeout, retry with large body after transient failures Also adds zenhttp-test to the xmake test runner (xmake test --run=http).
* structured compute basics (#714)Stefan Boberg2026-02-181-0/+12
| | | | | | | | | this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
* bump sentry to 0.12.1 (#721)Stefan Boberg2026-02-121-4/+4
|
* zen hub (#574)Stefan Boberg2026-01-211-4/+5
| | | | | Initial implementation of zenserver "hub" mode. This is an experimental feature. zenserver can be started in hub mode by specifying `hub` as the first argument to zenserver
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-271-0/+1
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* sentry/asan configuration tweaks (#649)v5.7.10-pre0Stefan Boberg2025-11-131-3/+10
| | | | | * Automated more of the decisions around which options to set when using ASAN * Also disabled Sentry by default as it's a bit annoying to have it upload crashes during development. Sentry is still automatically enabled and integrated as part of the `xmake bundle` step however so released builds will still have it.
* switch back to openssl on Mac (#641)Stefan Boberg2025-11-101-7/+3
| | | | | | | * switch back to openssl for Mac (fixes cross-compilation config) * add openssl 3.6.0 and change make target to install_dev for quicker install * disable LTO on Mac to reduce build time on openssl3 * add mbedTLS 3.6.5 (but this is not used anywhere right now)
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-31/+96
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-011-1/+1
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* move cpr in-tree (#605)Stefan Boberg2025-10-241-11/+9
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* in-tree spdlog (#602)Stefan Boberg2025-10-241-1/+0
| | | | move spdlog into the tree to remove dependency on vcpkg::spdlog, to allow diverging from the official version and evolve it to fit better with OTLP logging requirements
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-221-8/+10
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* ignore vla-cxx-extension warning on MacOS as well as on Linux (#593)Stefan Boberg2025-10-211-1/+7
| | | | | this triggers in more recent versions of the MacOS toolchain this also adds `--Wno-unknown-warning-option` for macosx until we can figure out how to detect toolchain version since the xcode version we use in CI fails on the `-Wno-vla-cxx-extension`
* added separate xmake.lua for thirdparty (#578)Stefan Boberg2025-10-151-0/+1
| | | | | Moves out third-party stuff from zencore Establishes new pattern for incorporating thirdparty code. The integration is cleaner, clearer and also surfaces the code in the generated .sln
* add zenremotestore lib (#540)Dan Engelbrecht2025-10-021-1/+3
|
* rpmalloc fixes (#499)Stefan Boberg2025-09-171-1/+2
| | | | | | | | * fixed rpmalloc build on Linux and Mac * updated rpmalloc from develop branch on the advice of mjansson * enabled rpmalloc on all platforms note that this does not change any behaviour unless `--malloc=rpmalloc` is passed in on the command line. The default is still `mimalloc`.
* Remove broken debug symbols from Oodle and re-enable symbol stripping (#444)Liam Mitchell2025-06-191-5/+0
|
* symbol gen hack-fix for linux (#442)v5.6.13-pre2Dan Engelbrecht2025-06-181-3/+2
|
* symbol gen hack-fix for linuxv5.6.13-pre3Dan Engelbrecht2025-06-181-0/+6
|
* remove explicit set_strip statementDan Engelbrecht2025-06-181-1/+0
|
* change set_strip config till none to generate full symbols on mac/linuxDan Engelbrecht2025-06-181-1/+1
|
* Temporarily disable stripping of symbols while investigating issues with ↵Liam Mitchell2025-06-131-0/+4
| | | | toolchain versions of objcopy
* Use llvm-objcopy provided by UE toolchainLiam Mitchell2025-06-131-0/+1
|
* Update to recent UE toolchain, and link statically against toolchain libc++ ↵Liam Mitchell2025-06-131-0/+10
| | | | and libc++abi
* remove xcode 12.1 workaround (#339)Dan Engelbrecht2025-04-021-6/+0
| | | | | * remove xcode 12.1 workaround * bump min macos version to 14.0
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-061-0/+3
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* enable LTO / optimize for speed (#256)Stefan Boberg2024-12-041-1/+5
| | | | | * changed so release build uses lto and optimizes for speed on Mac and Windows * Linux does not currently support LTO due to toolchain limitations
* don't force openssl to version 3.0.8 (#252)Stefan Boberg2024-12-031-3/+1
| | | should fix build issues caused by curl pulling in a different version
* Insights-compatible memory tracking (#214)Stefan Boberg2024-11-251-0/+19
| | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* Revert "remove temporary workaround involving _LIBCPP_DISABLE_AVAILABILITY ↵Dan Engelbrecht2024-10-101-0/+6
| | | | | (#191)" (#193) This reverts commit e809931618b443809e9740edb70a62d0cab01f87.
* remove temporary workaround involving _LIBCPP_DISABLE_AVAILABILITY (#191)Stefan Boberg2024-10-081-6/+0
| | | | | | * remove temporary workaround involving _LIBCPP_DISABLE_AVAILABILITY * temp disable signing on Mac this change should be revisited once we have resumed regular service wrt MacOS runners
* Separate UTF-8 flags by platform (#178)Dmytro Ivanov2024-10-011-2/+5
| | | | | | | Fixes the following warnings ``` 1>cl : Command line warning D9002: ignoring unknown option '-source-charset=utf-8' 1>cl : Command line warning D9002: ignoring unknown option '-execution-charset=utf-8' ```
* Fixing compilation errors with fmt v11 (#172)Dmytro Ivanov2024-09-271-0/+4
|
* cleaned up top level xmake (#125)Stefan Boberg2024-08-191-14/+12
|
* enable mimalloc on arm64 (#120)Stefan Boberg2024-08-161-10/+8
| | | * enable mimalloc on arm64
* enable sentry on arm64 (#119)Stefan Boberg2024-08-151-2/+3
| | | * enable sentry on arm64
* bump vcpkg and xmake to latest (#40)Dan Engelbrecht2024-04-241-1/+1
| | | - Improvement: Bumped xmake to 2.9.1 and vcpkg version to 2024.03.25
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-16 08:45:09 +0000