aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/zenserver.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Fix two potential SIGSEGV paths during early startupStefan Boberg5 days1-0/+6
| | | | | | | | | | | | | - sentryintegration.cpp: sentry_options_new() can return nullptr on OOM; all subsequent sentry_options_set_* calls would immediately SIGSEGV. Guard with an early return that sets m_SentryErrorCode = -1 so Close() correctly skips sentry_close(). - zenserver.cpp: ZenServerState::Register() returns nullptr if the shared memory entry table is full. The subsequent Entry->SignalHasInstanceInfo(), Entry->SignalNoNetwork(), Entry->AddSponsorProcess(), and DoRun(Entry) calls would all SIGSEGV. Throw a descriptive runtime_error instead, which is caught by the existing catch handlers in Run().
* Fix silent exit when second zenserver instance detects port conflictStefan Boberg5 days1-33/+39
| | | | | | | | | ZEN_WARN/INFO/CRITICAL calls before InitializeLogging() are silently discarded. Switch all log calls in ZenServerMain::Run() that precede InitializeLogging() — the ZenServerState early-exit block, the lock file retry block, and the catch handlers — to their ZEN_CONSOLE_* equivalents. Also fix the same issue in ZenServerState::Sweep() and Snapshot() which had been inconsistently converted to fprintf+std::format.
* HttpClient using libcurl, Unix Sockets for HTTP. HTTPS support (#770)Stefan Boberg12 days1-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main goal of this change is to eliminate the cpr back-end altogether and replace it with the curl implementation. I would expect to drop cpr as soon as we feel happy with the libcurl back-end. That would leave us with a direct dependency on libcurl only, and cpr can be eliminated as a dependency. ### HttpClient Backend Overhaul - Implemented a new **libcurl-based HttpClient** backend (`httpclientcurl.cpp`, ~2000 lines) as an alternative to the cpr-based one - Made HttpClient backend **configurable at runtime** via constructor arguments and `-httpclient=...` CLI option (for zen, zenserver, and tests) - Extended HttpClient test suite to cover multipart/content-range scenarios ### Unix Domain Socket Support - Added Unix domain socket support to **httpasio** (server side) - Added Unix domain socket support to **HttpClient** - Added Unix domain socket support to **HttpWsClient** (WebSocket client) - Templatized `HttpServerConnectionT<SocketType>` and `WsAsioConnectionT<SocketType>` to handle TCP, Unix, and SSL sockets uniformly via `if constexpr` dispatch ### HTTPS Support - Added **preliminary HTTPS support to httpasio** (for Mac/Linux via OpenSSL) - Added **basic HTTPS support for http.sys** (Windows) - Implemented HTTPS test for httpasio - Split `InitializeServer` into smaller sub-functions for http.sys ### Other Notable Changes - Improved **zenhttp-test stability** with dynamic port allocation - Enhanced port retry logic in http.sys (handles ERROR_ACCESS_DENIED) - Fatal signal/exception handlers for backtrace generation in tests - Added `zen bench http` subcommand to exercise network + HTTP client/server communication stack
* Merge branch 'main' into lm/oidctoken-exe-pathLiam Mitchell12 days1-20/+147
|\
| * Dashboard overhaul, compute integration (#814)Stefan Boberg13 days1-14/+89
| | | | | | | | | | | | | | | | | | | | - **Frontend dashboard overhaul**: Unified compute/main dashboards into a single shared UI. Added new pages for cache, projects, metrics, sessions, info (build/runtime config, system stats). Added live-update via WebSockets with pause control, sortable detail tables, themed styling. Refactored compute/hub/orchestrator pages into modular JS. - **HTTP server fixes and stats**: Fixed http.sys local-only fallback when default port is in use, implemented root endpoint redirect for http.sys, fixed Linux/Mac port reuse. Added /stats endpoint exposing HTTP server metrics (bytes transferred, request rates). Added WebSocket stats tracking. - **OTEL/diagnostics hardening**: Improved OTLP HTTP exporter with better error handling and resilience. Extended diagnostics services configuration. - **Session management**: Added new sessions service with HTTP endpoints for registering, updating, querying, and removing sessions. Includes session log file support. This is still WIP. - **CLI subcommand support**: Added support for commands with subcommands in the zen CLI tool, with improved command dispatch. - **Misc**: Exposed CPU usage/hostname to frontend, fixed JS compact binary float32/float64 decoding, limited projects displayed on front page to 25 sorted by last access, added vscode:// link support. Also contains some fixes from TSAN analysis.
| * GC - fix handling of attachment ranges, http access token expiration, lock ↵Stefan Boberg2026-02-201-0/+2
| | | | | | | | | | | | | | | | file retry logic (#766) * GC - fix handling of attachment ranges * fix trace/log strings * fix HTTP access token expiration time logic * added missing lock retry in zenserver startup
| * structured compute basics (#714)Stefan Boberg2026-02-181-0/+8
| | | | | | | | | | | | | | | | | | this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
| * add http server root password protection (#757)Dan Engelbrecht2026-02-171-4/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `--security-config-path` option to zenserver to configure security settings - Expects a path to a .json file - Default is an empty path resulting in no extra security settings and legacy behavior - Current support is a top level filter of incoming http requests restricted to the `password` type - `password` type will check the `Authorization` header and match it to the selected authorization strategy - Currently the security settings is very basic and configured to a fixed username+password at startup { "http" { "root": { "filter": { "type": "password", "config": { "password": { "username": "<username>", "password": "<password>" }, "protect-machine-local-requests": false, "unprotected-uris": [ "/health/", "/health/info", "/health/version" ] } } } } }
| * logging config move to zenutil (#754)Stefan Boberg2026-02-131-6/+6
| | | | | | made logging config options from zenserver available in zen CLI
* | Merge branch 'main' into lm/oidctoken-exe-pathLiam Mitchell12 days1-10/+76
|\|
| * restore missing healthinfo call (#737)Dan Engelbrecht2026-01-281-0/+5
| |
| * zenserver API changes, some other minor changes (#720)Stefan Boberg2026-01-191-9/+68
| | | | | | | | | | | | | | * add system metrics output to top command * removed unnecessary xmake directives * file system API/comment tweaks * fixed out-of-range access in httpserver test * updated ZenServer base API to allow customization by mode
| * if we fail to create the server mutex, gracefully report error without ↵Dan Engelbrecht2026-01-121-1/+3
| | | | | | | | sending error to Sentry (#705)
* | Pass command-line OidcToken option through config rather than env variables, ↵Liam Mitchell2026-01-151-5/+0
| | | | | | | | and add lua option
* | Use well-known OidcToken paths or command line arguments to determine ↵Liam Mitchell2026-01-141-0/+6
|/ | | | OidcToken executable path
* add otel instrumentation (#581)Stefan Boberg2025-12-111-0/+4
| | | | | | | | this change adds OTEL tracing to a few places * Top-level application lifecycle (config/init/cleanup, main loop) * http.sys requests it also brings some otlptrace optimizations and dynamic configuration of tracing. OTLP tracing is currently always disabled
* logging improvements (#664)Stefan Boberg2025-11-241-1/+1
| | | | | * Eliminate spurious `FlushFileBuffers` calls on Windows by tracking writes * Make cache log file creation optional
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-0/+23
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-011-2/+2
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* restructured zenserver configuration (#575)Stefan Boberg2025-10-151-2/+2
| | | this breaks out the configuration logic to allow multiple applications to share common configuration and initialization logic whilst customizing chosen aspects of the process
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-141-8/+2
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* extract storage server into separate source files (#569)Stefan Boberg2025-10-131-753/+136
| | | | this change moves common base service code into `zenserver.(cpp|h)` and storage server code into `zenstorageserver.(cpp|h)` to more clearly separate concerns but also to make way for another service
* move service common code into base class (#567)Stefan Boberg2025-10-131-298/+344
| | | | | Separates storage server code and generic server code into two classes. This is a change to prepare for different services to be implemented using the same framework, into the same executable
* hide http.sys options when unavailable (#568)Stefan Boberg2025-10-131-1/+1
|
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-0/+2
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* shorten thread pool names for Linux which has a limit of 15 characters (#563)Stefan Boberg2025-10-081-1/+1
| | | without this change we just get a whole bunch of threads named "main" or "zenserver" depending on timing
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-031-3/+7
| | | * move zen vfs implementation to zenstore
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-1/+1
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-031-5/+0
| | | * move remoteproject code to remotestorelib
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-021-1/+0
|
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-021-7/+8
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* initialize all services before healthy (#493)Dan Engelbrecht2025-09-121-35/+58
| | | * make sure all services are initialized and registered before we respond OK to health check
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-261-3/+16
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* revert multi-cid store (#475)Dan Engelbrecht2025-08-261-22/+16
|
* clean up trace options parsing (#473)Dan Engelbrecht2025-08-221-2/+13
| | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-201-57/+31
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-2/+2
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* add limitoverwrites option per bucket (#466)Dan Engelbrecht2025-08-121-2/+3
| | | | | | | | | | | | | | | - Feature: Added global zenserver option `--cache-bucket-limit-overwrites` controlling Whether to require policy flag pattern before allowing overwrites or not. Default `false` = overwrites always allowed - Feature: Add per bucket cache configuration option `limitoverwrites` (Lua options file only) cache = { bucket = { -- This is the default for all namespaces limitoverwrites = true }, buckets = { -- Here you can add matching per bucket name (matches accross namespaces) iostorecompression = { limitoverwrites = false }, }, }
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-081-4/+23
|\
| * refactor blobstore (#458)Dan Engelbrecht2025-08-061-4/+23
| | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-241-16/+53
|\|
| * sentry config (#430)Dan Engelbrecht2025-06-121-1/+1
| | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
| * revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-1/+1
| |
| * add sentry configurations options for debug/environmentDan Engelbrecht2025-06-101-1/+1
| | | | | | | | | | add env-variable parsing for sentry option
| * enable per bucket config (#388)Dan Engelbrecht2025-05-121-4/+20
| | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| * add sentry for zen command (#373)Dan Engelbrecht2025-05-061-2/+1
| | | | | | | | * refactor sentry integration and add to zen command line tool * move add_ldflags("-framework Security")
| * blobstore size limit (#342)Dan Engelbrecht2025-04-041-3/+4
| | | | | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
| * builds url discovery (#334)Dan Engelbrecht2025-04-011-6/+7
| | | | | | | | | | - Feature: Added `--host` option to use Jupiters list of cloud host and zen servers to resolve best hosts - Feature: Use local zenserver as builds cache if it has the `builds` service enabled and `--cloud-discovery-host` is provided and no remote zenserver cache hosts can be found - Improvement: Added `--override-host` option as a replacement for `--url` (`--url` still works, but `--override-host` is preferred)
| * long filename support (#330)Dan Engelbrecht2025-03-311-3/+3
| | | | | | - Bugfix: Long file paths now works correctly on Windows
| * zen build cache service (#318)Dan Engelbrecht2025-03-261-0/+19
| | | | | | | | | | | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-22 07:26:35 +0000