aboutsummaryrefslogtreecommitdiff
path: root/src/zen/zen.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Unix Domain Socket auto discovery (#833)Stefan Boberg2026-03-131-1/+45
| | | | | | | | This PR adds end-to-end Unix domain socket (UDS) support, allowing zen CLI to discover and connect to UDS-only servers automatically. - **`unix://` URI scheme in zen CLI**: The `-u` / `--hosturl` option now accepts `unix:///path/to/socket` to connect to a zenserver via a Unix domain socket instead of TCP. - **Per-instance shared memory for extended server info**: Each zenserver instance now publishes a small shared memory section (keyed by SessionId) containing per-instance data that doesn't fit in the fixed-size ZenServerEntry -- starting with the UDS socket path. This is a 4KB pagefile-backed section on Windows (`Global\ZenInstance_{sessionid}`) and a POSIX shared memory object on Linux/Mac (`/UnrealEngineZen_{sessionid}`). - **Client-side auto-discovery of UDS servers**: `zen info`, `zen status`, etc. now automatically discover and prefer UDS connections when a server publishes a socket path. Servers running with `--no-network` (UDS-only) are no longer invisible to the CLI. - **`kNoNetwork` flag in ZenServerEntry**: Servers started with `--no-network` advertise this in their shared state entry. Clients skip TCP fallback for these servers, and display commands (`ps`, `status`, `top`) show `-` instead of a port number to indicate TCP is not available.
* Switch httpclient default back-end over to libcurl (#832)Stefan Boberg2026-03-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Switches the default HTTP client to the libcurl-based backend and follows up with a series of correctness fixes and code quality improvements to `CurlHttpClient`. **Backend switch & build fixes:** - Switch default HTTP client to libcurl-based backend - Suppress `[[nodiscard]]` warning when building fmt - Miscellaneous bugfixes in HttpClient/libcurl - Pass `-y` to `xmake config` in `xmake test` task **Boilerplate reduction:** - Add `Session::SetHeaders()` for RAII ownership of `curl_slist`, eliminating manual `curl_slist_free_all` calls from every verb method - Add `Session::PerformWithResponseCallbacks()` to absorb the repeated 12-line write+header callback setup block - Extract `ParseHeaderLine()` shared helper, replacing 4 duplicate header-parsing implementations - Extract `BuildHeaderMap()` and `ApplyContentTypeFromHeaders()` helpers to deduplicate header-to-map conversion and Content-Type scanning - Unify the two `DoWithRetry` overloads (PayloadFile variant now delegates to the Validate variant) **Correctness fixes:** - `TransactPackage`: both phases now use `PerformWithResponseCallbacks()`, fixing missing abort support and a dead header collection loop - `TransactPackage`: error path now routes through `CommonResponse`, preserving curl error codes and messages for the caller - `ValidatePayload`: merged 3 separate header-scan loops into a single pass **Performance improvements:** - Replace `fmt::format` with `ExtendableStringBuilder` in `BuildHeaderList` and `BuildUrlWithParameters`, eliminating heap allocations in the common case - Replace `curl_easy_escape`/`curl_free` with inline URL percent-encoding using `AsciiSet` - Remove wasteful `CommonResponse(...)` construction in retry logging, formatting directly from `CurlResult` fields
* minor zenstore/blockstore fixes (#821)Stefan Boberg2026-03-101-1/+1
| | | | | | | | | | | - Fix clang-format error accidentally introduced by recent PR - Fix `FileSize()` CAS race that repeatedly invalidated the cache when concurrent callers both missed; remove `store(0)` on CAS failure - Fix `WriteChunks` not accounting for initial alignment padding in `m_TotalSize`, causing drift vs `WriteChunk`'s correct accounting - Fix Create retry sleep computing negative values (100 - N*100 instead of 100 + N*100), matching the Open retry pattern - Fix `~BlockStore` error log missing format placeholder for `Ex.what()` - Fix `GetFreeBlockIndex` infinite loop when all indexes have orphan files on disk but aren't in `m_ChunkBlocks`; bound probe to `m_MaxBlockCount` - Fix `IterateBlock` ignoring `SmallSizeCallback` return value for single out-of-bounds chunks, preventing early termination - Fix `BlockStoreCompactState::IterateBlocks` iterating map by value instead of const reference
* HttpClient using libcurl, Unix Sockets for HTTP. HTTPS support (#770)Stefan Boberg2026-03-101-4/+71
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main goal of this change is to eliminate the cpr back-end altogether and replace it with the curl implementation. I would expect to drop cpr as soon as we feel happy with the libcurl back-end. That would leave us with a direct dependency on libcurl only, and cpr can be eliminated as a dependency. ### HttpClient Backend Overhaul - Implemented a new **libcurl-based HttpClient** backend (`httpclientcurl.cpp`, ~2000 lines) as an alternative to the cpr-based one - Made HttpClient backend **configurable at runtime** via constructor arguments and `-httpclient=...` CLI option (for zen, zenserver, and tests) - Extended HttpClient test suite to cover multipart/content-range scenarios ### Unix Domain Socket Support - Added Unix domain socket support to **httpasio** (server side) - Added Unix domain socket support to **HttpClient** - Added Unix domain socket support to **HttpWsClient** (WebSocket client) - Templatized `HttpServerConnectionT<SocketType>` and `WsAsioConnectionT<SocketType>` to handle TCP, Unix, and SSL sockets uniformly via `if constexpr` dispatch ### HTTPS Support - Added **preliminary HTTPS support to httpasio** (for Mac/Linux via OpenSSL) - Added **basic HTTPS support for http.sys** (Windows) - Implemented HTTPS test for httpasio - Split `InitializeServer` into smaller sub-functions for http.sys ### Other Notable Changes - Improved **zenhttp-test stability** with dynamic port allocation - Enhanced port retry logic in http.sys (handles ERROR_ACCESS_DENIED) - Fatal signal/exception handlers for backtrace generation in tests - Added `zen bench http` subcommand to exercise network + HTTP client/server communication stack
* Dashboard overhaul, compute integration (#814)Stefan Boberg2026-03-091-0/+78
| | | | | | | | | | - **Frontend dashboard overhaul**: Unified compute/main dashboards into a single shared UI. Added new pages for cache, projects, metrics, sessions, info (build/runtime config, system stats). Added live-update via WebSockets with pause control, sortable detail tables, themed styling. Refactored compute/hub/orchestrator pages into modular JS. - **HTTP server fixes and stats**: Fixed http.sys local-only fallback when default port is in use, implemented root endpoint redirect for http.sys, fixed Linux/Mac port reuse. Added /stats endpoint exposing HTTP server metrics (bytes transferred, request rates). Added WebSocket stats tracking. - **OTEL/diagnostics hardening**: Improved OTLP HTTP exporter with better error handling and resilience. Extended diagnostics services configuration. - **Session management**: Added new sessions service with HTTP endpoints for registering, updating, querying, and removing sessions. Includes session log file support. This is still WIP. - **CLI subcommand support**: Added support for commands with subcommands in the zen CLI tool, with improved command dispatch. - **Misc**: Exposed CPU usage/hostname to frontend, fixed JS compact binary float32/float64 decoding, limited projects displayed on front page to 25 sorted by last access, added vscode:// link support. Also contains some fixes from TSAN analysis.
* Eliminate spdlog dependency (#773)Stefan Boberg2026-03-091-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes the vendored spdlog library (~12,000 lines) and replaces it with a purpose-built logging system in zencore (~1,800 lines). The new implementation provides the same functionality with fewer abstractions, no shared_ptr overhead, and full control over the logging pipeline. ### What changed **New logging core in zencore/logging/:** - LogMessage, Formatter, Sink, Logger, Registry - core abstractions matching spdlog's model but simplified - AnsiColorStdoutSink - ANSI color console output (replaces spdlog stdout_color_sink) - MsvcSink - OutputDebugString on Windows (replaces spdlog msvc_sink) - AsyncSink - async logging via BlockingQueue worker thread (replaces spdlog async_logger) - NullSink, MessageOnlyFormatter - utility types - Thread-safe timestamp caching in formatters using RwLock **Moved to zenutil/logging/:** - FullFormatter - full log formatting with timestamp, logger name, level, source location, multiline alignment - JsonFormatter - structured JSON log output - RotatingFileSink - rotating file sink with atomic size tracking **API changes:** - Log levels are now an enum (LogLevel) instead of int, eliminating the zen::logging::level namespace - LoggerRef no longer wraps shared_ptr - it holds a raw pointer with the registry owning lifetime - Logger error handler is wired through Registry and propagated to all loggers on registration - Logger::Log() now populates ThreadId on every message **Cleanup:** - Deleted thirdparty/spdlog/ entirely (110+ files) - Deleted full_test_formatter (was ~80% duplicate of FullFormatter) - Renamed snake_case classes to PascalCase (full_formatter -> FullFormatter, json_formatter -> JsonFormatter, sentry_sink -> SentrySink) - Removed spdlog from xmake dependency graph ### Build / test impact - zencore no longer depends on spdlog - zenutil and zenvfs xmake.lua updated to drop spdlog dep - zentelemetry xmake.lua updated to drop spdlog dep - All existing tests pass, no test changes required beyond formatter class renames
* added `--verbose` option to zenserver-test and `xmake test` (#798)Stefan Boberg2026-03-011-1/+0
| | | | | | * when `--verbose` is specified to zenserver-test, all child process output (typically, zenserver instances) is piped through to stdout. you can also pass `--verbose` to `xmake test` to accomplish the same thing. * this PR also consolidates all test runner `main` function logic (such as from zencore-test, zenhttp-test etc) into central implementation in zencore for consistency and ease of maintenance * also added extended utf8-tests including a fix to `Utf8ToWide()`
* Add `zen ui` command (#779)Stefan Boberg2026-02-241-1/+5
| | | | | Allows user to automate launching of zenserver dashboard, including when multiple instances are running. If multiple instances are running you can open all dashboards with `--all`, and also using the in-terminal chooser which also allows you to open a specific instance. Also includes a fix to `zen exec` when using offset/stride/limit
* changed command names and descriptions to use class members instead of ↵Stefan Boberg2026-02-231-43/+43
| | | | string literals in zen.cpp (#776)
* structured compute basics (#714)Stefan Boberg2026-02-181-16/+23
| | | | | | | | | this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
* logging config move to zenutil (#754)Stefan Boberg2026-02-131-6/+17
| | | made logging config options from zenserver available in zen CLI
* hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-221-20/+1
| | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
* use current locale when converting wide char command line arguments to ↵Dan Engelbrecht2026-01-151-1/+5
| | | | | | string (#712) * set utf8 locale, only set LC_CTYPE as we don't want to affect how decimal point or sorting orders are handled * set language/region explicitly so we can use LC_ALL
* remove error warning in output (#695)Dan Engelbrecht2025-12-171-4/+6
| | | * changed some logging string so they don't get caught in CI logging
* move progress bar to separate file (#638)Dan Engelbrecht2025-11-071-358/+2
| | | * move progress bar to separate file
* in-tree spdlog (#602)Stefan Boberg2025-10-241-2/+0
| | | | move spdlog into the tree to remove dependency on vcpkg::spdlog, to allow diverging from the official version and evolve it to fit better with OTLP logging requirements
* tweak worker pools for builds (#595)Dan Engelbrecht2025-10-221-0/+5
| | | * update logic for number of network and disk threads for `zen builds` commands
* fix log progress quotes (#580)Dan Engelbrecht2025-10-161-9/+37
| | | | * add quotes to message in log-progress mode * improved log progress for builds upload and download
* executable/version/pid log at command startup cleaned up (#557)Dan Engelbrecht2025-10-061-0/+7
|
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-60/+30
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* fetch cloud oplog (#502)Dan Engelbrecht2025-09-221-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* improve builds download partial logic (#501)Dan Engelbrecht2025-09-221-83/+85
| | | | | | | | | | - Improvement: `zen command` help now uses available horizontal space for output - Improvement: Partial block request analisys has been improved with reduced download size at lower number of requests - Improvement: Validate that `--zen-cache-host` exists and is responsive before begin used - Feature: Options `--allow-partial-block-requests` for `zen builds download` command has been augmented with two new modes, `zencacheonly` and `mixed`. Defaults to `mixed`. - `false` only full block requests allowed - `mixed` multiple partial block ranges requests per block allowed to zen cache, single partial block range request per block to host - `zencacheonly` multiple partial block ranges requests per block allowed to zen cache, only full block requests allowed to host - `true` multiple partial block ranges requests per block allowed to zen cache and host
* faster oplog entries with referenceset (#488)Dan Engelbrecht2025-09-081-1/+1
| | | | - Improvement: Faster project store `/entries` endpoint, 10-15% faster when using a reference set to limit entries fix missing space after task name in pretty progressbar
* refactor zen command return value handling (#487)Dan Engelbrecht2025-09-051-74/+130
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use consistent language for command line argument parsing errors - Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error - Improvement: Refactor zen command return codes to conform to valid Linux range (0-255) kSuccess = 0, kOtherError = 1, kBadInput = 2, kOutOfMemory = 16, kOutOfDisk = 17, kAssertError = 70, kHttpOtherClientError = 80, kHttpCantConnectError = 81, kHttpNotFound = 66, // NotFound(404) kHttpUnauthorized = 77, // Unauthorized(401), kHttpSLLError = 82, kHttpForbidden = 83, // Forbidden(403) kHttpTimeout = 84, // RequestTimeout(408) kHttpConflict = 85, // Conflict(409) kHttpNoHost = 86, kHttpOtherServerError = 90, kHttpInternalServerError = 91, // InternalServerError(500) kHttpServiceUnavailable = 69, // ServiceUnavailable(503) kHttpBadGateway = 92, // BadGateway(502) kHttpGatewayTimeout = 93, // GatewayTimeout(504)
* improve console output (#476)Dan Engelbrecht2025-08-261-9/+50
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
* Merge pull request #139 from ue-foundation/de/zen-service-commandLiam Mitchell2025-08-221-0/+3
|\ | | | | zen service command
| * Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-211-1/+2
| |\
| * \ Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-291-46/+315
| |\ \
| * \ \ Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-03-141-28/+144
| |\ \ \
| * | | | zen `service` commandDan Engelbrecht2025-01-081-0/+3
| | | | |
* | | | | clean up trace options parsing (#473)Dan Engelbrecht2025-08-221-33/+24
| |_|_|/ |/| | | | | | | | | | | | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* | | | Don't add quotes to arguments which are already quotedLiam Mitchell2025-07-151-1/+2
| |_|/ |/| |
* | | `--output-path` option added to `zen version` command (#440)Dan Engelbrecht2025-06-181-1/+1
| | | | | | | | | * `--output-path` option added to `zen version` command
* | | sentry config (#430)Dan Engelbrecht2025-06-121-6/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
* | | make sure we finish progress at 100% if not done (#432)Dan Engelbrecht2025-06-101-1/+1
| | | | | | | | | | | | | | | * make sure we finish progress at 100% if not done * fix condition
* | | revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-34/+6
| | |
* | | add sentry configurations options for debug/environmentDan Engelbrecht2025-06-101-6/+34
| | | | | | | | | | | | | | | add env-variable parsing for sentry option
* | | `--sentry-dsn` option for zen command line and zenserver to control Sentry ↵Dan Engelbrecht2025-06-091-8/+14
| | | | | | | | | | | | | | | reporting endpoint (#427) moved sentry database path to temporary directory for zen commandline
* | | pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-051-8/+35
| | | | | | | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* | | Initialize sentry after command line parsing is done (#420)Stefan Boberg2025-06-041-24/+25
| | | | | | | | | | | | Previously the option variables were used before options parsing had a chance to set them according to command line options
* | | skip empty or single-space command line arguments (#393)Dan Engelbrecht2025-05-131-1/+5
| | |
* | | add sentry for zen command (#373)Dan Engelbrecht2025-05-061-0/+35
| | | | | | | | | | | | * refactor sentry integration and add to zen command line tool * move add_ldflags("-framework Security")
* | | UE style formatted progress output (#380)Dan Engelbrecht2025-05-051-46/+135
| | | | | | | | | * add UE style @progress style progress
* | | zen wipe command (#366)Dan Engelbrecht2025-04-231-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: New `zen wipe` command for fast cleaning of directories, it will not remove the directory itself, only the content - `--directory` - path to directory to wipe, if the directory does not exist or is empty, no action will be taken - `--keep-readonly` - skip removal of read-only files found in directory, defaults to `true`, set to `false` to remove read-only files - `--quiet` - reduce output to console, defaults to `false` - `--dryrun` - simulate the wipe without removing anything, defaults to `false` - `--yes` - skips prompt to confirm wipe of directory - `--plain-progress` - show progress using plain output - `--verbose` - enable verbose console output - `--boost-workers` - increase the number of worker threads, may cause computer to be less responsive, defaults to `false`
* | | "unlimited" line length when using plain progress (#347)Dan Engelbrecht2025-04-041-7/+19
| | |
* | | progress bar improvements (#346)Dan Engelbrecht2025-04-041-18/+82
| |/ |/| | | | | * hide ETA until at least 5% is complete * dynamically adjust progres output length
* | fix quoted command lines arguments (#306)Dan Engelbrecht2025-03-121-0/+18
| | | | | | Handling of quotes and quotes with leading backslash for command line parsing - UE-231677
* | ProgressBar improvements (#305)Stefan Boberg2025-03-121-2/+41
| | | | | | | | | | * changed ProgressBar so it doesn't use printf. printf by default is very slow on Windows due to weird buffering behaviour. During a 2 minute build download I profiled 35 CPU seconds inside printf * changed so ProgressBar uses plain output mode if stdout is not a console/tty
* | improved block gen logic (#302)Dan Engelbrecht2025-03-121-0/+3
| | | | | | | | | | | | | | - Improvement: Reduced memory usage during upload and part upload validation - Improvement: Reduced I/O usage during upload and download - Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild - Improvement: More trace scopes for build upload operations - Bugfix: Fixed crash during download when trying to write outside a file range
* | Build command tweaks (#301)Stefan Boberg2025-03-111-33/+20
| | | | | | | | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-07 10:05:27 +0000