aboutsummaryrefslogtreecommitdiff
path: root/src/zen/cmds
Commit message (Collapse)AuthorAgeFilesLines
* Restructure zen builds using subcommands (#834)Stefan Boberg2026-03-162-2284/+2647
| | | Refactored builds_cmd to split subcommands into dedicated classes, in an effort to reduce surface area and complexity to improve maintainability.
* Enable cross compilation of Windows targets on Linux (#839)Stefan Boberg2026-03-161-2/+2
| | | | | | | This PR makes it *possible* to do a Windows build on Linux via `clang-cl`. It doesn't actually change any build process. No policy change, just mechanics and some code fixes to clear clang compilation. The code fixes are mainly related to #include file name casing, to match the on-disk casing of the SDK files (via xwin).
* Unix Domain Socket auto discovery (#833)Stefan Boberg2026-03-1314-87/+139
| | | | | | | | This PR adds end-to-end Unix domain socket (UDS) support, allowing zen CLI to discover and connect to UDS-only servers automatically. - **`unix://` URI scheme in zen CLI**: The `-u` / `--hosturl` option now accepts `unix:///path/to/socket` to connect to a zenserver via a Unix domain socket instead of TCP. - **Per-instance shared memory for extended server info**: Each zenserver instance now publishes a small shared memory section (keyed by SessionId) containing per-instance data that doesn't fit in the fixed-size ZenServerEntry -- starting with the UDS socket path. This is a 4KB pagefile-backed section on Windows (`Global\ZenInstance_{sessionid}`) and a POSIX shared memory object on Linux/Mac (`/UnrealEngineZen_{sessionid}`). - **Client-side auto-discovery of UDS servers**: `zen info`, `zen status`, etc. now automatically discover and prefer UDS connections when a server publishes a socket path. Servers running with `--no-network` (UDS-only) are no longer invisible to the CLI. - **`kNoNetwork` flag in ZenServerEntry**: Servers started with `--no-network` advertise this in their shared state entry. Clients skip TCP fallback for these servers, and display commands (`ps`, `status`, `top`) show `-` instead of a port number to indicate TCP is not available.
* Transparent proxy mode (#823)Stefan Boberg2026-03-122-67/+122
| | | | | | | | | | | | | | | | | Adds a **transparent TCP proxy mode** to zenserver (activated via `zenserver proxy`), allowing it to sit between clients and upstream Zen servers to inspect and monitor HTTP/1.x traffic in real time. Primarily useful during development, to be able to observe multi-server/client interactions in one place. - **Dedicated proxy port** -- Proxy mode defaults to port 8118 with its own data directory to avoid collisions with a normal zenserver instance. - **TCP proxy core** (`src/zenserver/proxy/`) -- A new transparent TCP proxy that forwards connections to upstream targets, with support for both TCP/IP and Unix socket listeners. Multi-threaded I/O for connection handling. Supports Unix domain sockets for both upstream/downstream. - **HTTP traffic inspection** -- Parses HTTP/1.x request/response streams inline to extract method, path, status, content length, and WebSocket upgrades without breaking the proxied data. - **Proxy dashboard** -- A web UI showing live connection stats, per-target request counts, active connections, bytes transferred, and client IP/session ID rollups. - **Server mode display** -- Dashboard banner now shows the running server mode (Zen Proxy, Zen Compute, etc.). Supporting changes included in this branch: - **Wildcard log level matching** -- Log levels can now be set per-category using wildcard patterns (e.g. `proxy.*=debug`). - **`zen down --all`** -- New flag to shut down all running zenserver instances; also used by the new `xmake kill` task. - Minor test stability fixes (flaky hash collisions, per-thread RNG seeds). - Support ZEN_MALLOC environment variable for default allocator selection and switch default to rpmalloc - Fixed sentry-native build to allow LTO on Windows
* improved oplog import progress reporting (#825)Dan Engelbrecht2026-03-111-8/+10
|
* minor zenstore/blockstore fixes (#821)Stefan Boberg2026-03-102-69/+49
| | | | | | | | | | | - Fix clang-format error accidentally introduced by recent PR - Fix `FileSize()` CAS race that repeatedly invalidated the cache when concurrent callers both missed; remove `store(0)` on CAS failure - Fix `WriteChunks` not accounting for initial alignment padding in `m_TotalSize`, causing drift vs `WriteChunk`'s correct accounting - Fix Create retry sleep computing negative values (100 - N*100 instead of 100 + N*100), matching the Open retry pattern - Fix `~BlockStore` error log missing format placeholder for `Ex.what()` - Fix `GetFreeBlockIndex` infinite loop when all indexes have orphan files on disk but aren't in `m_ChunkBlocks`; bound probe to `m_MaxBlockCount` - Fix `IterateBlock` ignoring `SmallSizeCallback` return value for single out-of-bounds chunks, preventing early termination - Fix `BlockStoreCompactState::IterateBlocks` iterating map by value instead of const reference
* HttpClient using libcurl, Unix Sockets for HTTP. HTTPS support (#770)Stefan Boberg2026-03-102-53/+504
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main goal of this change is to eliminate the cpr back-end altogether and replace it with the curl implementation. I would expect to drop cpr as soon as we feel happy with the libcurl back-end. That would leave us with a direct dependency on libcurl only, and cpr can be eliminated as a dependency. ### HttpClient Backend Overhaul - Implemented a new **libcurl-based HttpClient** backend (`httpclientcurl.cpp`, ~2000 lines) as an alternative to the cpr-based one - Made HttpClient backend **configurable at runtime** via constructor arguments and `-httpclient=...` CLI option (for zen, zenserver, and tests) - Extended HttpClient test suite to cover multipart/content-range scenarios ### Unix Domain Socket Support - Added Unix domain socket support to **httpasio** (server side) - Added Unix domain socket support to **HttpClient** - Added Unix domain socket support to **HttpWsClient** (WebSocket client) - Templatized `HttpServerConnectionT<SocketType>` and `WsAsioConnectionT<SocketType>` to handle TCP, Unix, and SSL sockets uniformly via `if constexpr` dispatch ### HTTPS Support - Added **preliminary HTTPS support to httpasio** (for Mac/Linux via OpenSSL) - Added **basic HTTPS support for http.sys** (Windows) - Implemented HTTPS test for httpasio - Split `InitializeServer` into smaller sub-functions for http.sys ### Other Notable Changes - Improved **zenhttp-test stability** with dynamic port allocation - Enhanced port retry logic in http.sys (handles ERROR_ACCESS_DENIED) - Fatal signal/exception handlers for backtrace generation in tests - Added `zen bench http` subcommand to exercise network + HTTP client/server communication stack
* Merge branch 'main' into lm/restrict-content-typeLiam Mitchell2026-03-0926-171/+2100
|\
| * add fallback for zencache multirange (#816)Dan Engelbrecht2026-03-092-77/+88
| | | | | | | | | | | | | | | | | | | | | | * clean up BuildStorageResolveResult to allow capabilities * add check for multirange request capability * add MaxRangeCountPerRequest capabilities * project export tests * add InMemoryBuildStorageCache * progress and logging improvements * fix ElapsedSeconds calculations in fileremoteprojectstore.cpp * oplogs/builds test script
| * Eliminate spdlog dependency (#773)Stefan Boberg2026-03-092-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes the vendored spdlog library (~12,000 lines) and replaces it with a purpose-built logging system in zencore (~1,800 lines). The new implementation provides the same functionality with fewer abstractions, no shared_ptr overhead, and full control over the logging pipeline. ### What changed **New logging core in zencore/logging/:** - LogMessage, Formatter, Sink, Logger, Registry - core abstractions matching spdlog's model but simplified - AnsiColorStdoutSink - ANSI color console output (replaces spdlog stdout_color_sink) - MsvcSink - OutputDebugString on Windows (replaces spdlog msvc_sink) - AsyncSink - async logging via BlockingQueue worker thread (replaces spdlog async_logger) - NullSink, MessageOnlyFormatter - utility types - Thread-safe timestamp caching in formatters using RwLock **Moved to zenutil/logging/:** - FullFormatter - full log formatting with timestamp, logger name, level, source location, multiline alignment - JsonFormatter - structured JSON log output - RotatingFileSink - rotating file sink with atomic size tracking **API changes:** - Log levels are now an enum (LogLevel) instead of int, eliminating the zen::logging::level namespace - LoggerRef no longer wraps shared_ptr - it holds a raw pointer with the registry owning lifetime - Logger error handler is wired through Registry and propagated to all loggers on registration - Logger::Log() now populates ThreadId on every message **Cleanup:** - Deleted thirdparty/spdlog/ entirely (110+ files) - Deleted full_test_formatter (was ~80% duplicate of FullFormatter) - Renamed snake_case classes to PascalCase (full_formatter -> FullFormatter, json_formatter -> JsonFormatter, sentry_sink -> SentrySink) - Removed spdlog from xmake dependency graph ### Build / test impact - zencore no longer depends on spdlog - zenutil and zenvfs xmake.lua updated to drop spdlog dep - zentelemetry xmake.lua updated to drop spdlog dep - All existing tests pass, no test changes required beyond formatter class renames
| * more feedback during auth option parsing (#806)Dan Engelbrecht2026-03-041-2/+0
| | | | | | | | | | * remove stray std::unique_ptr<AuthMgr> Auth; causing crashes * add more feedback during parsing of auth options
| * compute orchestration (#763)Stefan Boberg2026-03-042-22/+739
| | | | | | | | | | | | | | | | | | | | - Added local process runners for Linux/Wine, Mac with some sandboxing support - Horde & Nomad provisioning for development and testing - Client session queues with lifecycle management (active/draining/cancelled), automatic retry with configurable limits, and manual reschedule API - Improved web UI for orchestrator, compute, and hub dashboards with WebSocket push updates - Some security hardening - Improved scalability and `zen exec` command Still experimental - compute support is disabled by default
| * unity build fixes (#802)Stefan Boberg2026-03-043-10/+23
| | | | | | | | | | Various fixes to make cpp files build in unity build mode as an aside using Unity build doesn't really seem to work on Linux, unsure why but it leads to link-time issues
| * use multi range requests (#800)Dan Engelbrecht2026-03-033-3/+3
| | | | | | | | | | | | | | - Improvement: `zen builds download` now uses multi-range requests for blocks to reduce download size - Improvement: `zen oplog-import` now uses partial block with multi-range requests for blocks to reduce download size - Improvement: Improved feedback in log/console during `zen oplog-import` - Improvement: `--allow-partial-block-requests` now defaults to `true` for `zen builds download` and `zen oplog-import` (was `mixed`) - Improvement: Improved range merging analysis when downloading partial blocks
| * use partial blocks for oplog import (#780)Dan Engelbrecht2026-02-243-11/+47
| | | | | | | | | | Feature: Add --allow-partial-block-requests to zen oplog-import Improvement: zen oplog-import now uses partial block requests to reduce download size Improvement: Use latency to Cloud Storage host and Zen Cache host when calculating partial block requests
| * Add `zen ui` command (#779)Stefan Boberg2026-02-243-1/+276
| | | | | | | | | | Allows user to automate launching of zenserver dashboard, including when multiple instances are running. If multiple instances are running you can open all dashboards with `--all`, and also using the in-terminal chooser which also allows you to open a specific instance. Also includes a fix to `zen exec` when using offset/stride/limit
| * changed command names and descriptions to use class members instead of ↵Stefan Boberg2026-02-2316-47/+168
| | | | | | | | string literals in zen.cpp (#776)
| * fix builds download indexing timer (#769)Dan Engelbrecht2026-02-201-0/+7
| | | | | | * fix build download indexing timer log
| * fix MakeSafeAbsolutePathInPlace mis-spelling (#765)Stefan Boberg2026-02-205-27/+27
| | | | | | | | | | (was MakeSafeAbsolutePathÍnPlace - note accent) Also fixed misleading comments on multiple functions in filesystem.h
| * structured compute basics (#714)Stefan Boberg2026-02-182-0/+751
| | | | | | | | | | | | | | | | | | this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
| * spelling fixes (#755)Dan Engelbrecht2026-02-131-2/+2
| |
* | Restrict content-type on POST requests to compact binary or JSONLiam Mitchell2026-02-111-1/+3
|/
* add command line option for scrub timeslice (#742)Dan Engelbrecht2026-02-032-4/+14
|
* Avoid conversion from JSon to compact binary when querying for builds to ↵Dan Engelbrecht2026-01-281-5/+13
| | | | avoid integer vs float conversion issues (#735)
* allow download specification for zen builds download (#734)Dan Engelbrecht2026-01-272-18/+51
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds download` now supports `--download-spec-path` to determine what content to download from a build - The unstructured format expects one line per file relative to the root with '/' as a path delimiter - The structured format uses JSon format and the `--download-spec-path` must have extension `.json` to enable structured input { "parts": { "default" : { "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* builds scanning cache (#727)v5.7.19-pre0Dan Engelbrecht2026-01-232-115/+218
| | | | - Feature: Added `--chunking-cache-path` option to `zen builds upload` and `zen builds diff` - Path to cache for chunking information of scanned files. Default is empty resulting in no caching
* hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-221-14/+10
| | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
* builds multipart upload (#722)Dan Engelbrecht2026-01-201-637/+259
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds upload` now support structure manifest input for `--manifest-path` when the path has a `.json` extension - The structured manifest supports splitting a build into multiple parts { "parts": { "default" : { "partId": "f939f3939939fff3f3202", # optional - used to control the id of each part "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* zenserver API changes, some other minor changes (#720)Stefan Boberg2026-01-191-0/+42
| | | | | | | * add system metrics output to top command * removed unnecessary xmake directives * file system API/comment tweaks * fixed out-of-range access in httpserver test * updated ZenServer base API to allow customization by mode
* ZenServerProcess API changes (#719)Stefan Boberg2026-01-191-0/+1
| | | | | | | This refactor aims to improve the `ZenServerProcess` classes by making them useful for managing child zenserver instances in more scenarios than just automated tests. This involves changing some functions to not talk about "test directory" and instead use "data directory" etc As a consequence of the API changes, some tests have changed accordingly. The code includes som reference to the "hub" mode but there is not yet any other code using this mode, it's just included in this PR to simplify future merges.
* Merge pull request #701 from ue-foundation/lm/mac-daemon-modeLiam Mitchell2026-01-151-9/+13
|\ | | | | Implement final changes required for daemon mode on Mac
| * Rename IsElevated to RequiresElevationLiam Mitchell2026-01-151-9/+9
| |
| * Implement final changes required for daemon mode on MacLiam Mitchell2026-01-071-1/+5
| |
* | throw error on bad oidctoken-exe-path (#713)Dan Engelbrecht2026-01-151-10/+20
| | | | | | | | * throw error on bad oidctoken-exe-path
* | structured output for builds ls (#709)Dan Engelbrecht2026-01-142-67/+176
| | | | | | | | * make ResolveBuildStore respect Verbose flag * add structured output to zen builds ls command
* | minor fixes (#708)Stefan Boberg2026-01-131-2/+0
| | | | | | | | | | * remove unreferenced local in projectstore_cmd * fix minor atomic memory issue in RotatingFileSink
* | added options to configure exclude folders and extensions to zen build ↵Dan Engelbrecht2026-01-132-41/+121
|/ | | | | | | | commands (#706) * added `--exclude-folders` to `zen upload`, `zen download` and `zen diff` added `--exclude-extensions` to `zen upload` and `zen diff` excluded folder names are now matched by folder name in subfolders in addition to root level folders * allow multiple token separators
* optimize scavenge (#697)Dan Engelbrecht2025-12-191-85/+19
| | | | * optimize FindScavengeContent * optimize GetValidFolderContent
* remove error warning in output (#695)Dan Engelbrecht2025-12-171-1/+1
| | | * changed some logging string so they don't get caught in CI logging
* add boost-worker oplog import export options (#693)Dan Engelbrecht2025-12-164-74/+202
| | | | | | | | | | - Feature: `zen oplog-export`, `zen oplog-import` and `zen oplog-download` now has options to boost workers - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive - Improvement: Refactored boost options for `zen builds` operations `upload`, `download`, `diff`, `prime-cache`, `fetch-blob` and `validate-part` - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive
* oplog download size (#690)Dan Engelbrecht2025-12-151-16/+19
| | | | - Bugfix: Upload of oplogs could reference multiple blocks for the same chunk causing redundant downloads of blocks - Improvement: Use the improved block reuse selection function from zen builds upload in zen oplog-export to reduce oplog download size
* show download source data (#689)Dan Engelbrecht2025-12-121-9/+60
| | | * show source stats for jupiter/cache
* remove direct console output from code that is used from service mode (#688)Dan Engelbrecht2025-12-112-79/+99
|
* fix buffer memory in builds cache (#682)Dan Engelbrecht2025-12-101-0/+6
| | | | * add --zen-cache-upload option to zen oplog-import command * fix buildstoragecache to not hold on to possibly materialized buffers
* upload to zen cache from oplog import now disabled by default (#680)Dan Engelbrecht2025-12-091-1/+1
|
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-271-1/+10
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* remove 'auto' option for zen builds download (#665)Dan Engelbrecht2025-11-252-52/+4
|
* update state when wildcard (#657)Dan Engelbrecht2025-11-242-363/+524
| | | * add --append option and improve state handling when using downloads for `zen builds download`
* Merge pull request #653 from ue-foundation/lm/service-install-permissionsLiam Mitchell2025-11-191-0/+4
|\ | | | | Make files writeable after full service install to avoid issues when attempting to update from a read-only source such as p4
| * Make files writeable after full service install to avoid issues when ↵Liam Mitchell2025-11-171-0/+4
| | | | | | | | attempting to update from a read-only source such as p4
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-07 05:05:09 +0000