aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil/include
Commit message (Collapse)AuthorAgeFilesLines
* Add sequence numbers to log stream protocol and tests for drop detectionStefan Boberg40 hours2-1/+11
| | | | | | | | TcpLogStreamSink now stamps each message with a monotonic sequence number. LogStreamListener tracks the expected sequence per session and emits a synthetic "dropped" notice when gaps appear. Includes tests covering basic delivery, multi-line splitting, drop detection, and contiguous sequencing. Also simplifies LogMessage::s_DefaultPoint to a single entry.
* Use gathered buffer sequence in TcpLogStreamSink for batched TCP writesStefan Boberg40 hours1-8/+15
| | | | | | Replace per-message asio::write loop with a single gathered write using a fixed_vector of const_buffers, reducing syscall overhead when flushing batches of log messages.
* Add LogStreamListener and TcpLogStreamSink from compute-scheduler branchStefan Boberg2 days2-0/+237
| | | | | | Brings over TCP log streaming infrastructure for use in session management. LogStreamListener accepts connections and delivers log lines via the LogStreamTarget interface; TcpLogStreamSink is the sending counterpart.
* Merge main into sb/sessionizeStefan Boberg3 days1-2/+54
|\ | | | | | | Resolve html.zip conflict by accepting deletion from main.
| * Unix Domain Socket auto discovery (#833)Stefan Boberg4 days1-2/+54
| | | | | | | | | | | | | | | | This PR adds end-to-end Unix domain socket (UDS) support, allowing zen CLI to discover and connect to UDS-only servers automatically. - **`unix://` URI scheme in zen CLI**: The `-u` / `--hosturl` option now accepts `unix:///path/to/socket` to connect to a zenserver via a Unix domain socket instead of TCP. - **Per-instance shared memory for extended server info**: Each zenserver instance now publishes a small shared memory section (keyed by SessionId) containing per-instance data that doesn't fit in the fixed-size ZenServerEntry -- starting with the UDS socket path. This is a 4KB pagefile-backed section on Windows (`Global\ZenInstance_{sessionid}`) and a POSIX shared memory object on Linux/Mac (`/UnrealEngineZen_{sessionid}`). - **Client-side auto-discovery of UDS servers**: `zen info`, `zen status`, etc. now automatically discover and prefer UDS connections when a server publishes a socket path. Servers running with `--no-network` (UDS-only) are no longer invisible to the CLI. - **`kNoNetwork` flag in ZenServerEntry**: Servers started with `--no-network` advertise this in their shared state entry. Clients skip TCP fallback for these servers, and display commands (`ps`, `status`, `top`) show `-` instead of a port number to indicate TCP is not available.
* | Add session log ingestion, dashboard visualization, and log forwarding sinkStefan Boberg3 days1-0/+11
| | | | | | | | | | | | | | | | | | | | - Add POST/GET endpoints for session logs (/sessions/{id}/log) supporting raw text (newline-split) and structured JSON/CbObject with batch "entries" array - Add in-memory log storage per session (capped at 10k entries) with pagination - Add log panel to sessions dashboard with polling, auto-scroll, and level coloring - Add SessionLogSink that batches log messages on a background thread and forwards them to the sessions endpoint via HTTP, integrated into ZenStorageServer when a remote sessions URL is configured
* | Add SessionsServiceClient and enhanced sessions dashboardStefan Boberg4 days1-0/+54
|/ | | | | | | | | | | | | - Add SessionsServiceClient in zenutil for announcing sessions to a remote zenserver (POST/PUT/DELETE lifecycle with 15s heartbeat timer) - Storage server registers itself into its own local sessions service - Add session mode attribute coupled to server mode (Compute, Proxy, etc) - Track ended sessions with ended_at timestamp and status filtering - Sessions dashboard: master-detail layout with selectable rows, Active/Ended/All tabs, metadata panel, live WebSocket updates, paging, abbreviated date formatting, and "this" pill for self session - Accept JSON payloads on sessions POST endpoint via ReadPayloadObject - Add --sessions-url config option for remote session announcement
* hub consul integration (#820)Dan Engelbrecht6 days2-0/+87
| | | | | | | | - Feature: Basic consul integration for zenserver hub mode, restricted to host local consul agent and register/deregister of services - Feature: Added new options to zenserver hub mode - `consul-endpoint` - Consul endpoint URL for service registration (empty = disabled) - `hub-base-port-number` - Base port number for provisioned instances - `hub-instance-limit` - Maximum number of provisioned instances for this hub - `hub-use-job-object` - Enable the use of a Windows Job Object for child process management (Windows only)
* Merge branch 'main' into lm/oidctoken-exe-pathLiam Mitchell7 days10-357/+283
|\
| * Eliminate spdlog dependency (#773)Stefan Boberg8 days6-351/+168
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes the vendored spdlog library (~12,000 lines) and replaces it with a purpose-built logging system in zencore (~1,800 lines). The new implementation provides the same functionality with fewer abstractions, no shared_ptr overhead, and full control over the logging pipeline. ### What changed **New logging core in zencore/logging/:** - LogMessage, Formatter, Sink, Logger, Registry - core abstractions matching spdlog's model but simplified - AnsiColorStdoutSink - ANSI color console output (replaces spdlog stdout_color_sink) - MsvcSink - OutputDebugString on Windows (replaces spdlog msvc_sink) - AsyncSink - async logging via BlockingQueue worker thread (replaces spdlog async_logger) - NullSink, MessageOnlyFormatter - utility types - Thread-safe timestamp caching in formatters using RwLock **Moved to zenutil/logging/:** - FullFormatter - full log formatting with timestamp, logger name, level, source location, multiline alignment - JsonFormatter - structured JSON log output - RotatingFileSink - rotating file sink with atomic size tracking **API changes:** - Log levels are now an enum (LogLevel) instead of int, eliminating the zen::logging::level namespace - LoggerRef no longer wraps shared_ptr - it holds a raw pointer with the registry owning lifetime - Logger error handler is wired through Registry and propagated to all loggers on registration - Logger::Log() now populates ThreadId on every message **Cleanup:** - Deleted thirdparty/spdlog/ entirely (110+ files) - Deleted full_test_formatter (was ~80% duplicate of FullFormatter) - Renamed snake_case classes to PascalCase (full_formatter -> FullFormatter, json_formatter -> JsonFormatter, sentry_sink -> SentrySink) - Removed spdlog from xmake dependency graph ### Build / test impact - zencore no longer depends on spdlog - zenutil and zenvfs xmake.lua updated to drop spdlog dep - zentelemetry xmake.lua updated to drop spdlog dep - All existing tests pass, no test changes required beyond formatter class renames
| * compute orchestration (#763)Stefan Boberg13 days2-0/+2
| | | | | | | | | | | | | | | | | | | | - Added local process runners for Linux/Wine, Mac with some sandboxing support - Horde & Nomad provisioning for development and testing - Client session queues with lifecycle management (active/draining/cancelled), automatic retry with configurable limits, and manual reschedule API - Improved web UI for orchestrator, compute, and hub dashboards with WebSocket push updates - Some security hardening - Improved scalability and `zen exec` command Still experimental - compute support is disabled by default
| * added `--verbose` option to zenserver-test and `xmake test` (#798)Stefan Boberg2026-03-011-3/+7
| | | | | | | | | | | | * when `--verbose` is specified to zenserver-test, all child process output (typically, zenserver instances) is piped through to stdout. you can also pass `--verbose` to `xmake test` to accomplish the same thing. * this PR also consolidates all test runner `main` function logic (such as from zencore-test, zenhttp-test etc) into central implementation in zencore for consistency and ease of maintenance * also added extended utf8-tests including a fix to `Utf8ToWide()`
| * subprocess tracking using Jobs on Windows/hub (#796)Stefan Boberg2026-02-281-3/+9
| | | | | | | | | | This change introduces job object support on Windows to be able to more accurately track and limit resource usage on storage instances created by the hub service. It also ensures that all child instances can be torn down reliably on exit. Also made it so hub tests no longer pop up console windows while running.
| * Add test summary table and failure reporting to xmake test (#794)Stefan Boberg2026-02-271-0/+1
| | | | | | | | | | | | | | | | | | | | - Add a summary table printed after all test suites complete, showing per-suite test case counts, assertion counts, timings and pass/fail status. - Add failure reporting: individual failing test cases are listed at the end with their file path and line number for easy navigation. - Made zenserver instances spawned by a hub not create new console windows for a better background testing experience - The TestListener in testing.cpp now writes a machine-readable summary file (via `ZEN_TEST_SUMMARY_FILE` env var) containing aggregate counts and per-test-case failure details. This runs as a doctest listener alongside any active reporter, so it works with both console and JUnit modes. - Tests now run in a deterministic order defined by a single ordered list that also serves as the test name/target mapping, replacing the previous unordered table + separate order list. - The `--run` option now accepts comma-separated values (e.g. `--run=core,http,util`) and validates each name, reporting unknown test names early. - Fix platform detection in `xmake test`: the config command now passes `-p` explicitly, fixing "mingw" misdetection when running from Git Bash on Windows. - Add missing "util" entry to the help text for `--run`.
| * Add `zen ui` command (#779)Stefan Boberg2026-02-241-0/+59
| | | | | | | | | | Allows user to automate launching of zenserver dashboard, including when multiple instances are running. If multiple instances are running you can open all dashboards with `--all`, and also using the in-terminal chooser which also allows you to open a specific instance. Also includes a fix to `zen exec` when using offset/stride/limit
| * logging config move to zenutil (#754)Stefan Boberg2026-02-133-1/+38
| | | | | | made logging config options from zenserver available in zen CLI
* | Merge branch 'main' into lm/oidctoken-exe-pathLiam Mitchell7 days4-6/+92
|\|
| * hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-221-0/+13
| | | | | | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
| * ZenServerProcess API changes (#719)Stefan Boberg2026-01-191-5/+30
| | | | | | | | | | | | | | This refactor aims to improve the `ZenServerProcess` classes by making them useful for managing child zenserver instances in more scenarios than just automated tests. This involves changing some functions to not talk about "test directory" and instead use "data directory" etc As a consequence of the API changes, some tests have changed accordingly. The code includes som reference to the "hub" mode but there is not yet any other code using this mode, it's just included in this PR to simplify future merges.
| * consul package and basic client added (#716)Stefan Boberg2026-01-191-0/+47
| | | | | | | | | | | | | | | | * this adds a consul package which can be used to fetch a consul binary * it also adds a `ConsulProcess` helper which can be used to spawn and manage a consul service instance * zencore dependencies brought across: - `except_fmt.h` for easer generation of formatted exception messages - `process.h/cpp` changes (adds `Kill` operation and process group support on Windows) - `string.h` changes to allow generic use of `WideToUtf8()`
| * minor fixes (#708)Stefan Boberg2026-01-131-1/+2
| | | | | | | | | | * remove unreferenced local in projectstore_cmd * fix minor atomic memory issue in RotatingFileSink
* | Use well-known OidcToken paths or command line arguments to determine ↵Liam Mitchell2026-01-141-0/+12
|/ | | | OidcToken executable path
* logging improvements (#664)Stefan Boberg2025-11-241-0/+10
| | | | | * Eliminate spurious `FlushFileBuffers` calls on Windows by tracking writes * Make cache log file creation optional
* update state when wildcard (#657)Dan Engelbrecht2025-11-241-0/+5
| | | * add --append option and improve state handling when using downloads for `zen builds download`
* in-tree spdlog (#602)Stefan Boberg2025-10-242-2/+2
| | | | move spdlog into the tree to remove dependency on vcpkg::spdlog, to allow diverging from the official version and evolve it to fit better with OTLP logging requirements
* move file i/o related files to separate file and remove duplicated code (#576)Dan Engelbrecht2025-10-151-43/+0
|
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-6/+4
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-0/+43
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-3/+3
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-031-0/+0
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-034-641/+0
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-032-109/+0
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-034-350/+0
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-034-462/+0
|
* move zenutil builds code to zenremotestore (#543)Dan Engelbrecht2025-10-034-178/+0
| | | | * move buildstorage implementations to zenremotestore lib * move builds storage to zenremotelib
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-1/+2
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-223-0/+51
|\
| * fetch cloud oplog (#502)Dan Engelbrecht2025-09-223-0/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* | Change batch put responses for client reportingzousar2025-09-191-1/+2
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-18/+21
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-261-0/+1
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* Move windows service utilities to zenutil and fix clang-format errorsLiam Mitchell2025-08-221-0/+24
|
* Move ReportServiceStatus to zenutil and remove extraneous loggingLiam Mitchell2025-08-221-0/+2
|
* Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-213-0/+19
|\
| * use new builds api for oplogs (#464)Dan Engelbrecht2025-08-122-0/+6
| | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
| * list build part content (#462)Dan Engelbrecht2025-08-111-0/+13
| | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* | Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2913-174/+462
|\|
| * move nodiscard to proper location (#447)Dan Engelbrecht2025-06-191-1/+1
| |
| * add retry for failed block metadata upload (#445)Dan Engelbrecht2025-06-191-3/+3
| | | | | | * add retry for failed block metadata upload
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-1/+2
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-17 03:32:04 +0000