aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil
Commit message (Collapse)AuthorAgeFilesLines
* subprocess tracking using Jobs on Windows/hub (#796)HEADmainStefan Boberg11 days2-5/+24
| | | | | This change introduces job object support on Windows to be able to more accurately track and limit resource usage on storage instances created by the hub service. It also ensures that all child instances can be torn down reliably on exit. Also made it so hub tests no longer pop up console windows while running.
* Add test summary table and failure reporting to xmake test (#794)Stefan Boberg12 days2-1/+3
| | | | | | | | | | - Add a summary table printed after all test suites complete, showing per-suite test case counts, assertion counts, timings and pass/fail status. - Add failure reporting: individual failing test cases are listed at the end with their file path and line number for easy navigation. - Made zenserver instances spawned by a hub not create new console windows for a better background testing experience - The TestListener in testing.cpp now writes a machine-readable summary file (via `ZEN_TEST_SUMMARY_FILE` env var) containing aggregate counts and per-test-case failure details. This runs as a doctest listener alongside any active reporter, so it works with both console and JUnit modes. - Tests now run in a deterministic order defined by a single ordered list that also serves as the test name/target mapping, replacing the previous unordered table + separate order list. - The `--run` option now accepts comma-separated values (e.g. `--run=core,http,util`) and validates each name, reporting unknown test names early. - Fix platform detection in `xmake test`: the config command now passes `-p` explicitly, fixing "mingw" misdetection when running from Git Bash on Windows. - Add missing "util" entry to the help text for `--run`.
* Add `zen ui` command (#779)Stefan Boberg2026-02-242-0/+542
| | | | | Allows user to automate launching of zenserver dashboard, including when multiple instances are running. If multiple instances are running you can open all dashboards with `--all`, and also using the in-terminal chooser which also allows you to open a specific instance. Also includes a fix to `zen exec` when using offset/stride/limit
* added ResetConsoleLog (#758)Stefan Boberg2026-02-161-0/+5
| | | also made sure log initialization calls it to ensure the console output format is retained even if the console logger was set up before logging is initialized
* logging config move to zenutil (#754)Stefan Boberg2026-02-137-4/+118
| | | made logging config options from zenserver available in zen CLI
* reduce batch size for reads (#740)Dan Engelbrecht2026-01-291-1/+1
| | | | | * reduce maximum size per chunk to read to reduce disk contention * increase timeout before warning on slow shut down of zenserver * reduce default window size for blockstore chunk iteration
* hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-222-0/+40
| | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
* ZenServerProcess API changes (#719)Stefan Boberg2026-01-192-20/+133
| | | | | | | This refactor aims to improve the `ZenServerProcess` classes by making them useful for managing child zenserver instances in more scenarios than just automated tests. This involves changing some functions to not talk about "test directory" and instead use "data directory" etc As a consequence of the API changes, some tests have changed accordingly. The code includes som reference to the "hub" mode but there is not yet any other code using this mode, it's just included in this PR to simplify future merges.
* consul package and basic client added (#716)Stefan Boberg2026-01-192-0/+187
| | | | | | | | * this adds a consul package which can be used to fetch a consul binary * it also adds a `ConsulProcess` helper which can be used to spawn and manage a consul service instance * zencore dependencies brought across: - `except_fmt.h` for easer generation of formatted exception messages - `process.h/cpp` changes (adds `Kill` operation and process group support on Windows) - `string.h` changes to allow generic use of `WideToUtf8()`
* Merge pull request #701 from ue-foundation/lm/mac-daemon-modeLiam Mitchell2026-01-151-32/+22
|\ | | | | Implement final changes required for daemon mode on Mac
| * Run clang-format on service.cppLiam Mitchell2026-01-151-3/+2
| |
| * Implement final changes required for daemon mode on MacLiam Mitchell2026-01-071-32/+23
| |
* | minor fixes (#708)Stefan Boberg2026-01-131-1/+2
|/ | | | | * remove unreferenced local in projectstore_cmd * fix minor atomic memory issue in RotatingFileSink
* logging improvements (#664)Stefan Boberg2025-11-241-0/+10
| | | | | * Eliminate spurious `FlushFileBuffers` calls on Windows by tracking writes * Make cache log file creation optional
* update state when wildcard (#657)Dan Engelbrecht2025-11-242-0/+38
| | | * add --append option and improve state handling when using downloads for `zen builds download`
* fix wildcard matching (#650)Dan Engelbrecht2025-11-171-0/+10
|
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-1/+2
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-011-1/+1
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* move cpr in-tree (#605)Stefan Boberg2025-10-241-0/+2
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* in-tree spdlog (#602)Stefan Boberg2025-10-243-4/+4
| | | | move spdlog into the tree to remove dependency on vcpkg::spdlog, to allow diverging from the official version and evolve it to fit better with OTLP logging requirements
* move file i/o related files to separate file and remove duplicated code (#576)Dan Engelbrecht2025-10-152-118/+0
|
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-142-33/+4
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-132-0/+118
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* add ability to limit concurrency (#565)Stefan Boberg2025-10-102-50/+71
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* shorten thread pool names for Linux which has a limit of 15 characters (#563)Stefan Boberg2025-10-081-9/+9
| | | without this change we just get a whole bunch of threads named "main" or "zenserver" depending on timing
* speed up tests (#555)Dan Engelbrecht2025-10-061-3/+3
| | | | | | | | | | | | * faster FileSystemTraversal test * faster jobqueue test * faster NamedEvent test * faster cache tests * faster basic http tests * faster blockstore test * faster cache store tests * faster compactcas tests * more responsive zenserver launch * tweak worker pool sizes in tests
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-033-2/+2
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-038-2768/+0
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-035-143/+2
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-039-1581/+0
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-0311-2997/+0
|
* move zenutil builds code to zenremotestore (#543)Dan Engelbrecht2025-10-037-1979/+0
| | | | * move buildstorage implementations to zenremotestore lib * move builds storage to zenremotelib
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-1/+1
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* remove spurious cpr.h includes (#520)Stefan Boberg2025-09-291-2/+0
|
* Make sure we call the previous terminate handle if present when we intercept ↵Dan Engelbrecht2025-09-261-6/+39
| | | | | | terminate calls (#514) Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
* fix for C4244 truncation warning (#515)Stefan Boberg2025-09-261-1/+3
| | | Fixes warning/error introduced with msvc cl 19.38.33145
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-1/+2
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-226-0/+182
|\
| * fetch cloud oplog (#502)Dan Engelbrecht2025-09-226-0/+182
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* | Change batch put responses for client reportingzousar2025-09-192-2/+22
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* fix quoted wildcard options (#500)Dan Engelbrecht2025-09-171-7/+23
| | | * improve command line parsing to automatically remove quotes
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-104-37/+63
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-042-5/+43
| | | * add validation of compact binary payloads before reading them
* remove debug log messages (#479)Dan Engelbrecht2025-08-271-2/+0
|
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-262-0/+5
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* Move windows service utilities to zenutil and fix clang-format errorsLiam Mitchell2025-08-223-0/+664
|
* Move ReportServiceStatus to zenutil and remove extraneous loggingLiam Mitchell2025-08-223-0/+53
|
* Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-219-149/+393
|\
| * use new builds api for oplogs (#464)Dan Engelbrecht2025-08-125-133/+232
| | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
| * list build part content (#462)Dan Engelbrecht2025-08-113-0/+127
| | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 01:25:34 +0000