aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil/include
Commit message (Collapse)AuthorAgeFilesLines
* added logging of zenserver child process startup/shutdown timingsStefan Boberg2026-02-231-0/+2
|
* logging config move to zenutil (#754)Stefan Boberg2026-02-133-1/+38
| | | made logging config options from zenserver available in zen CLI
* hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-221-0/+13
| | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
* ZenServerProcess API changes (#719)Stefan Boberg2026-01-191-5/+30
| | | | | | | This refactor aims to improve the `ZenServerProcess` classes by making them useful for managing child zenserver instances in more scenarios than just automated tests. This involves changing some functions to not talk about "test directory" and instead use "data directory" etc As a consequence of the API changes, some tests have changed accordingly. The code includes som reference to the "hub" mode but there is not yet any other code using this mode, it's just included in this PR to simplify future merges.
* consul package and basic client added (#716)Stefan Boberg2026-01-191-0/+47
| | | | | | | | * this adds a consul package which can be used to fetch a consul binary * it also adds a `ConsulProcess` helper which can be used to spawn and manage a consul service instance * zencore dependencies brought across: - `except_fmt.h` for easer generation of formatted exception messages - `process.h/cpp` changes (adds `Kill` operation and process group support on Windows) - `string.h` changes to allow generic use of `WideToUtf8()`
* minor fixes (#708)Stefan Boberg2026-01-131-1/+2
| | | | | * remove unreferenced local in projectstore_cmd * fix minor atomic memory issue in RotatingFileSink
* logging improvements (#664)Stefan Boberg2025-11-241-0/+10
| | | | | * Eliminate spurious `FlushFileBuffers` calls on Windows by tracking writes * Make cache log file creation optional
* update state when wildcard (#657)Dan Engelbrecht2025-11-241-0/+5
| | | * add --append option and improve state handling when using downloads for `zen builds download`
* in-tree spdlog (#602)Stefan Boberg2025-10-242-2/+2
| | | | move spdlog into the tree to remove dependency on vcpkg::spdlog, to allow diverging from the official version and evolve it to fit better with OTLP logging requirements
* move file i/o related files to separate file and remove duplicated code (#576)Dan Engelbrecht2025-10-151-43/+0
|
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-6/+4
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-0/+43
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-3/+3
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-031-0/+0
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-034-641/+0
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-032-109/+0
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-034-350/+0
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-034-462/+0
|
* move zenutil builds code to zenremotestore (#543)Dan Engelbrecht2025-10-034-178/+0
| | | | * move buildstorage implementations to zenremotestore lib * move builds storage to zenremotelib
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-1/+2
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-223-0/+51
|\
| * fetch cloud oplog (#502)Dan Engelbrecht2025-09-223-0/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* | Change batch put responses for client reportingzousar2025-09-191-1/+2
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-18/+21
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-261-0/+1
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* Move windows service utilities to zenutil and fix clang-format errorsLiam Mitchell2025-08-221-0/+24
|
* Move ReportServiceStatus to zenutil and remove extraneous loggingLiam Mitchell2025-08-221-0/+2
|
* Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-213-0/+19
|\
| * use new builds api for oplogs (#464)Dan Engelbrecht2025-08-122-0/+6
| | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
| * list build part content (#462)Dan Engelbrecht2025-08-111-0/+13
| | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* | Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2913-174/+462
|\|
| * move nodiscard to proper location (#447)Dan Engelbrecht2025-06-191-1/+1
| |
| * add retry for failed block metadata upload (#445)Dan Engelbrecht2025-06-191-3/+3
| | | | | | * add retry for failed block metadata upload
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-1/+2
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
| * sentry config (#430)Dan Engelbrecht2025-06-121-0/+92
| | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
| * revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-92/+0
| |
| * add EnvironmentOptions helperDan Engelbrecht2025-06-101-0/+92
| |
| * pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-052-12/+18
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
| * builds download url (#419)Dan Engelbrecht2025-06-041-0/+1
| | | | | | | | * RemoveQuotes helper * `--url` option for `zen builds` command has been reworked to accept a "Cloud Artifact URL", removing the need to specify "host", "namespace" and "bucket" separately
| * faster scavenge (#417)Dan Engelbrecht2025-06-041-18/+21
| | | | | | | | - Improvement: Multithreaded scavenge pass for zen builds download - Improvement: Optimized check for modified files when verifying state of scavenged paths
| * fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-033-15/+137
| | | | | | | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
| * parallel work handle dispatch exception (#400)Dan Engelbrecht2025-05-164-122/+74
| | | | | | - Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception
| * builds allow redirect option (#379)Dan Engelbrecht2025-05-052-1/+3
| | | | | | * add --allow-redirect to zen builds upload/download
| * Merge branch 'main' into zs/zencli-list-namespaces-bucketsZousar Shaker2025-04-241-0/+28
| |\
| | * parse system dir for builds (#365)Dan Engelbrecht2025-04-231-5/+5
| | | | | | | | | | | | * make sure we always parse system options for zen builds command * make MakeSafeAbsolutePath nodiscard
| | * add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-221-0/+28
| | |
| * | Replace container with namespaceszousar2025-04-221-1/+1
| | |
| * | Add a list-container subcommand to zen builds commandzousar2025-04-142-0/+3
| |/
| * fix race condition in multipart download (#358)Dan Engelbrecht2025-04-112-13/+14
| |
| * added --find-max-block-count option to builds upload (#337)Dan Engelbrecht2025-04-022-2/+2
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 03:30:55 +0000