aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil
Commit message (Collapse)AuthorAgeFilesLines
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-1/+1
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* remove spurious cpr.h includes (#520)Stefan Boberg2025-09-291-2/+0
|
* Make sure we call the previous terminate handle if present when we intercept ↵Dan Engelbrecht2025-09-261-6/+39
| | | | | | terminate calls (#514) Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
* fix for C4244 truncation warning (#515)Stefan Boberg2025-09-261-1/+3
| | | Fixes warning/error introduced with msvc cl 19.38.33145
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-1/+2
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-226-0/+182
|\
| * fetch cloud oplog (#502)Dan Engelbrecht2025-09-226-0/+182
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* | Change batch put responses for client reportingzousar2025-09-192-2/+22
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* fix quoted wildcard options (#500)Dan Engelbrecht2025-09-171-7/+23
| | | * improve command line parsing to automatically remove quotes
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-104-37/+63
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-042-5/+43
| | | * add validation of compact binary payloads before reading them
* remove debug log messages (#479)Dan Engelbrecht2025-08-271-2/+0
|
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-262-0/+5
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* Move windows service utilities to zenutil and fix clang-format errorsLiam Mitchell2025-08-223-0/+664
|
* Move ReportServiceStatus to zenutil and remove extraneous loggingLiam Mitchell2025-08-223-0/+53
|
* Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-219-149/+393
|\
| * use new builds api for oplogs (#464)Dan Engelbrecht2025-08-125-133/+232
| | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
| * list build part content (#462)Dan Engelbrecht2025-08-113-0/+127
| | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
| * more details in parallel work when wait fails/destructor has inconsistent ↵Dan Engelbrecht2025-08-061-12/+30
| | | | | | | | state (#457)
| * Don't set m_DispatchComplete in ParallelWork until after pending work ↵Dan Engelbrecht2025-08-041-4/+4
| | | | | | | | countdown succeeds (#455)
* | Fix for MacOS compile failureLiam Mitchell2025-08-201-0/+2
| |
* | Merge pull request #452 from ue-foundation/lm/full-service-installLiam Mitchell2025-08-061-5/+2
|\ \ | | | | | | Add --full option to service install, which will handle stop/uninstall if necessary, and copy binaries to install location
| * | Fix permissions and ownership issues with service binary copy and remove ↵Liam Mitchell2025-07-251-5/+2
| | | | | | | | | | | | unnecessary alias from unit file
* | | Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2928-454/+2263
|\ \ \ | |/ / |/| / | |/
| * move nodiscard to proper location (#447)Dan Engelbrecht2025-06-191-1/+1
| |
| * add retry for failed block metadata upload (#445)Dan Engelbrecht2025-06-193-5/+11
| | | | | | * add retry for failed block metadata upload
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-162-2/+30
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
| * sentry config (#430)Dan Engelbrecht2025-06-122-0/+176
| | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
| * revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-102-176/+0
| |
| * add EnvironmentOptions helperDan Engelbrecht2025-06-102-0/+176
| |
| * pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-055-31/+44
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
| * builds download url (#419)Dan Engelbrecht2025-06-042-6/+15
| | | | | | | | * RemoveQuotes helper * `--url` option for `zen builds` command has been reworked to accept a "Cloud Artifact URL", removing the need to specify "host", "namespace" and "bucket" separately
| * new builds search (#418)Dan Engelbrecht2025-06-041-3/+4
| | | | | | * don't require bucket for search operations to allow multi-bucket search
| * faster scavenge (#417)Dan Engelbrecht2025-06-042-46/+84
| | | | | | | | - Improvement: Multithreaded scavenge pass for zen builds download - Improvement: Optimized check for modified files when verifying state of scavenged paths
| * fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-037-140/+501
| | | | | | | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
| * faster oplog validate (#408)Dan Engelbrecht2025-05-301-1/+1
| | | | | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
| * frequent disk space check (#407)Dan Engelbrecht2025-05-271-1/+5
| | | | | | | | * check low disk space condition more frequently and trigger GC when low water mark is reached * show waited time when waiting for zenserver instance to exit
| * use explicit capture for lambdas (#404)Dan Engelbrecht2025-05-201-20/+27
| |
| * handle exception with batch work (#401)Dan Engelbrecht2025-05-191-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
| * parallel work handle dispatch exception (#400)Dan Engelbrecht2025-05-168-151/+295
| | | | | | - Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception
| * builds allow redirect option (#379)Dan Engelbrecht2025-05-054-23/+60
| | | | | | * add --allow-redirect to zen builds upload/download
| * fix race in rpcrecorder (#375)Stefan Boberg2025-04-251-0/+6
| |
| * Merge branch 'main' into zs/zencli-list-namespaces-bucketsZousar Shaker2025-04-243-0/+243
| |\
| | * parse system dir for builds (#365)Dan Engelbrecht2025-04-231-5/+5
| | | | | | | | | | | | * make sure we always parse system options for zen builds command * make MakeSafeAbsolutePath nodiscard
| | * add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-223-0/+243
| | |
| * | Replace container with namespaceszousar2025-04-223-21/+25
| | |
| * | Add a list-container subcommand to zen builds commandzousar2025-04-145-1/+91
| |/
| * fix race condition in multipart download (#358)Dan Engelbrecht2025-04-115-43/+68
| |
| * scavenge builds (#352)Dan Engelbrecht2025-04-081-2/+2
| | | | | | | | - Improvement: `zen builds` now scavenges previous download locations for data to reduce download size, enabled by default, disable with `--enable-scavenge=false` - Bugfix: Failing to rename a file during download sometimes reported an error when it succeeded when retrying
| * added --find-max-block-count option to builds upload (#337)Dan Engelbrecht2025-04-025-8/+17
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-22 07:51:15 +0000