aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil/include
Commit message (Collapse)AuthorAgeFilesLines
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-031-0/+0
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-034-641/+0
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-032-109/+0
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-034-350/+0
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-034-462/+0
|
* move zenutil builds code to zenremotestore (#543)Dan Engelbrecht2025-10-034-178/+0
| | | | * move buildstorage implementations to zenremotestore lib * move builds storage to zenremotelib
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-1/+2
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-223-0/+51
|\
| * fetch cloud oplog (#502)Dan Engelbrecht2025-09-223-0/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* | Change batch put responses for client reportingzousar2025-09-191-1/+2
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-18/+21
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-261-0/+1
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* Move windows service utilities to zenutil and fix clang-format errorsLiam Mitchell2025-08-221-0/+24
|
* Move ReportServiceStatus to zenutil and remove extraneous loggingLiam Mitchell2025-08-221-0/+2
|
* Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-213-0/+19
|\
| * use new builds api for oplogs (#464)Dan Engelbrecht2025-08-122-0/+6
| | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
| * list build part content (#462)Dan Engelbrecht2025-08-111-0/+13
| | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* | Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2913-174/+462
|\|
| * move nodiscard to proper location (#447)Dan Engelbrecht2025-06-191-1/+1
| |
| * add retry for failed block metadata upload (#445)Dan Engelbrecht2025-06-191-3/+3
| | | | | | * add retry for failed block metadata upload
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-1/+2
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
| * sentry config (#430)Dan Engelbrecht2025-06-121-0/+92
| | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
| * revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-92/+0
| |
| * add EnvironmentOptions helperDan Engelbrecht2025-06-101-0/+92
| |
| * pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-052-12/+18
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
| * builds download url (#419)Dan Engelbrecht2025-06-041-0/+1
| | | | | | | | * RemoveQuotes helper * `--url` option for `zen builds` command has been reworked to accept a "Cloud Artifact URL", removing the need to specify "host", "namespace" and "bucket" separately
| * faster scavenge (#417)Dan Engelbrecht2025-06-041-18/+21
| | | | | | | | - Improvement: Multithreaded scavenge pass for zen builds download - Improvement: Optimized check for modified files when verifying state of scavenged paths
| * fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-033-15/+137
| | | | | | | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
| * parallel work handle dispatch exception (#400)Dan Engelbrecht2025-05-164-122/+74
| | | | | | - Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception
| * builds allow redirect option (#379)Dan Engelbrecht2025-05-052-1/+3
| | | | | | * add --allow-redirect to zen builds upload/download
| * Merge branch 'main' into zs/zencli-list-namespaces-bucketsZousar Shaker2025-04-241-0/+28
| |\
| | * parse system dir for builds (#365)Dan Engelbrecht2025-04-231-5/+5
| | | | | | | | | | | | * make sure we always parse system options for zen builds command * make MakeSafeAbsolutePath nodiscard
| | * add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-221-0/+28
| | |
| * | Replace container with namespaceszousar2025-04-221-1/+1
| | |
| * | Add a list-container subcommand to zen builds commandzousar2025-04-142-0/+3
| |/
| * fix race condition in multipart download (#358)Dan Engelbrecht2025-04-112-13/+14
| |
| * added --find-max-block-count option to builds upload (#337)Dan Engelbrecht2025-04-022-2/+2
| |
| * build cache prime (#327)Dan Engelbrecht2025-03-272-1/+9
| | | | | | | | - Feature: zen `--boost-workers` option to builds `upload`, `download` and `validate-part` that will increase the number of worker threads, may cause computer to be less responsive - Feature: zen `--cache-prime-only` that uploads referenced data from a part to `--zen-cache-host` if it is not already present. Target folder will be untouched.
| * optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-0/+4
| | | | | | | | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
| * zen build cache service (#318)Dan Engelbrecht2025-03-263-4/+55
| | | | | | | | | | | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
| * jupiter builds stats upload (#315)Dan Engelbrecht2025-03-192-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: At end of build upload we post statistics to the Jupiter build stats endpoint: - `totalSize` - `reusedRatio` - `reusedBlockCount` - `reusedBlockByteCount` - `newBlockCount` - `newBlockByteCount` - `uploadedCount` - `uploadedByteCount` - `elapsedTimeSec` - `uploadedBytesPerSec`
| * improved reporting on async error (#312)Dan Engelbrecht2025-03-181-7/+9
| | | | | | * report async errors as individual errors
* | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-03-1413-32/+706
|\|
| * fixes for log timestamps (#304)Stefan Boberg2025-03-121-1/+6
| | | | | | | | | | | | * add GetTimeSinceProcessStart returning time since process start. implemented using https://github.com/maxliani/GetTimeSinceProcessStart/tree/main * fix fractions when using epoch mode. Previously it would show the fraction from the absolute time stamp and not relative to epoch * used GetTimeSinceProcessStart to offset the epoch so that it represents the process spawn time
| * improved block gen logic (#302)Dan Engelbrecht2025-03-121-2/+2
| | | | | | | | | | | | | | - Improvement: Reduced memory usage during upload and part upload validation - Improvement: Reduced I/O usage during upload and download - Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild - Improvement: More trace scopes for build upload operations - Bugfix: Fixed crash during download when trying to write outside a file range
| * Build command tweaks (#301)Stefan Boberg2025-03-112-1/+2
| | | | | | | | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
| * pick up existing cache (#299)Dan Engelbrecht2025-03-101-4/+5
| | | | | | | | | | | | - Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run - Improvement: Faster abort during stream compression - Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write - Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails
| * partial block fetch (#298)Dan Engelbrecht2025-03-072-2/+7
| | | | | | | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
| * reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-061-0/+6
| | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
| * refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-031-13/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-29 01:15:06 +0000