aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil
Commit message (Collapse)AuthorAgeFilesLines
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-125-133/+232
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* list build part content (#462)Dan Engelbrecht2025-08-113-0/+127
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* more details in parallel work when wait fails/destructor has inconsistent ↵Dan Engelbrecht2025-08-061-12/+30
| | | | state (#457)
* Don't set m_DispatchComplete in ParallelWork until after pending work ↵Dan Engelbrecht2025-08-041-4/+4
| | | | countdown succeeds (#455)
* move nodiscard to proper location (#447)Dan Engelbrecht2025-06-191-1/+1
|
* add retry for failed block metadata upload (#445)Dan Engelbrecht2025-06-193-5/+11
| | | * add retry for failed block metadata upload
* graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-162-2/+30
| | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
* sentry config (#430)Dan Engelbrecht2025-06-122-0/+176
| | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
* revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-102-176/+0
|
* add EnvironmentOptions helperDan Engelbrecht2025-06-102-0/+176
|
* pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-055-31/+44
| | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* builds download url (#419)Dan Engelbrecht2025-06-042-6/+15
| | | | * RemoveQuotes helper * `--url` option for `zen builds` command has been reworked to accept a "Cloud Artifact URL", removing the need to specify "host", "namespace" and "bucket" separately
* new builds search (#418)Dan Engelbrecht2025-06-041-3/+4
| | | * don't require bucket for search operations to allow multi-bucket search
* faster scavenge (#417)Dan Engelbrecht2025-06-042-46/+84
| | | | - Improvement: Multithreaded scavenge pass for zen builds download - Improvement: Optimized check for modified files when verifying state of scavenged paths
* fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-037-140/+501
| | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
* faster oplog validate (#408)Dan Engelbrecht2025-05-301-1/+1
| | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
* frequent disk space check (#407)Dan Engelbrecht2025-05-271-1/+5
| | | | * check low disk space condition more frequently and trigger GC when low water mark is reached * show waited time when waiting for zenserver instance to exit
* use explicit capture for lambdas (#404)Dan Engelbrecht2025-05-201-20/+27
|
* handle exception with batch work (#401)Dan Engelbrecht2025-05-191-1/+0
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* parallel work handle dispatch exception (#400)Dan Engelbrecht2025-05-168-151/+295
| | | - Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception
* builds allow redirect option (#379)Dan Engelbrecht2025-05-054-23/+60
| | | * add --allow-redirect to zen builds upload/download
* fix race in rpcrecorder (#375)Stefan Boberg2025-04-251-0/+6
|
* Merge branch 'main' into zs/zencli-list-namespaces-bucketsZousar Shaker2025-04-243-0/+243
|\
| * parse system dir for builds (#365)Dan Engelbrecht2025-04-231-5/+5
| | | | | | | | * make sure we always parse system options for zen builds command * make MakeSafeAbsolutePath nodiscard
| * add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-223-0/+243
| |
* | Replace container with namespaceszousar2025-04-223-21/+25
| |
* | Add a list-container subcommand to zen builds commandzousar2025-04-145-1/+91
|/
* fix race condition in multipart download (#358)Dan Engelbrecht2025-04-115-43/+68
|
* scavenge builds (#352)Dan Engelbrecht2025-04-081-2/+2
| | | | - Improvement: `zen builds` now scavenges previous download locations for data to reduce download size, enabled by default, disable with `--enable-scavenge=false` - Bugfix: Failing to rename a file during download sometimes reported an error when it succeeded when retrying
* added --find-max-block-count option to builds upload (#337)Dan Engelbrecht2025-04-025-8/+17
|
* verify that we can read input files that are only hashed (#333)Dan Engelbrecht2025-04-011-2/+6
| | | | | * output build and part details by default * output executable and version at start of builds command * verify that we can read files we do not chunk
* long filename support (#330)Dan Engelbrecht2025-03-313-15/+15
| | | - Bugfix: Long file paths now works correctly on Windows
* temp path options and reduced scanning of target folder (#328)Dan Engelbrecht2025-03-281-2/+16
| | | | | | - Feature: zen: `--zen-folder-path` added to `builds` command, `list`, `upload`, `download`, `fetch-blob`, `validate-part` to control where `.zen` folder is placed and named - Improvement: Only check known files from remote state when downloading to a target folder with no local state file - Improvement: Don't move existing local to cache and back if they are untouched
* build cache prime (#327)Dan Engelbrecht2025-03-274-9/+74
| | | | - Feature: zen `--boost-workers` option to builds `upload`, `download` and `validate-part` that will increase the number of worker threads, may cause computer to be less responsive - Feature: zen `--cache-prime-only` that uploads referenced data from a part to `--zen-cache-host` if it is not already present. Target folder will be untouched.
* optional compress of block chunks (#326)Dan Engelbrecht2025-03-272-1/+42
| | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* zen build cache service (#318)Dan Engelbrecht2025-03-267-41/+444
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* jupiter builds stats upload (#315)Dan Engelbrecht2025-03-195-0/+80
| | | | | | | | | | | | | - Improvement: At end of build upload we post statistics to the Jupiter build stats endpoint: - `totalSize` - `reusedRatio` - `reusedBlockCount` - `reusedBlockByteCount` - `newBlockCount` - `newBlockByteCount` - `uploadedCount` - `uploadedByteCount` - `elapsedTimeSec` - `uploadedBytesPerSec`
* improved reporting on async error (#312)Dan Engelbrecht2025-03-181-7/+9
| | | * report async errors as individual errors
* fixes for log timestamps (#304)Stefan Boberg2025-03-122-3/+10
| | | | | | * add GetTimeSinceProcessStart returning time since process start. implemented using https://github.com/maxliani/GetTimeSinceProcessStart/tree/main * fix fractions when using epoch mode. Previously it would show the fraction from the absolute time stamp and not relative to epoch * used GetTimeSinceProcessStart to offset the epoch so that it represents the process spawn time
* improved block gen logic (#302)Dan Engelbrecht2025-03-125-4/+42
| | | | | | | - Improvement: Reduced memory usage during upload and part upload validation - Improvement: Reduced I/O usage during upload and download - Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild - Improvement: More trace scopes for build upload operations - Bugfix: Fixed crash during download when trying to write outside a file range
* Build command tweaks (#301)Stefan Boberg2025-03-113-13/+17
| | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* pick up existing cache (#299)Dan Engelbrecht2025-03-102-18/+42
| | | | | | - Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run - Improvement: Faster abort during stream compression - Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write - Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails
* partial block fetch (#298)Dan Engelbrecht2025-03-075-12/+36
| | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-061-0/+6
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* Add trace support for zen CLI command (#296)Stefan Boberg2025-03-052-34/+76
| | | | - This change adds support for `--trace`, `--tracehost` and `--tracefile` command arguments to enable and control tracing to Insights - It also adds profiling scopes primarily to build download command related code
* refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-032-47/+84
| | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
* builds download incremental (#290)Dan Engelbrecht2025-03-011-5/+7
| | | | | * incremental download * merge rebuild state and output state building * fix writing when > 1 zero size file
* improve error handling (#289)Dan Engelbrecht2025-02-286-15/+79
| | | | | | | | * clearer errors * quicker abort * handle deleted local files * simplify parallellwork error handling * don't finish progress on destructor - gives wrong impression * graceful ctrl-c handling
* Zs/auth bad function fix (#287)Zousar Shaker2025-02-272-7/+5
| | | | | | | | | * Describe fix in changelog * remove JupiterClient::m_TokenProvider --------- Co-authored-by: zousar <[email protected]> Co-authored-by: Dan Engelbrecht <[email protected]>
* builds upload command (#278)Dan Engelbrecht2025-02-2612-18/+2678
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 11:04:49 +0000