aboutsummaryrefslogtreecommitdiff
path: root/src/zen/cmds/builds_cmd.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* allow --zen-cache-host option when using builds filestore (for testing) (#627)Dan Engelbrecht2025-11-031-0/+32
|
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-011-19/+26
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* use already built lookup when verifying folder (#615)Dan Engelbrecht2025-10-271-3/+2
|
* fixed progress bar when scanning changed local files (#608)Dan Engelbrecht2025-10-241-23/+28
| | | * fixed progress bar when scanning changed local files
* add host discovery and zen cache support for oplog import (#601)Dan Engelbrecht2025-10-231-181/+63
| | | * add host discovery and zen cache support for oplog import
* add --boost-workers option to zen builds prime-cache (#604)Dan Engelbrecht2025-10-231-0/+1
|
* add `zen builds prime-cache` command (#598)Dan Engelbrecht2025-10-221-24/+154
|
* tweak worker pools for builds (#595)Dan Engelbrecht2025-10-221-17/+115
| | | * update logic for number of network and disk threads for `zen builds` commands
* make validation of completed sequences in builds download optional (#596)Dan Engelbrecht2025-10-221-0/+1
|
* updated chunking strategy (#589)Dan Engelbrecht2025-10-201-7/+16
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
* add ability to abort http requests (#586)Dan Engelbrecht2025-10-171-2/+3
| | | * add abort-check function to httpclient
* builds download progress include validate (#582)Dan Engelbrecht2025-10-161-2/+8
| | | * take validation into account for progress feedback when downloading builds
* fix log progress quotes (#580)Dan Engelbrecht2025-10-161-1/+5
| | | | * add quotes to message in log-progress mode * improved log progress for builds upload and download
* move builds state functions to buildsavedstate.h/cpp (#577)Dan Engelbrecht2025-10-151-442/+22
|
* move file i/o related files to separate file and remove duplicated code (#576)Dan Engelbrecht2025-10-151-289/+6
|
* refactor builds cmd part3 (#573)Dan Engelbrecht2025-10-141-482/+29
| | | | * move lambdas to member functions * add BuildsOperationValidateBuildPart
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-2668/+293
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-4446/+152
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* block reference couting copy support windows (#564)Dan Engelbrecht2025-10-111-35/+207
| | | | - Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during `zen builds download` operations - Enabled by default, disable with `--allow-file-clone=false`
* executable/version/pid log at command startup cleaned up (#557)Dan Engelbrecht2025-10-061-18/+8
|
* changed std::vector<bool> to std::vector<uint8_t> to avoid threading issues ↵Stefan Boberg2025-10-061-6/+7
| | | | (#561)
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-061-2/+7
|
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-10/+10
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-031-4/+4
|
* move zenutil builds code to zenremotestore (#543)Dan Engelbrecht2025-10-031-3/+3
| | | | * move buildstorage implementations to zenremotestore lib * move builds storage to zenremotelib
* fix bounds check when finalizing build state (#533)Dan Engelbrecht2025-09-301-1/+2
|
* parsing of `zen builds` `--log-progress` option fixed (#530)Dan Engelbrecht2025-09-291-9/+13
|
* builds multi wildcard (#524)Dan Engelbrecht2025-09-291-50/+83
| | | * allow multiple include/exclude wildcards in zen builds command
* remove spurious cpr.h includes (#520)Stefan Boberg2025-09-291-0/+1
|
* list missing block hashes when --verbose is enabled (#512)Dan Engelbrecht2025-09-261-0/+14
|
* limit number of chunks per block (#508)Dan Engelbrecht2025-09-231-2/+109
| | | | | - Improvement: Chunk block generation for `zen oplog-export` and `zen builds upload` command are now limited to max 4000 chunks per block zen builds list-block testing function to check results for recently used blocks in a context
* fetch cloud oplog (#502)Dan Engelbrecht2025-09-221-356/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `zen oplog-download` command to download the oplog body of a cooked output stored in Cloud DDC - Oplog source is specified using one of the following options - `--cloud-url` Cloud artifact URL for oplog - `--host` Base host to resolve download host from - `--override-host` Specific host to use without resolve - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - `--build-id` an Oid in hex form for the source identifier to use - `--yes` suppress conformation query when doing output of a very large oplog to console - `--quiet` suppress all non-essential console output - `--output-path` path to oplog output, extension .json or .cb (compact binary). Default is output to console - `--system-dir` override default system root path - Authentication options - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - `--oidctoken-exe-path` path to OidcToken executable - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Bugfix: `zen print` command now properly outputs very large compact binary objects as json to console
* improve builds download partial logic (#501)Dan Engelbrecht2025-09-221-292/+585
| | | | | | | | | | - Improvement: `zen command` help now uses available horizontal space for output - Improvement: Partial block request analisys has been improved with reduced download size at lower number of requests - Improvement: Validate that `--zen-cache-host` exists and is responsive before begin used - Feature: Options `--allow-partial-block-requests` for `zen builds download` command has been augmented with two new modes, `zencacheonly` and `mixed`. Defaults to `mixed`. - `false` only full block requests allowed - `mixed` multiple partial block ranges requests per block allowed to zen cache, single partial block range request per block to host - `zencacheonly` multiple partial block ranges requests per block allowed to zen cache, only full block requests allowed to host - `true` multiple partial block ranges requests per block allowed to zen cache and host
* fix quoted wildcard options (#500)Dan Engelbrecht2025-09-171-27/+1
| | | * improve command line parsing to automatically remove quotes
* add builds download force option (#498)Dan Engelbrecht2025-09-171-270/+296
| | | * add --force option to builds download command
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-53/+55
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* refactor zen command return value handling (#487)Dan Engelbrecht2025-09-051-77/+90
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use consistent language for command line argument parsing errors - Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error - Improvement: Refactor zen command return codes to conform to valid Linux range (0-255) kSuccess = 0, kOtherError = 1, kBadInput = 2, kOutOfMemory = 16, kOutOfDisk = 17, kAssertError = 70, kHttpOtherClientError = 80, kHttpCantConnectError = 81, kHttpNotFound = 66, // NotFound(404) kHttpUnauthorized = 77, // Unauthorized(401), kHttpSLLError = 82, kHttpForbidden = 83, // Forbidden(403) kHttpTimeout = 84, // RequestTimeout(408) kHttpConflict = 85, // Conflict(409) kHttpNoHost = 86, kHttpOtherServerError = 90, kHttpInternalServerError = 91, // InternalServerError(500) kHttpServiceUnavailable = 69, // ServiceUnavailable(503) kHttpBadGateway = 92, // BadGateway(502) kHttpGatewayTimeout = 93, // GatewayTimeout(504)
* improve console output (#476)Dan Engelbrecht2025-08-261-861/+805
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-2/+2
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-121-3/+6
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* add filtering to builds download (#463)Dan Engelbrecht2025-08-111-54/+210
| | | | | | - Feature: Added wildcard options for `zen build download` - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Remove early wipe of target folder for `zen download` to allow for scavenging useful data
* list build part content (#462)Dan Engelbrecht2025-08-111-291/+666
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* skip query for metada if no blocks are referenced (#460)Dan Engelbrecht2025-08-071-1/+1
|
* retry finalize if we get a "needs" on a blob we did not create (#461)Dan Engelbrecht2025-08-061-2/+4
|
* add retry for failed block metadata upload (#445)Dan Engelbrecht2025-06-191-27/+49
| | | * add retry for failed block metadata upload
* accept Cloud urls without the api/v2/builds/ part (#439)Dan Engelbrecht2025-06-171-25/+45
|
* fix build store range check (#437)Dan Engelbrecht2025-06-161-2/+9
| | | | | * fix range check for blob store fetch * don't try to parse blockdesriptions if empty result is returned * add range to log when fetching blob range fails
* Don't require `--namespace` option when using `zen list-namespaces` command ↵Dan Engelbrecht2025-06-101-19/+44
| | | | (#429)
* restore legacy --url option for builds download (#428)Dan Engelbrecht2025-06-101-1/+7
|
* fix namespace picking up wrong argument (#425)Dan Engelbrecht2025-06-091-1/+1
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-07 04:55:51 +0000