diff options
Diffstat (limited to 'CHANGELOG.md')
| -rw-r--r-- | CHANGELOG.md | 326 |
1 files changed, 326 insertions, 0 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index 38b152f8a..86592d471 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,336 @@ ## +- Improvement: If `zen builds upload` fails to upload metadata for a block with a 404 response (due to race condition from hitting different server) we save and retry metadata upload at end of upload +- Improvement: Updated Oodle libs to 2.9.14 + +## 5.6.13 +- Feature: Added `--sentry-environment` to `zen` and `zenserver` +- Feature: Added `--sentry-debug` to `zen` and `zenserver` +- Feature: Added environment variable parsing for the following options: + - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) + - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` + - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` + - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` + - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment` +- Feature: Added `--output-path` option to `zen version` command to save version information to a file +- Improvement: `zen build --cloud-url` option now accepts URLs without the `api/v2/builds/` part in urls +- Bugfix: Range requests for build blobs that reached end of blob now works correctly +- Bugfix: Gracefully handle a malformed response when querying list of blocks +- Bugfix: Make sure we unregister cache namespaces and buckets when dropping them + +## 5.6.12 +- Bugfix: Don't require `--namespace` option when using `zen list-namespaces` command +- Bugfix: Crash in upload of blobs to Cloud DDC due to buffer range error +- Improvement: Zcache namespace and bucket information now shown on self-hosted dashboard. + +## 5.6.11 +- Bugfix: Reinstate `zen builds --url` option to old behaviour, use `zen builds --cloud-url` for "Cloud Artifact URL" + +## 5.6.10 +- Feature: `zen builds list` now allows `--bucket` to be skipped to allow for multi-bucket search, `bucketRegex` field must be included in the search query +- Feature: `--url` option for `zen builds` command has been reworked to accept a "Cloud Artifact URL", removing the need to specify "host", "namespace" and "bucket" separately +- Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command + - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself +- Feature: New `--sentry-dsn` option for zen command line and zenserver to control Sentry reporting endpoint +- Improvement: Use fixed size block chunking for know encrypted/compressed file types +- Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed +- Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80% +- Improvement: Multithreaded scavenge pass for zen builds download +- Improvement: Optimized check for modified files when verifying state of scavenged paths +- Improvement: Process report now indicates if it is pausing or aborting +- Improvement: Don't report OOD or OOM errors to Sentry when running `zen builds` commands +- Improvement: Use unique temporary file name when calling OidcToken executable to generate auth token +- Improvement: Moved sentry database path to temporary directory for zen commandline +- Bugfix: Zen CLI now initializes Sentry after command line options parsing, so that the options can be propetly taken into account during init +- Bugfix: Revert: `zen builds upload` now use the system temp directory for temporary files leaving the source folder untouched +- Bugfix: Use selected subcommand when displaying help for failed command line options in zen builds +- Bugfix: Make sure we recreate the log file in CAS/cache bucket when creating snapshot at startup causing lost changes. UE-291196 + +## 5.6.9 +- Bugfix: Remove long running exclusive namespace wide locks when dropping buckets or namespaces +- Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 +- Bugfix: Fix stats for memcached entries in disk cache buckets +- Feature: Drop unreachable CAS data during GC pass. UE-291196 +- Improvement: Check available disk space more frequently and trigger GC operation if low water mark is reached +- Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure +- Improvement: `zen builds upload` now use the system temp directory for temporary files leaving the source folder untouched +- Improvement: NoneDecoder::DecompressToStream and NoneDecoder::CompressToStream now uses direct disk I/O +- Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage + +## 5.6.8 +- Feature: Add per bucket cache configuration (Lua options file only) + + cache = { + bucket = { -- This is the default for all namespaces + maxblocksize = 1073741824, + payloadalignment = 4, + memlayer = { + sizethreshold = 1024, + }, + largeobjectthreshold = 2097152, + }, + buckets = { -- Here you can add matching per bucket name (matches accross namespaces) + iostorecompression = { + maxblocksize = 1073741824, + payloadalignment = 4, + memlayer = { + sizethreshold = 1024, + }, + largeobjectthreshold = 2097152, + }, + }, + } +- Feature: `zen oplog-import` and `zen oplog-export` now supports `--oidctoken-exe-path` option +- Feature: Added `--use-sparse-files` option to `zen builds` command improving write performance of large files. Enabled by default. +- Bugfix: Handle invalid plugin config file without terminating +- Bugfix: Gracefully handle errors while running `oplog-mirror` +- Bugfix: Wait for async threads if dispatching of work using ParallellWork throws exception +- Bugfix: Fix crash when batch-fetching chunks and an exception was thrown in non-async code +- Bugfix: Implemented compact binary validation of custom fields +- Bugfix: Fix oplog creation during `zen oplog-import` +- Improvement: `--cache-memlayer-sizethreshold` is now deprecated and has a new name: `--cache-bucket-memlayer-sizethreshold` to line up with per cache bucket configuration +- Improvement: Remove CAS log files at startup when we write full state index +- Improvement: Optimize memory-buffered chunk iteration sizes +- Improvement: Skip command line arguments that are empty or just a single space for `zen` command line tool on Mac/Linux to handle "triple space" problem. UE-273411 +- Improvement: Extend log information when httpsys response sending fails +- Improvement: Log warning when port remappng occurs due to desired port is already in use +- Improvement: Custom CopyFile in `zen builds` command increasing throughput by 50% on Windows and give better progress update + +## 5.6.7 +- Feature: Support for `--log-progress` to output UE style `@progress` log messages has been added to + - `zen builds upload` + - `zen builds download` + - `zen builds diff` + - `zen builds validate-part` + - `zen wipe` +- Feature: `zen builds upload` and `zen builds download` not accepts `--allow-redirect` for Cloud hosts allowing direct upload to/download from backend store (such as S3) +- Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references +- Bugfix: Validate compact binary objects stored in cache before attempting to parse them to avoid crash when data is corrupt +- Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references +- Bugfix: Validate compact binary objects stored in cache before attempting to parse them to avoid crash when data is corrupt +- Bugfix: Don't report redundant error if disk is full when writing GC state or GC log +- Bugfix: Properly handle empty directories when using `zen wipe` command +- Improvement: If garbage collection fails due to out of disk or out of memory we issue a warning instead of reporting an error to Sentry +- Improvement: Improved cache bucket state index write time with ~15% and cache bucket side-car write time with ~20% +- Improvement: Optimize block compact reducing memcpy operations +- Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer +- Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer +- Improvement: Reduce CPU and lock contention during non-critical phases of GC + +## 5.6.6 +- Improvement: Made metadata presentation for cooked output entries more generic +- Improvement: `builds diff` command uses state file if available reducing need to hash content of files +- Improvement: Retry http client requests if we get an unspecified internal error or a bad gateway response +- Improvement: Added Sentry integration to zen command line + - `--no-sentry` disables Sentry integration, defaults to `false` (Sentry is enabled) + - `--sentry-allow-personal-info` to allow personally identifiable information in Sentry reports, disabled by default +- Bugfix: Use proper FindClose call when using fallback when getting file attributes on windows +- Bugfix: Fixed race condition at final chunks when downloading multipart blobs which could lead to corruption and/or crash +- Bugfix: Fixed BigInt conversion error affecting the tree view in the web UI +- Bugfix: Limit retry count when finalizing build part to avoid getting stuck in an infinite loop +- Bugfix: Fixed lua config naming for zenserver `--buildstore-disksizelimit` option +- Feature: `zen builds list-namespaces` subcommand allows listing namespace and bucket information from jupiter builds API. +- Feature: New `zen wipe` command for fast cleaning of directories, it will not remove the directory itself, only the content + - `--directory` - path to directory to wipe, if the directory does not exist or is empty, no action will be taken + - `--keep-readonly` - skip removal of read-only files found in directory, defaults to `true`, set to `false` to remove read-only files + - `--quiet` - reduce output to console, defaults to `false` + - `--dryrun` - simulate the wipe without removing anything, defaults to `false` + - `--yes` - skips prompt to confirm wipe of directory + - `--plain-progress` - show progress using plain output + - `--verbose` - enable verbose console output + - `--boost-workers` - increase the number of worker threads, may cause computer to be less responsive, defaults to `false` +- Feature: **EXPERIMENTAL** New `--plugins-config` option to load plugins based on transport-sdk. + - It accepts a json of format `[{"name": "%path_to_dll%", "%opt1%": "%val1%", ...}, ]`. + +## 5.6.5 +- Bugfix: `zen builds` multipart download of large chunks could result in crash +- Bugfix: Fixed invalid hash check on messages from Jupiter when doing range http requests +- Bugfix: Handle temporary locked files more gracefully (Crowdstrike workaround) +- Bugfix: Avoid integer overflow in web UI causing numbers to show up as negative + +## 5.6.4 +- Improvement: `zen builds` now scavenges previous download locations for data to reduce download size, enabled by default, disable with `--enable-scavenge=false` +- Improvement: `zen builds upload` Fixed and improved layout of end of run stats output +- Improvement: `zen builds` Reduced clutter and added timing to some logging entries when `--verbose` is enabled +- Bugfix: If both `--host` and `--override-host` was given to a `zen builds` command, `-override-host` no host would be resolved +- Bugfix: Failing to rename a file during download sometimes reported an error when it succeeded when retrying + +## 5.6.3 +- Feature: `zen oplog-export`, `zen oplog-import` for `--url` (cloud) and `--builds` (builds) option now has `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token +- Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB. +- Bugfix: Restore Mac minver back to 12.5 +- Bugfix: Fixing bug where some imports were shown as a blank string in the web UI +- Improvement: Hide progress ETA number until at least 5% of work is done to reduce misleading processing times. UE-256121 +- Improvement: Dynamically adjust console progress output based on console width. UE-256126 +- Improvement: Usability improvements to oplog search in web UI. Case insensitve, short string search, reset search. + +## 5.6.2 +- Bugfix: Changed Mac minver from 12.5 to 14.0, and removed `_LIBCPP_DISABLE_AVAILABILITY` as a define on Mac due to executable incompatibility issues +- Bugfix: Fixed missing trailing quote when converting binary data from compact binary to json +- **EXPERIMENTAL** `zen builds` + - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token + - Bugfix: Fixed upload failure on Mac/Linux if `--zen-folder-path` was not given. UE-265170 + - Feature: `zen builds upload` command has new option `--find-max-block-count` to control how many blocks we search for reuse. + - Improvement: Bumped the default number of blocks to search during upload to 10000 (from 5000). + +## 5.6.1 +- Bugfix: GetModificationTickFromPath and CopyFile now works correctly on Windows/Mac +- Bugfix: Handling of quotes and quotes with leading backslash for command line parsing - UE-231677 +- Improvement: When logging with a epoch time prefix, the milliseconds/fraction is now correct. We now also set the epoch to the process spawn time rather than the time when the logger is created +- Improvement: Reduce concurrent disk I/O during GC for cache buckets +- **EXPERIMENTAL** `zen builds` + - Feature: `zen builds list` command has new options + - `--query-path` - path to a .json (json format) or .cbo (compact binary object format) with the search query to use + - `--result-path` - path to a .json (json format) or .cbo (compact binary object format) to write output result to, if omitted json format will be output to console + - Feature: New `/builds` endpoint for caching build blobs and blob metadata + - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs + - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs + - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs + - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs + - Feature: zen: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata + - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data + - Feature: zen `--boost-workers` option to builds `upload`, `download` and `validate-part` that will increase the number of worker threads, may cause computer to be less responsive + - Feature: zen `--cache-prime-only` that uploads referenced data from a part to `--zen-cache-host` if it is not already present. Target folder will be untouched. + - Feature: zen: `--zen-folder-path` added to `builds` command, `list`, `upload`, `download`, `fetch-blob`, `validate-part` to control where `.zen` folder is placed and named + - Feature: Added `--host` option to use Jupiters list of cloud host and zen servers to resolve best hosts + - Feature: Use local zenserver as builds cache if it has the `builds` service enabled and `--cloud-discovery-host` is provided and no remote zenserver cache hosts can be found + - Improvement: Do partial requests of blocks if not all of the block is needed + - Improvement: Better progress/statistics on upload and download + - Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run + - Improvement: Faster abort during stream compression + - Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write + - Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails + - Improvement: Do put/get build and find blocks while scanning local folder when uploading + - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates + - Improvement: Emit build name to console output when downloading a build + - Improvement: Reduced memory usage during upload and part upload validation + - Improvement: Reduced I/O usage during upload and download + - Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild + - Improvement: More trace scopes for build upload operations + - Improvement: Progress bar automatically switches to plain mode when stdout is not a console + - Improvement: Progress bar is much more efficient on Windows (switched away from printf) + - Improvement: Improved stats output at end of upload and download operations + - Improvement: Reduced disk I/O when writing out chunk blocks during download + - Improvement: Collapse consecutive ranges when reading/writing data from local cache state + - Improvement: If a chunk or block write operation results in more than one completed chunk sequence, do the additional verifications as async work + - Improvement: Improved error reporting when async tasks fail + - Improvement: At end of build upload we post statistics to the Jupiter build stats endpoint: + - `totalSize` + - `reusedRatio` + - `reusedBlockCount` + - `reusedBlockByteCount` + - `newBlockCount` + - `newBlockByteCount` + - `uploadedCount` + - `uploadedByteCount` + - `elapsedTimeSec` + - `uploadedBytesPerSec` + - Improvement: Allow cook metadata to be browsed in the web UI + - Improvement: ELF and MachO executable files are no longer chunked + - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) + - Improvement: Only check known files from remote state when downloading to a target folder with no local state file + - Improvement: Don't move existing local to cache and back if they are untouched + - Improvement: Faster cleaning of directories + - Improvement: Faster initial scanning of local state + - Improvement: Added `--override-host` option as a replacement for `--url` (`--url` still works, but `--override-host` is preferred) + - Improvement: Output Build and Build Part information to console during `builds download` + - Improvement: Output zen executable path and version at start of `builds` commands + - Bugfix: Strip path delimiter at end of string in StringToPath + - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase + - Bugfix: Fixed crash during download when trying to write outside a file range + - Bugfix: MacOS / Linux zen build download now works correctly + - Bugfix: Env auth parsing blocked parsing OAuth and OpenId options + - Bugfix: Long file paths now works correctly on Windows + - Bugfix: Validate that we can read input files correctly for files that are not chunked + +## 5.6.0 +- Feature: Added support for `--trace`, `--tracehost` and `--tracefile` options to zen CLI command +- Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat +- Improvement: Instrumented `zen builds download` command code so we get more useful Insights output +- **EXPERIMENTAL** `zen builds` + - Improvement: Better error reporting and faster exit on error + - Improvement: Downloads without `--clean` now preserves untracked files in target folder + - Improvement: Reduce write operations for duplicate files in dowloads + - Improvement: Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed + - Improvement: Do streaming decompression of loose blobs to improve memory and I/O performance + - Improvement: Validate hash of decompressed data inline with streaming decompression + - Improvement: Do streaming compression of large blobs to improve memory and I/O performance + - Feature: Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete + - Feature: Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete + +## 5.5.20 +- Bugfix: Fix bug causing crash if large file was exact multiple of 256KB when using zen builds command +- Bugfix: Fix authentication error affecting Jupiter client that manifests as 'bad function call' error UE-252123 + +## 5.5.19 +- Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API + - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) + - `builds upload` upload a folder to Cloud Build API + - `--local-path` source folder to upload + - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given + - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated + - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated + - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` + - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set + - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set + - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks + - `--allow-multipart` enable usage of multi-part http upload requests + - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` + - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) + - `--local-path` target folder to download to + - `--build-id` an Oid in hex form for the Build identifier to use + - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` + - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` + - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) + - `--allow-multipart` enable usage of multi-part http download reqeusts + - `builds diff` download a folder from Cloud Build API + - `--local-path` target folder to download to + - `--compare-path` folder to compare target with + - `--only-chunked` compare only files that would be chunked + - `builds fetch-blob` fetch and validate a blob from remote store + - `--build-id` an Oid in hex form for the Build identifier to use + - `--blob-hash` an IoHash in hex form identifying the blob to download + - `builds validate part` fetch a build part and validate all referenced attachments + - `--build-id` an Oid in hex form for the Build identifier to use + - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` + - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` + - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations + - `--local-path` source folder to upload + - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) + - `--url` Cloud Builds URL + - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake + - `--namespace` Builds Storage namespace + - `--bucket` Builds Storage bucket + - Authentication options for Cloud Build API + - Auth token + - `--access-token` http auth Cloud Storage access token + - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token + - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token + - OpenId authentication + - `--openid-provider-name` Open ID provider name + - `--openid-provider-url` Open ID provider url + - `--openid-client-id`Open ID client id + - `--openid-refresh-token` Open ID refresh token + - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials + - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials + - OAuth authentication + - `--oauth-url` OAuth provier url + - `--oauth-clientid` OAuth client id + - `--oauth-clientsecret` OAuth client secret + - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) + - `--storage-path` path to folder to store builds data + - `--json-metadata` enable json output in store for all compact binary objects (off by default) + - Output options for all builds commands + - `--plain-progress` use plain line-by-line progress output + - `--verbose` + +## 5.5.18 - Bugfix: Fix parsing of workspace options in Lua config - Bugfix: Add missing Lua option for option `--gc-projectstore-duration-seconds` - Bugfix: Add missing Lua mapping option to `--statsd` command line option - Bugfix: Verify that chunking is allowed before chunking loose files duriung oplog export - Bugfix: Fix oplog target url for oplog export to remote zenserver - Bugfix: Handle workspace share paths enclosed in quotes and ending with backslash UE-231677 +- Bugfix: Strip leading path separator when creating workspace shares ## 5.5.17 - Improvement: Batch fetch record attachments when appropriate |