## - Bugfix: Fix sentry using wrong folder path when data path contains non-ascii characters UE-210530 - Bugfix: Get raw size for compressed chunks correctly for `/prj/{project}/oplog/{log}/chunkinfos` - Bugfix: Fix log of Success/Failure for oplog import - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/chunkinfos` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `rawhash` and `rawsize` (translates to `?fieldnames=id,rawhash,rawsize`) - Use `?fieldnames=*` to get all the fields - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/files` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `clientpath` and `serverpath` (translates to `?fieldnames=id,clientpath,serverpath`), `filter=client` only applies if `fieldnames` is not given as a parameter - Use `?fieldnames=*` to get all the fields - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()` - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import - Improvement: Optimize `CompositeBuffer::ViewOrCopyRange` speeding up compressed buffer headers by not materializing entire source buffer - Improvement: Optimize `CompressedBuffer::GetRange()` with new `CompressedBuffer::ReadHeader()` that does one less read from source data resulting in a 30% perf increase. - Improvement: Validate lengths of chunks fetched from CAS/Cache store, if full chunk can not be retrieved, treat it as missing ## 5.4.2 - Bugfix: Shared memory for zenserver state may hang around after all zenserver processes exit - make sure we find a valid entry in `zen up` before bailing - Bugfix: Httpasio only call listen() once - Bugfix: Make sure exception do not leak out of async (worker thread pool) work and make sure we always wait for completion of all work - Bugfix: Limit number of headers parsed to 127 as that is the maximum supported by Zen - Bugfix: Don't capture for loop variables by reference when executing async code - Bugfix: Make sure WriteFile() does not leave incomplete files - Bugfix: Use TemporaryFile and MoveTemporaryIntoPlace to avoid leaving partial files on error - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Bugfix: Fix startup issues where data path contains non-ascii characters UE-210530 - Feature: Added option `--access-token-path` to `zen oplog-export` and `zen-oplog-import` enabling it to read a cloud access token from a json file - Feature: Added support for generating yaml format responses via Accept: yaml or .yaml suffix - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Httpasio explicitly close acceptor sockets - Improvement: Httpasio add retry for desired port - Improvement: Move structuredcachestore tests to zenstore-test - Improvement: CompositeBuffer::Mid no longer materializes segment buffers - Improvement: Don't materialize entire files when hashing if it is a large file - Improvement: Added context to some http.sys warnings caused by HTTP API error returns - Improvement: Improved logging for block store GCV2 operations - Improvement: Added more tests for GCV2 (added GCV2 versions of existing GCV2 tests) - Improvement: Add disk cache to reading and writing blocks when moving data in GCV2 - Improvement: Cleaned up some asio server state machine details (minor) - Improvement: Added support for request tracing when using asio path (use `--log-trace=http_requests` to enable) - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export - Improvement: Make sure zenserver reacts and exist on SIGTERM signal - Improvement: Retry to create the .lock file at startup to avoid failing launch due to race condition with UE - Improvement: Add CompressedBuffer::GetRange that references source data rather than make a memory copy - Improvement: Delay exiting due to no sponsor processes by one second to handle race conditions - Improvement: Safer IsProcessRunning check - Improvement: Make sure we can RequestApplicationExit safely from any thread - Improvement: Check if a block exists in remote store before considering it for reuse in oplog export - Improvement: Add limit to the number of times we attempt to finalize and exported oplog - Improvement: Switch to large thread pool when executing oplog export/import - Improvement: Clean up reporting of missing attachments in oplog export/import - Improvement: Remove double-reporting of abort reason for oplog export/import - Improvement: Add support to filter projectstore `entries` request using the `fieldfilter` where the wanted fields are comma (,) delimited - Improvement: Add support for responding with compressed payloads for projectstore `entries` requests by adding AcceptType `compressed-binary` to the request header - Improvement: Add support for responding with compressed payloads for projectstore `files` requests by adding AcceptType `compressed-binary` to the request header - Improvement: Add support for responding with compressed payloads for projectstore `chunkinfo` requests by adding AcceptType `compressed-binary` to the request header - Removed: `--cache-reference-cache-enabled` option has been removed along with the implementation for reference caching in disk cache ## 5.4.1 - Feature: Added `--copy-log`, `--copy-cache-log` and `copy-http-log` option to zen `logs` command to copy logs from a local running zenserver instance - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Improvement: Return system error code on exception throw in zen command - Improvement: Clean up HttpClient::Response::ErrorMessage to remove redundant ": " - Improvement: respond with BadRequest result instead of throwing exception and causing a Sentry report on bad request input - Improvement: Speed up oplog export by fetching/compressing big attachments on demand - Improvement: Speed up oplog export by batch-fetcing small attachments - Improvement: Speed up oplog import by batching writes of oplog ops - Improvement: Tweak oplog export default block size and embed size limit - Improvement: Add more messaging and progress during oplog import/export - Improvement: Large loose file attachments are now saved to temp files after compressing during oplog export to reduce memory pressure - Improvement: Keep track of added ops during GCV2 instead of rescanning full oplog when added ops are detected - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk - Bugfix: Only try to traverse an objectstore bucket if it really exists - Bugfix: Actually throw exception if we can't parse the JobId when starting async job - Bugfix: Implement two listening sockets in ASIO (ipv4+ipv6) when either we start with `--http-forceloopback` or we resort to that mode because of a failure to bind to the "any" address ## 5.4.0 - Improvement: Add details when reading from BasicFile fails (number of bytes read is not expected size) - Bugfix: No longer outputs illegal characters when Jupiter responds with an error payload in compact binary format ## 0.2.39 - Feature: add `--ignore-missing-attachments` to `oplog-import` command - Feature: add `--ignore-missing-attachments` to `oplog-export` command - Improvement: Removed use of in stats, for better performance (runtime as well as build) - Improvement: Separated cache RPC handling code from general structured cache HTTP code - Improvement: Get more detailed information on Jupiter upstream errors - Improvement: Improved performance when saving oplog via oplog import command - Improvement: Add more feedback and progress information when executing oplog import/export - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging - Improvement: Reduce oplog block size to 64MB to reduce amount of redundant chunks to download - Bugfix: RPC recording would not release memory as early as intended which resulted in memory buildup during long recording sessions. Previously certain memory was only released when recording stopped, now it gets released immediately when a segment is complete and written to disk. - Bugfix: File log format now contains dates again (PR #631) - Bugfix: Jobqueue - Allow multiple threads to report progress/messages (oplog import/export) - Bugfix: Jobqueue - Add AbortReason and properly propagate error when running async command (oplog import/export) - Bugfix: Make sure to write the correct data in BasicFileWriter when writing items that are not a multiple of the buffer size - Bugfix: Restructured RPC recorder to use a MPSC queue instead of shared datastructures to fix a data race condition ## 0.2.38 - Bugfix: Cache RPC recording would drop data when it reached 4GB of inline chunk data in a segment - Bugfix: Fixed thread safety issues in RPC recorder v2 - Bugfix: `IoBuffer::Materialize` would leak memory for small buffers - Bugfix: Fix crash bug when trying to inspect non-open block file in GC - Bugfix: Fixed up code so we can build everything even when trace support is disabled - Bugfix: Make sure we initialize the pattern of FileSink before it is added as a usable logger - Bugfix: Fixed capture of loop-local variables in lambdas for GCv2 - Bugfix: Various minor TSAN/ASAN fixes (see PR #622) - Improvement: Cache RPC replay can now process partial recordings by recovering metadata from available files - Improvement: Cache RPC recording now limits duration of individual segments to 1h - Improvement: Made RPC replay command line parsing more robust by ensuring at least one processing thread is in use - Improvement: Windows executables are now signed with official cert when creating a release - Improvement: Each block in block store that is rewritten will now be logged for better feedback ## 0.2.37 - Bugfix: ShutdownLogging code would throw an exception if it was called before everything had been initialised properly - Bugfix: Reorder shutdown to avoid crash due to late async log messages (spdlog workaround) - Bugfix: Correctly calculate peak disk write size in GC status message - Bugfix: Skip invalid chunks in block store GC when moving existing chunks - Bugfix: Don't use copy of Payloads array when fetching memcached payload in GC - Bugfix: Make sure IoBuffer is a valid null-buffer after move operation - Improvement: Adjusted and added some trace scopes ## 0.2.36 - Feature: Added xmake task `updatefrontend` which updates the zip file containing the frontend html (`/src/zenserver/frontend/html.zip`) - Feature: Added `--powercycle` option to zenserver which causes it do shut down immediately after initialization is completed. This is useful for profiling startup/shutdown primarily but could also be useful for some kinds of validation/state upgrade scenarios - Feature: New endpoint `/admin/gc-stop` to cancel a running garbage collect operation - Feature: Added `zen gc-stop` command to cancel a running garbage collect operation - Feature: Added ability to configure logger verbosity on the command line. You can now use `--log-debug=http_requests` to configure the `http_requests` logger to DEBUG level. The provided options are `--log-trace`, `--log-debug`, `--log-info`, `--log-warn`, `--log-error`, `--log-critical`, `--log-off` and each accepts a comma-separated list of logger names to apply the threshold to. - Bugfix: Fix sentry host name where last character of name was being truncated - Bugfix: GCv2 - make sure to discover all projects and oplogs before checking for expired data - Bugfix: Fix sync of log position and state log when writing cas index snapshot - Bugfix: Make sure we can override flags to "false" when running `zen gc` commmand - `smallobjects`, `skipcid`, `skipdelete`, `verbose` - Bugfix: fixed file log timestamp format so the milliseconds are appended after the time not the date - Bugfix: Shut down thread pools earlier so worker threads have a chance to terminate before main thread calls `atexit()` - Bugfix: Use correct lookup index when checking for memcached buffer when finding references in diskcache GC - Bugfix: CasContainerStrategy::ReadIndexFile issue could cause CAS items to not be found after a shutdown/restart cycle - Bugfix: Make sure we don't hold the namespace bucket lock when we create buckets to avoid deadlock - Bugfix: Make sure that PathFromHandle don't hide true error when throwing exceptions - Bugfix: Allow attachments that contains a raw size of zero - Improvement: The frontend html content is no longer appended at the end of the executable which prevented signing, instead it is compiled in from the `/src/zenserver/frontend/html.zip` archive - Improvement: MacOS now does ad-hoc code signing by default when issuing `xmake bundle`, signing with proper cert is done on CI builds - Improvement: Updated branding to be consistent with current working name ("Unreal Zen Storage Server" etc) - Improvement: GcScheduler will now cancel any running GC when it shuts down. - Current GC is rather limited in *when* it reacts to cancel of GC. GCv2 is more responsive. - Improvement: Cache metadata snapshot (`*.uidx`/`.zen_manifest` and now `*.meta`) files are read and written in a streaming fashion instead of all-at-once to/from memory like before. This eliminates some spiky memory usage patterns during garbage collection and also improves overall performance considerably. - Improvement: `zen copy-state` command now utilizes block cloning where possible (i.e on ReFS volumes) for near-instant snapshots - Improvement: Added more trace scopes for GCv2 - Improvement: Use two global worker thread pools instead of ad-hoc creation of worker pools - Improvement: GCv2: Use separate PreCache step to improve concurrency when checking references - Improvement: GCv2: Improved verbose logging - Improvement: GCv2: Sort chunks to read by block/offset when finding references - Improvement: GCv2: Exit as soon as no more unreferenced items are left - Improvement: Reduce memory usage in GC and diskbucket flush - Improvement: Added command line to trace initialization (Windows only for now) - Improvement: Added a `{project}/oplog/{log}/chunkinfos` endpoint that can be used for getting all chunk info within an oplog in batch - Improvement: Reserve vector sizes in GCv2 to reduce reallocations - Improvement: Set min/max load factor for cachedisk/compactcas/filecas indexes to reduce memory footprint - Improvement: Added context (upstream host name) to Zen upstream resolve error message - Improvement: Make a more accurate estimation of memory usage for in-memory cache values - Improvement: Added detailed debug logging for pluggable transports - Improvement: Improved formatting of multi-line logging. Each line is now indented to line up with the initial line to make reading the output easier - Improvement: Refactor memory cache for faster trimming and correct trim reporting - Improvement: Added trace scopes for memory cache trimming - Improvement: Pass lock scope to helper functions to clarify locking rules - Improvement: Block flush and gc operations for a bucket that is not yet initialized - Improvement: Add ZenCacheDiskLayer::GetOrCreateBucket to avoid code duplication - Improvement: Scrub operation now validates compressed buffer hashes in filecas storage (used for large chunks) - Improvement: Added `--dry`, `--no-gc` and `--no-cas` options to `zen scrub` command - Improvement: Implemented oplog scrubbing (previously was a no-op) - Improvement: Implemented support for running scrubbint at startup with --scrub= ## 0.2.35 - Bugfix: Fix timeout calculation for semtimedop call - Bugfix: Fix NameEvent test to avoid race condition - Bugfix: Fix BlockingQueue asserts - Bugfix: Catch exceptions in WorkerThreadPool when running single-threaded - Bugfix: Improved block cloning copy argument validation, to properly catching the case where source or target trees overlap - Feature: Adding a file named `root_manifest.ignore_schema_mismatch` in the root of zenserver data dir prevents wipe of data when schema mismatches - Feature: Added `zen run` command which can be used to run a stress test or benchmark repeatedly while redirecting output and other state to separate subdirectories - Example usage: `zen run -n 10 -- zenserver-test` will run the `zenserver-test` command 10 times - Example usage: `zen run -n 10 -- zenserver-test --ts=core.assert` will` run zenserver-test 10 times (testing only the core.assert test suite) - Example usage: `zen run --time 600 --basepath=d:\test_dir\test1 -- zenserver-test` keeps spawning new instances for 10 minutes (600 seconds). Each run will execute in a separate subdirectory in `d:\test_dir\test1\` where stdout will be captured alongside any data generated by the executed command - Feature: Added new options to zenserver for GC V2 - `--gc-compactblock-threshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90% - `--gc-verbose` GCV2 - enable more verbose output when running a GC pass - Feature: Added new options to `zen gc` command for GC V2 - `--compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90% - `--verbose` GCV2 - enable more verbose output when running a GC pass - Feature: Added new parameters for endpoint `admin/gc` (PUT) - `compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90% - `verbose` GCV2 - enable more verbose output when running a GC pass - Improvement: Removed `zen runtests` command since it's no longer useful - Improvement: Simplified zenserver-test code by implementing dynamic port assignment, and also implemented transparent handling of port relocation for increased test robustness against environmental differences and socket lifetime 'noise' - Improvement: Refactor GCV2 so GcReferencer::RemoveExpiredData returns a store compactor, moving out the actual disk work from deleting items in the index. - Improvement: Refactor GCV2 GcResult to reuse GcCompactStoreStats and GcStats - Improvement: Make GCV2 Compacting of stores non-parallel to not eat all the disk I/O when running GC - Improvement: Added `ZEN_ASSERT_FORMAT` implementation in `zencore/assertfmt.h` for better logging of errors. Introduced it into compact binary building code which had some existing use cases. ## 0.2.34 - Bumped zenserver data schema (to '5') to wipe corrupted state caused by version 0.2.31/0.2.32 - Bugfix: Fix hang on ZenServerInstance shutdown on Mac/Linux - Improvement: Event implementation now uses `std::atomic_bool` instead of `volatile bool` for correctness - Improvement: Removed dependency on cxxopts exception types, to enable use of library versions >3.0.0 - Improvement: Posix event implementation is now more robust and works around apparent condition variable issue - Improvement: `ProcessHandle::Wait` no longer returns without calling `kill` - Improvement: We now set Sentry to `@` if personal information is allowed to be sent - Improvement: Make object store endpoint S3 compatible. ## 0.2.33 - Bugfix: Fix index out of bounds in CacheBucket::CompactState - Bugfix: Implement != operator for DiskLocation to avoid comparing uninitialized data - Improvement: Shrink data structures to fit after CacheBucket::CompactReferences ## 0.2.32 - Feature: Writes a `gc.log` with settings and detailed result after each GC execution (version 2 only) - Bugfix: Fix memory stop in disk cache bucket ReadManifest - Improvement: Package dependency clean-ups ## 0.2.31 - Feature: New parameter for endpoint `admin/gc` (GET) `details=true` which gives details stats on GC operation when using GC V2 - Feature: New options for zen command `gc-status` - `--details` that enables the detailed output from the last GC operation when using GC V2 - Feature: Add new `copy-state` zen command which copies a zenserver data directory state excluding bulk data for analysis purposes - Feature: New garbage collection implementation, still in evaluation mode. Enabled by `--gc-v2` command line option - Feature: zen `print` command can now print the `gc.log` file - Feature: New option for zenserver - `--http-forceloopback` which forces opening of the server http server using loopback (local) connection only (UE-199776) - Bugfix: Build script now sets up arch properly when running tests on MacOS - Bugfix: Corrected initialization of block store MaxBlockCount - Bugfix: Corrected total disk size usage in block store - Bugfix: Server log files were using the wrong log line prefix due to a mistake when consolidating logging setup code - Bugfix: Sponsor processes are now registered synchronously at startup, to close potential race condition in very short-lived subprocesses such as the automated tests - Bugfix: Fix error in GC when reclaiming disk reserve is not enough to accommodate the new block - Bugfix: If a directory is deleted while we try to traverse it, skip it and continue - Improvement: Multithread init and flush of cache bucket for faster startup and exit - Improvement: Renamed BlockStoreCompactState::AddBlock to BlockStoreCompactState::IncludeBlock for clarity - Improvement: Added tests for BlockStore::CompactBlocks - Improvement: Reduce memory consumption in cache disk layer - Improvement: Refactored logging so that spdlog details are hidden from the majority of client code - Improvement: Use GC reserve when writing index/manifest for a disk cache bucket when disk is low when available - Improvement: Demote errors to warning for issues that are not critical and we handle gracefully - Improvement: Treat more out of memory errors from windows as Out Of Memory errors - Improvement: Factored out some compiler / platform definitions into standalone `zenbase` header-only library, along with header-only helpers which can be used standalone. This is intended to support out-of-tree code like pluggable transports etc but also provides more fine grained dependencies - Improvement: Replaced use of openssl on Windows with bcrypt, which reduces executable size by some 40% - Improvement: We no longer put cache entries into the memory cache on Put, only on Get - Improvement: Dedicated servers now have a different heuristic for deciding when to use standalone files to store cache records/values in the structured cache disk layer. This helps performance on heavy-traffic servers - Improvement: Reduced memory footprint of cache index by 10% or so by limiting the maximum number of entries in a bucket to 2^32 (was 2^64) ## 0.2.30 - Bugfix: Block sending error reports from sentry_sink to Sentry unless the log is actually an error log - Bugfix: Make sure we have an exclusive lock in CacheBucket::CollectGarbage when removing standalone entries from the index - Bugfix: Fixed problem with missing session/request context in cache record PUT operations ## 0.2.29 - Feature: Add `skipdelete` parameter to `admin/gc` endpoint to do a dry run of GC - Feature: Add `--skipdelete` option to `zen gc` command to do a dry run of GC - Feature: New endpoint `/admin/flush ` to flush all storage - CAS, Cache and ProjectStore - Feature: New command `zen flush` to flush all storage - CAS, Cache and ProjectStore - Feature: Added `--cache-memlayer-sizethreshold` option to zenserver to control at which size cache entries get cached in memory - Bugfix: Filter cache keys against set of expired cache keys, not against set of CAS keys to keep - Bugfix: Fix implementation when claiming GC reserve during GC - Bugfix: Catch exceptions when processing requests with asio http and log errors - Improvement: Command `zen gc-status` now gives details about storage, when last GC occured, how long until next GC etc - Improvement: New rotating file logger that keeps on running regardless of errors to avoid spamming error reports on OOM or OOD - Improvement: Removed HttpCidStore (was already deprecated) - Improvement: Optimized cases in CompactBinary reader where we would measure a variable integer twice - Changed: Cache access and write log are disabled by default - Changed: We no longer purge out block location for missing blocks to allow testing/analisys of snapshots of server states without copying full set of data - Changed: Merged cache memory layer with cache disk layer to reduce memory and cpu overhead ## 0.2.28 - Feature: Implemented initial and experimental support for pluggable transports via (`--http=plugin`) - Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC - disabled by default, enable with `--cache-reference-cache-enabled` - Feature: Added `--clean` command line option which can be used to wipe all server state at startup (useful for testing/benchmarking) - Feature: Added `--dry` command line option to `zen rpc-record-replay` to allow analysis of recordings without making changes to any server - Changed: The default port for zenserver has been changed from 1337 to 8558 - Bugfix: GC logging now correctly reports used/free disk space in log message - Bugfix: Fixed calculation of cache memory layer total size - Improvement: Implemented a new RPC recording strategy (aka v2) to make long-running recordings use less resources and perform better over time - Recordings are split into segments to limit the amount of data and number of files in each segment directory. Each segment is independent to allow future disk space management improvements and partial analysis/replay of recordings - The new recording strategy also records the client session id for each entry to enable better traffic analysis - Improvement: Rewrite the state_marker file at startup to make sure we have write access to the data directory - Improvement: Faster reading of project store oplogs - Improvement: Faster collection of referenced CId content in project store - Improvement: Also reject bad bucket GET operations to prevent the buckets from being created on disk - Improvement: GC will now skip a lightweight GC if a full GC is due to run within the next lightweight GC interval - Improvement: when dedicated mode is enabled via --dedicated or server.dedicated then we tune http.sys server settings to be more suitable for a shared server. Initially we tune two things: - the thread pool used to service I/O requests allows a larger number of threads to be created when needed. The minimum thread count is unchanged but in dedicated server mode we double the maximum number of threads allowed - the http.sys request queue length (HttpServerQueueLengthProperty) is increased to 50,000 in dedicated mode. The regular default is 1,000. A larger queue means the server will deal with small intermittent stalls (for example due to GC locking) even at higher loads like 100k req/s without rejecting requests via HTTP 503 results - Removed: Removed legacy compute interface (will be replaced with new implementation in the future) ## 0.2.27 - Bugfix: Remove double counting of memory usage in memcachelayer - Bugfix: Make sure we don't busy loop if Garbage Collection fails - Bugfix: If we can't check if a project/oplog is expired via the marker file, assume it is not expired - Improvement: added rejection of known bad buckets in structured cache (caused by inappropriate client use of buckets) - buckets consisting of 32 characters in hexadecimal are rejected in put operations - buckets are also rejected at bucket discovery time, and any bad directories are deleted - Improvement: restructured zenserver project to improve maintainability (PR#442) ## 0.2.26 - Feature: Limit the size ZenCacheMemoryLayer may use - `--cache-memlayer-targetfootprint` option to set which size (in bytes) it should be limited to, zero to have it unbounded - `--cache-memlayer-maxage` option to set how long (in seconds) cache items should be kept in the memory cache - Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store - Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false" - Add `--skipcid` option to `zen gc` command, defaults to false - Add `--gc-lightweight-interval-seconds` option to zenserver - Bugfix: Correctly calculate the total number of RPC ops in the stats page for structured cache - Bugfix: Change "chunks" title to "count" for RPC chunk requests in stats page for structured cache - Bugfix: Sentry username string no longer includes the trailing NUL - Bugfix: Fix scrub messing up payload and access time in disk cache bucket when compacting index - Bugfix: Cache bucket index of chunks in block store could get corrupted after a GC - Bugfix: Probe disk for existsing block file before writing a new block - Bugfix: IoBufferBuilder::ReadFromFileMaybe did not propagate content type to a new IoBuffer - Bugfix: IoBufferBuilder::ReadFromFileMaybe Linux/MacOS pread success/error condition check was incorrect - Bugfix: Memory cache layer could end up holding references to file handles between GC runs, delaying file deletion - Improvement: Split up disk cache bucket index into hash lookup and payload array to improve performance - Improvement: Reserve space up front for compact binary output when saving cache bucket manifest to improve performance - Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting - Change format for faster read/write and reduced size on disk - Don't lock index while writing manifest to disk - Skip garbage collect if we are currently in a Flush operation - BlockStore::Flush no longer terminates currently writing block - Garbage collect references to currently writing block but keep the block as new data may be added - Fix BlockStore::Prune used disk space calculation - Don't materialize data in filecas when we just need the size - Don't lock entire disk cache layer when doing GatherReferences/CollectGarbage - Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to requester without reporting an error to Sentry - Improvement: If creating bucket fails when storing an item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry - Improvement: Make an explicit flush of the active block written to in blockstore flush - Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit - Improvement: http.sys I/O completion handler no longer holds transaction lock while enqueueing new requests. This eliminates some lock contention and improves latency/throughput for certain types of requests - Improvement: date logging in GC no longer emits extraneous newlines - Improvement: named main thread and thread handle cache thread - Improvement: removed websockets support as it is not used and likely won't be used in the future - Improvement: added `--quiet` command line option which can be used to suppress all stdout logging ## 0.2.25 - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint - Improvement: GarbageCollect for ZenCacheMemoryLayer now respects `--gc-cache-duration-seconds` - Improvement: HttpSys: When a response fails, we now include more information including metadata about the contents of the reponse - Improvement: Flush current data block to disk when switching to a new block - Improvement: Handle cache RPCs synchronously instead of dispatching to async worker threads when there is no upstream server - Improvement: Endpoint for cache upstream stats improved - added `active` and `worker_threads` reflects actual number of threads - Improvement: Cache upstream only starts worker threads once at least one Endpoint is registered - Improvement: http.sys only starts async work threads when there is async work to do (which is often never if there is no upstream) ## 0.2.24 - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `cacheenablewritelog`=`true`/`false` parameter to control cache write logging - `cacheenableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level - Feature: Add endpoint for controlling Insights tracing - GET `/admin/trace` to query if tracing is currently running or not - POST `/admin/trace/start` to start tracing - `host=` start tracing to a trace server at ip `` - `file=` start tracing to file at path `` - POST `/admin/trace/stop` stop the currently running trace - Feature: Add `zen trace` command to control Insights tracing - `zen trace` to show the status of tracing ("enabled" or "enabled") - `zen trace --host=` start tracing to a trace server at ip `` - `zen trace --file=` start tracing to file at path `` - `zen trace --stop` stop the currently running trace - Feature: Implemented virtual file system (VFS) support for debugging and introspection purposes - `zen vfs mount ` will initialize a virtual file system at the specified mount point. The mount point should ideally not exist already as the server can delete the entirety of it at exit or in other situations. Within the mounted tree you will find directories which allow you to enumerate contents of DDC and the project store - `zen vfs unmount` will stop the VFS - `zen vfs info` can be used to check the status of the VFS - Bugfix: Do controlled shut down order of zenserver and catch exceptions thrown during shutdown - Bugfix: Make sure we don't throw exceptions when reporting errors to Sentry - Improvement: Add names to background jobs for easier debugging - Improvement: Background jobs now temporarily sets thread name to background job name while executing - Improvement: Background jobs tracks worker thread id used while executing - Improvement: `xmake sln` can now be used on Mac as well to generate project files - Improvement: http.sys request queues are named to make it easier to find performance counters in Performance Monitor and such - Improvement: http.sys - if request rate is too high then rejected requests will get a 503 response instead of a dropped connection ## 0.2.23 - Bugfix: Respect result from FinalizeRef in Jupiter oplog upload where it requests missing attachments - Improvement: Increase timeout when doing import/export of oplogs to jupiter to 30 min per request - Improvement: Better logging/progress report on oplog export - Improvement: Ignore OOM errors in spdlog, just drop the error since we can't do anything useful if we run out of memory here - Improvement: Try to catch any exceptions in spdlog error handling to avoid abort termination of process - Improvement: Block cache access/write log from writing to log if disk is low on free space ## 0.2.22 - Bugfix: Under heavy load, an http.sys async response handler could end up deleting the HTTP transaction object before the issuing call had completed. This is now fixed - Improvement: More tracing scopes in zenserver ## 0.2.21 - Feature: New http endpoint for background jobs `/admin/jobs` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back - Feature: Incremental oplog export for block-base target (Cloud/File). If a base is given it will download an existing oplog (excluding attachments) and try to reuse existing block references in that oplog. - `--basename` option for file based `oplog-export` - `--basekey` option for cloud based (Jupiter) `oplog-export` - Feature: Added `--cache-write-log` and `--cache-access-log` command line option to enable/disable cache write/access logs - Feature: Added `--http-threads`, `--httpsys-async-work-threads`, `--httpsys-enable-request-logging` and `--httpsys-enable-async-response` command line options to zenserver - Feature: More statistics for Cache, Project Store and Cid Store - Cache: `requestcount`, `badrequestcount`, `writes` - Project Store: `requestcount` - Cid Store: `cidhits`, `cidmisses`, `cidwrites` - Bugfix: Make sure cache logging thread does not crash on errors - Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle - Bugfix: Issue warning instead of assert on bad data in cid store - Bugfix: Don't index out of string_view range when parsing URI in httpsys - Improvement: Sorting attachments in oplog blocks based on Op key to group op attachments together - Improvement: Don't split attachments associated with the same op across oplog blocks - Improvement: 25% faster oplog op reading, only read and parse op data of latest op for particular key speeding up reading of oplog with old oplog data ## 0.2.20 - Feature: `zen up` command has two new command line options - `--config ` tells zenserver to start with a specific config file - `--owner-pid ` tells zenserver to start with a owning process id - Feature: `zen attach` command to add additional owning processes to a running zenserver instance - `--owner-pid ` adds pid to running zenserver instance list of owning processes - Feature: `--write-config` command line option for zenserver - `--write-config ` path to a file which will contain a lua config file for zenserver combining all command line options and optional lua config files - Bugfix: Only write disk usage log if disk writes are allowed (disk space is not cirtically low) - Improvement: `zen up` command will check if zenserver is currently running before starting up a new instance - Improvement: Add retry logic when creating oplog temp files and cas block files - Improvement: Large attachments fetched from Jupiter while doing oplog-import now streams to disk and are moved in place ## 0.2.19 - Bugfix: Fix deadlock in project store garbage collection - Bugfix: Fix zen command executable not being able to access shared status memory (`zen status`, `zen down` etc fails) on MacOs - Bugfix: All options given on command line now overrides lua config file settings - Improvement: All options available from command line can now be configured in the lua config file (with a few exceptions such as `owner-pid`, `install` and `uninstall`) ## 0.2.18 - Feature: Add `--embedloosefiles` option to `oplog-export` which adds loose files to the export, removing need to call `oplog-snapshot` - Bugfix: Fix construction order in OpenProcessCache to avoid crash in OpenProcessCache::GcWorker - Bugfix: Retain `ServerPath` in oplog when performing `oplog-snapshot`. This is a short-term fix for current incompatibility with the UE cooker. - Bugfix: Fix OpenProcessCache state error causing assert/error - Bugfix: Make sure to reset cache logging worker thread event to avoid busy-looping looking for more work - Improvement: Make sure we have disk space available to do GC and use reserve up front if need be - Improvement: We now build the Linux target using the UE toolchain to be compliant with the VFX platform that UE uses for Linux. ## 0.2.17 - Feature: Add `oplog-mirror` command to Zen command line tool. It can be used to export the contents of an oplog as files. Currently it will export all files, filtering options will be added at a later time - Feature: Add `--force-update` option to Zen command line tool `project-create` to update or create a project store project. It will update meta information about the project without affecting existing oplogs if project exists. - Feature: Add `--force-update` option to Zen command line tool `oplog-update` to update or create a project store oplog. It will update meta information about the project without affecting existing oplog data if oplog exists. - Feature: Zen command line tool `project-delete` to delete a project store project and all its oplogs. - `--project` Project name (id) - Feature: Zen command line tool `oplog-delete` to delete a project store oplog and the oplog data. - `--project` Project name (id) - `--oplog` Oplog name (id) - Bugfix: Make sure to check oplog op attachments when gathering references for GC - Bugfix: Reduce log level of RLIMIT message on Mac/Linux to avoid it interfering with parsing of stdout from `zen version` command - Bugfix: Make sure we close our trace session properly at exit when trace is enabled - Improvement: Use buffered file reading when replaying oplog - Improvement: Add endpoint in project store to update the information in a project without deleting the stored data/oplog - Improvement: Add endpoint in project store to update the information in a project without deleting the stored data/oplog - Improvement: Add oplog op content to error result if attachment is missing when doing `oplog-export` - Improvement: Windows: Cache process handles for FormatPackageMessage reducing function execution time from 100+us to ~1 us - Improvement: Skip upstream logic early if we have no upstream endpoints - Improvement: Cachestore logging of CbObjects are now async - Improvement: Use better hashing algorithm for instance pointers when using shared lock in IoBufferExtendedCore::Materialize - Improvement: Use tsl/robin-map/robin-set in compactcas and projectstore for 30% faster GC ## 0.2.16 - Feature: Add more stats for `stats/prj` - Project: read/write/delete count - Oplog: read/write/delete count - Chunk: hit/miss/write count - Op: hit/miss/write count - BadRequest count - Bugfix: Allow oplog file mapping where ServerPath is missing if a attachment hash is specified - Bugfix: Make sure we always write "data" attachment hash for snapshotted oplog entries - Bugfix: Fixed expiry limit for GC of project/oplogs - Improvement: Add `response.text` to output in log when jupiter request fails - Improvement: Only hash jupiter oplog ref once when uploading - Improvement: Increase request timeout when uploading to Jupiter to 3 min (to handle very large attachments) - Improvement: Fix issues with latest fmt vcpk dependency (10.0.0) and sentry-native for linux ## 0.2.15 - Feature: Add `--assume-http2` option to cloud style import/export command to use a HTTP/2 endpoint without without HTTP/1.1 upgrade - Bugfix: Make sure out of memory condition does not terminate http-asio service thread. UE-191531 - Bugfix: `oplog-import` with `--file` source now sends the oplog folder correctly to zenserver - Bugfix: If `oplog-export` fails while creating blocks, wait for background jobs to finish before aborting to avoid crash - Bugfix: If `GetChunkInfo` in project store finds a chunk in the wrong format, return a readable error instead of ASSERT - Bugfix: If checking for state_marker throws exception, exit gracefully rather than throw exception - Improvement: More details in zenserver logfile if jupiter operation fails ## 0.2.14 - Feature: Added `zen serve` command for establishing a link to a directory tree for use with staged UE builds and the `-Mount=` option - Bugfix: Make sure to validate return pointer when calling Memory::Alloc in all locations - Bugfix: Log error instead of hard crash if GC scheduler thread throws exception - Improvement: In oplog import/export, try to resolve access token via env variable on zen command side as first option, with resolve on zenserver side as second option. Resolves [UE-189978](https://jira.it.epicgames.com/browse/UE-189978) - Improvement: Keep reason and status code when parsing responses from jupiter remote requests - Improvement: Add additional context for errors when importing/exporting oplogs - Improvement: Added `ZenServerInstance::SpawnServerAndWait` and improved logic around process termination when using `ZenServerInstance::AttachToRunningServer` - Improvement: When uploading compressed blob to jupiter, use streaming reading of source file if it is a "whole file" - a large attachment. ## 0.2.13 - Feature: Project store now has a `snapshot` RPC on oplogs which may be used to inline any files referenced by name into Zen store. This makes the oplog transportable - Feature: Zen command line tool `oplog-snapshot` which may be used to inline any files referenced by name into Zen store. This makes the oplog transportable - `--project` Project name (id) - `--oplog` Oplog name (id) - Feature: Session Id and Request Id are now logged in log for cache put/get operations - Bugfix: Prevent destructors in ProjectStore::Project, ScopedActivityBase and FileMapping from throwing exceptions to avoid abort termination - Bugfix: Zen CLI command help now include descriptions for positional arguments - Bugfix: Correctly prefix auth token when using a bare token in project oplog import/export - Bugfix: Make sure GetEnvVariable can handle values that are longer than 1023 characters - Improvement: Throw exception with information on failed memory allocation instead of calling ZEN_ASSERT - Improvement: Added support for streaming decompression - Improvement: Added zenserver.exe and zen.exe/zen.pdb to Sentry debug information upload to populate unwind information - Improvement: Front-end can now be served from a development directory in release mode as well as debug if there's no zipfs attached - Improvement: Increased retry logic in diskcachelayer when we are denied moving a temporary file into place - Improvement: Named some additional background threads for better debug / sentry reporting - Update: Bump CI VCPKG version to 2023.04.15 and xmake to 2.7.9 (was 2022.08.15 and 2.6.5) ## 0.2.12 - Feature: zenserver/zen: Added zen command line command `scrub` which can be used to trigger a data scrubbing pass which traverses all stored data and verifies its integrity. If any checksum mismatches or structural errors are found the content is dropped. For now this does not provide much feedback in the console, but the zenserver logs will contain information about the operation - Feature: zen: added zen `bench` command which has an option to empty Windows standby lists. This effectively empties the system (disk) cache, which can be useful when performing benchmarks since this puts the system in a more consistent state - Feature: zen: added zen `copy` command which can be used to perform copy-on-write copies of files and directories on supported file systems (i.e ReFS on Windows). This is useful when working with test datasets where you want to avoid tests modifying the original test data - Feature: zenserver: Add command line option `--gc-projectstore-duration-seconds` to control GC life time of project store data - Bugfix: Improve error handling when processing requests in http asio - Bugfix: Error out if `test` is passed to zenserver in release builds (tests are only compiled in for debug) - Bugfix: Gracefully exit with error code if problems arise during startup (used to cause abort termination) - Bugfix: Project oplog delete fixed so it works even right after server startup, before the oplog has been instantiated in memory - Bugfix: Corrected argument name in oplog-export file target RPC message - Improvement: Change state_marker detection (deletion of DDC folder) log to WARN, it is not an error but useful information in the log output - Improvement: Added logging when bad chunks are detected in `BlockStore` - Improvement: `zen::SetCurrentThreadName` now also sets trace (Insights) thread name - Improvement: All thread pool threads now have names - Improvement: zenserver now emits session information to trace (Insights) for a better session browser experience - Improvement: Add more trace instrumentation - Improvement: Eliminated ATL header dependency - Improvement: If no `-hosturl=...` parameter is passed to zen CLI commands we use the current session state from shared memory to pick an instance to communicate with - Improvement: Better option validation in zen command line parsing ## 0.2.11 - Feature: Gracefully exit if Ctrl-C is pressed - Feature: Structured cache now writes an activity log to `logs/z$` which may be used to understand client interactions better. Enabled by default for the time being - Bugfix: Return error code on exit as set by application - Bugfix: Fix crash at startup if dead process handles are detected in ZenServerState - Bugfix: Fixed assert/error when running block store GC and a block to GC does not exist - Bugfix: GC could mix up locations of cache bucket items causing it to return the wrong item for a specific key. All cache buckets from previous versions will be wiped to remove inconsistent state - Improvement: Log details about file and read operation when it fails inside IoBuffer::Materialize() ## 0.2.10 - Feature: zenserver now writes a state_marker file in the root of the data directory. Deleting this file will cause zenserver to exit. This is used to detect if user is deleting the data folder while zenserver is running - Feature: Disk writes are now blocked early and return an insufficient storage error if free disk space falls below the `--low-diskspace-threshold` value - Feature: zenserver: Add command line option `--sentry-allow-personal-info` to allow personally identifiable information in sentry reports, disabled by default - Feature: Age-based GC of oplogs in project store - Improvement: Failing to write index snapshots or access times are now considered a warning rather than error - Bugfix: Validate that block store entries points inside valid blocks when initializing - Bugfix: Close down http server gracefully when exiting even while requests are still being processed - Bugfix: Flush snapshot for filecase on flush/exit - Bugfix: Fix log of size found when scanning for files in filecas ## 0.2.9 - Bugfix: Treat reading outside of block store file as a not found error. We may encounter truncated blocks due to earlier abnormal termination of zenserver or disk failures ## 0.2.8 - Feature: ASSERTs triggered at runtime are sent directly to Sentry with callstack if sentry is enabled - Bugfix: Verify that there are blocks to GC for block store garbage collect (void division by zero) - Bugfix: Write log error and flush log before reporting error to Sentry/error logger - Bugfix: Log ERROR in scope guard if function throws exception, throwing exception causes application abort ## 0.2.7 - Bugfix: Safely handle missing blocks when doing garbage collection in block store data - Bugfix: Only strip uri accept type suffix if it can be parsed to a known type - Bugfix: Keep system error code on Windows when file mapping fails and propagate to log/exception - Bugfix: Catch any errors throw in HttpAsioServer() destructor and log error. ## 0.2.6 - Strip __FILE__ macro names in logging to only include the file name as to not expose file paths of the machine building the executable - Bugfix: Reporting the correct callstack to sentry on ERROR/CRITICAL failure. - Extend sentry message with triggering file/line/function ## 0.2.5 - Feature: Zen command line tool `rpc-record-start` to record all RPC requests to the structured cache - `--path` Recording file path where the rpc requests will be stored - Feature: Zen command line tool `rpc-record-stop` stop the currently active RPC request recording started with `rpc-record-start` - Feature: Zen command line tool `rpc-record-replay` replacy a RPC request recording created with `rpc-record-start` - `--path` Recording file path where the rpc requests are stored - `--numthreads` Number of worker threads to use while replaying the RPC requests - `--numproc` Number of worker processes to run, if more than one new processes will be spawn with `` workers each - `--offset` Offset into request playback to start at - `--stride` The stride to use when selecting requests to playback - `--onhost` Replay the recording inside the zenserver bypassing http overhead - `--showmethodstats` Show statistics of which RPC methods are used - `--forceallowlocalrefs` Force the requests to allow local references (file path/file handle) - `--disablelocalrefs` Force disable local references in request (file path/file handle) - `--forceallowlocalhandlerefs` Force the requests to allow local references via duplicated file handles for requests that allow local refs - `--disablelocalhandlerefs` Force disable local references via duplicated file handles in requests - `--forceallowpartiallocalref` Force the requests to allow local references for files that are not saved as whole files for requests that allow local refs - `--disablepartiallocalrefs` Force disable local references for files that are not saved as whole files for requests that allow local refs - Feature: Zen command line tool `cache-stats` to give stats result about the zen cache - Feature: Zen command line tool `project-stats` to give stats result about the zen project store - Feature: Zen command line tool `cache-details` to give detail result about the zen cache, defaults to overview information about the cache - `--namespace` Get information about cache values in a namespace - `--bucket` Get information about cache values limited to a specific bucket in a namespace - `--valuekey` Get information about a cache value in a specific bucket in a namespace, valuekey is specified as IoHash hex string - `--details` Get detailed information about each cache record - `--attachmentdetails` Get detailed information about each attachments for each cache record - `--csv` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: Zen command line tool `project-details` to give detail result about the zen project store, defaults to overview information about the project store - `--project` The project id to get information about - `--oplog` The oplog id to get information about - `--opid` The op Oid to get information about - `--details` Get detailed information about the op - `--opdetails` Extract the entire op information (not available in CSV output) - `--attachmentdetails` Get detailed information about each attachments for each op - `--csv` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: New project store stats endpoint `/stats/prj` to get stats info for zen project store - Feature: New project store details endpoints `/prj/details$`, `/prj/details$/{project}`, `/prj/details$/{project}/{oplog}`, `/prj/details$/{project}/{oplog}/{op}` to give detail result about the zen project store, defaults to overview information about the project store items - `details=true` Get detailed information about the op - `opdetails=true` Extract the entire op information - `attachmentdetails=true` Get detailed information about each attachments for each op - `csv=true` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: New cache detail endpoints `/z$/details$`, `/z$/details$/{namespace}`, `/z$/details$/{namespace}/{bucket}`, `/z$/details$/{namespace}/{bucket}/{key}` has been added - `details=true` Get detailed information about each cache record - `attachmentdetails=true` Get detailed information about each attachments for each cache record - `csv=true` Format the response as a comma delimited CSV file. If not specified it defaults to CbObject but can auto-format to json - Feature: `--junit` switch to `xmake test` to generate junit style reports of tests. - Feature: CI build on GitHub now uploads junit test reports as artifact to the check for PR validation and mainline validation - Feature: Payloads from zenserver can now be sent using duplicated file handles if caller requests provides client ProcessId (Windows only). - Feature: Add `--port` option to zen down command to shut down servers on different base ports - Bugfix: Make sure async responses are sent async correctly in httpsys - Bugfix: Don't delete manifest file in cas root when initializing a new filecas folder - Bugfix: Sentry does not like UNC paths, so strip the prefix before passing them to sentry - Bugfix: Make sure zen down command uses the correct port for shutdown event - Improvement: FileCas now keeps an up to date index of all the entries improving performance when getting cache misses on large payloads - Improvement: Structured cache now keeps RawHash and RawSize in memory avoiding materialization of cache values before sending response - Changed: Exit with failure code on port conflict rather than reporting crash to Sentry - Changed: removed catch2 support for now since it does not handle multithreaded tests - Bugfix: fixed bug in dashboard content serving (see PR #255) ## 0.2.4 - Bugfix: Don't send empty http responses with content type set to Text. Fixes UE-177895 ## 0.2.3 - Feature: Add support for "packagedata" mapping in oplog entries - Feature: Zen command line tool `project-create` to create a project store project - `--project` Project name (id) - `--rootdir` Absolute path to root directory (optional) - `--enginedir` Absolute path to engine root directory (optional) - `--projectdir` Absolute path to project directory (optional) - `--projectfile` Absolute path to .uproject file (optional) - Feature: Zen command line tool `oplog-create` to create a project store oplog - `--project` Project name (id) - `--oplog` Oplog name (id) - `--gcpath` Absolute path to oplog lifetime marker file (optional) - Feature: Build scripts and tooling to build zen compliant with VFX reference platform CY2022/2021 matching UE linux builds - Feature: added `xmake sln` task which replaces `generate_projects.bat` - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: ```json { "method" : "", "params" : { "maxblocksize": "", "maxchunkembedsize": "", "file" : { "path" : "", "name" : "" }, "cloud" : { "url" : "", "namespace" : "", "bucket" : "", "key" : "", "openid-provider" : "", "access-token" : "", "access-token-env": ", "disableblocks" : "", "disabletempblocks" : "" }, "zen" : { "url" : "", "project" : "", "oplog" : "" } } } ``` - `"method"`supported methods are `"export"` and `"import"` to import/export an oplog - `"params"` container for parameters - `"maxblocksize"` - Optional. The maximum size of a block of attachments, default 134217728 (128 Mb) (export only) - `"maxchunkembedsize"` - Optional. The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments, default 1048576 (1Mb) (export only) - `"force"` - Optional. Boolean flag to indicate weather attachments should be uploaded/downloaded disregarding prior existance - External location types are "file" (File system), "cloud" (UE Cloud Storage service) or "zen" (Zen server instance), provide one of those as remote location. - `"file"` - Optional. Indicates remote location is the local file system - `"path"` - File system path folder to export to / import from - `"name"` - File name of oplog output, written into - `"cloud"` - Optional. Indicates remote location is UE Cloud Storage service - `"url"` - Jupiter service endpoint url - `"namespace"` - Name of namespace to store data to - `"bucket"` - Name of bucket to store data to - `"key"` - IoHash key to the stored oplog container - `"openid-provider"` - Optional. Name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for - `"access-token"` - Optional. JWT access token to authenticate with - `"access-token-env"` - Optional. Name of environment variable that holds an JWT access token to authenticate with - `"disableblocks"` - Optional. Disable creation of attachments blocks - "true"/"false" (export only) - `"disabletempblocks"` - Optional. Disable creation of attachments temp blocks forcing upload before oplog container - "true"/"false" (export only) - `"zen"` - Optional. Indicates remote location is a Zen server instance - `"url"` - Zen server instance url - `"project"` - The remote project name (id) - `"oplog" - The remote oplog name (id) - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. ```json { "method" : "getchunks", "chunks" : [ "", ] } ``` - CbPackage RPC format for `putchunks` method, attachments are stored in CidStore ```json { "method" : "putchunks", } ``` - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - `--project` Project name (id) - `--oplog` Project name (id) - `--maxblocksize` The maximum size of a block of attachments (optional) - `--maxchunkembedsize` The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments (optional) - `--force` Force upload/download of attachments even if they already exist. - `--file` File system path folder to export to / import from - `--name` File name of oplog output, written into `--file` path - `--disableblocks` Disable block creation and save all attachments individually - `--forcetempblocks` Force creation of temp attachment blocks - `--cloud` Jupiter service endpoint to export to / import from - `namespace` Name of namespace to store data to - `bucket` Name of bucket to store data to - `key` Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket) - `openid-provider` Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name - `access-token` Optional JWT access token to authenticate with - `access-token-env` - Optional name of environment variable that holds an JWT access token to authenticate with - `disableblocks` Disable block creation and save all attachments individually - `disabletempblocks` Disable temp block creation and upload blocks without waiting for oplog container to be uploaded - `--zen` Zen server instance url to export to / import from - `--target-project` The remote project name (id) (optional, defaults to same as `project`) - `--taret-oplog` The remote oplog name (id) (optional, defaults to same as `olplog`) - `--clean` Delete and create a new oplog before starting export - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint. - `--project` Project name (id) - `--oplog` Project name (id) - `--force` Force upload/download of attachments even if they already exist. - `--file` File system path folder to export to / import from - `--name` File name of oplog output, written into `--file` path - `--cloud` Jupiter service endpoint to export to / import from - `namespace` Name of namespace to store data to - `bucket` Name of bucket to store data to - `key` Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket) - `openid-provider` Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name - `access-token` Optional JWT access token to authenticate with - `access-token-env` - Optional name of environment variable that holds an JWT access token to authenticate with - `--zen` Zen server instance url to export to / import from - `--source-project` The remote project name (id) (optional, defaults to same as `project`) - `--source-oplog` The remote oplog name (id) (optional, defaults to same as `olplog`) - `--clean` Delete and create a new oplog before starting import - Improvement: Faster oplog replay - reduces time to open an existing oplog - Improvement: Clearer error messages and logging when requests to project store fails - Changed: Removed remnants of old mesh experiment - Changed: Remove obsolete export-project command - Changed: Removed remnants import-project command - Changed: Removed unused remote build scripts - Changed: Removed very old and invalid TODO.md - Changed: Removed some deprecated scripts ## 0.2.2 - Feature: Added info (GET) endpoints for structured cache - `/z$` - get a list of namespaces and global info - `/z$/{namespace}` - get list of buckets in a namespace and namespace related info - `/z$/{namespace}/{bucket}` - get bucket info - Feature: Added project store oplog info: `markerpath`, `totalsize`, `opcount`, `expired` on GET requests for oplog - Feature: Added project store project info: `expired` on GET requests for project - Feature :Added project store root route `/prj` which is identical to `/prj/list` - Feature: Zen command line tool `cache-info` to show cache, namespace or bucket info - Feature: Zen command line tool `project-info` to show store, project or oplog info - Feature: Zen command line tool `project-drop` to drop project or oplog - Feature: Zen command line tool `gc` to trigger a GC run - Feature: Zen command line tool `gc-info` to check status of GC - Feature: Added version information to zenserver and zen command line tool executables - Bugfix: Don't log "time to next GC" if time to next GC is not set - Improvement: Don't wait for GC monitor interval before doing first GC check - Improvement: Zen command line tool now fails on any unrecognized arguments - Improvement: Zen command line tool now displays extra help for all sub-commands - Improvement: Host address can now be configured for zen command line tool `drop` command - Improvement: Added precommit xmake task `xmake precommit` to run precommit checks - Changed: Default GC interval set to 1 hour - Changed: Default GC cache duration set to 2 weeks - Changed: Removed HttpLaunchService and related code - Changed: Removed dead/experimental file system related code - Changed: Removed faux vfs config option ## 0.2.1 - Feature: Oplog level GC in project store. If gc marker file path is given by UE, oplogs will be GC:d when marker file is deleted (and GC is triggered) - Bugfix: Index handling for cache large object store was broken resulting in log always being played back - Bugfix: Make sure to flush cache store on call to flush on service and exit - Improvement: Don't write index snapshots if no new entries has been added to log ## 0.2.0 - Feature: Recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request - `/z$/exec$/start-recording?` - `/z$/exec$/stop-recording` - `/z$/exec$/replay-recording?&` - Feature: Disk size triggered GC, a soft disk usage limit for cache data - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging - Improvement: Much improved performance, between 2x to 9x improvement under heavy load (excluding http service overhead). See https://github.com/EpicGames/zen/pull/200 for details - Bugfix: Always store records or oplog entries before storing attachments to avoid GC finding unreferenced chunks i CidStore - Bugfix: Updated Zen `drop` command to support namespaces - Bugfix: Use ZEN_CONSOLE for output to console in Zen commands - Bugfix: Zen `status` command now shows info about found Zen instances - Bugfix: Zen `top` command now shows session id string - Bugfix: On Windows platforms explicitly set the special `SO_EXCLUSIVEADDRUSE` flag as `SO_REUSEADDR` still allows shared use of sockets - Bugfix: Fix logging of number of entries read from caslog at startup - Bugfix: Fix asio http handling of very large/malformed headers and handle mismatching content size - Changed: Reduced GC `INFO` spam by converting to `DEBUG` log messages - Changed: Use Iso8601 format for logging start and end message ## 0.1.9 - Feature: Adds two command to Zen command tool to export/import project store oplogs with attachments - `export-project [oplogs...]` - `import-project [oplogs...]` - Feature: Adds command to query Zen version, specify `host-name` url to query running service version, otherwise you get zen command version. `detailed` option gives you long form version. - `version [host-name] [detailed]` - Feature: New service endpoint to query Zen server version, add `?detailed=true` to get long form version - `/health/version` - Feature: Configure OpenID providers from cmd line and Lua cfg - Feature: Added zen command line executable to release distribution - Bugfix: Fix double reporting of disk usage for namespaces - Bugfix: Fix double garbage collection analisys and garbage collection execution of namespaces - Improvement: Improve tracking of used disk space for filecas and compactcas - Improvement: Add tracking of used disk space for project store - Improvement: Bumped limit for storing cache values as separate files to reduce number of loose files - Improvement: Optimizations when handling compressed buffer (less materialization and reading of headers) - Improvement: Send attachments as file references if the IoBuffer we find represents a complete file and `AcceptFlags` in RPC request allows it. - Improvement: Don't reserve full block size for block store files at creation ## v0.1.8 - Change: Responding with new wire format for RPC requests requires the requestor to add a `Accept` field in the request. This is to allow compatability with older clients for shared instances. - Improvement: Fixed concurrency issues in project store - project and oplog lifetime issues. - Improvement: Don't open oplogs until we require use of them. - Cleanup: Remove rocksdb experimental code. - Feature: Add GC to projects store. Checks path to project file in UE side to determine when a project may be GCd on the Zen side. ## v0.1.7 - Change: All RPC responses are now formatted using dedicated wire format, Zen server has fallback to enable compatability with legacy upstreams - Feature: Adding a `.json` extension to the `--abslog` option will make zenserver log in json format to file - Feature: Create release in Sentry and use `sentry_options_set_release` to associate the executable - Bugfix: CompactBinary: Fixed LoadCompactBinary to gracefully handle read failures and sizes larger than the archive. From http1s://p4-swarm.epicgames.net/changes/21983905 - Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid - Bugfix: Handle edge case when trying to materialize a IoBuffer of zero size via memory mapping - Improvement: Logging: don't do formatting of messages the will not be logged - Improvement: Logging: Timing and upstream source information in upstream logging when debug level logging is enabled - Improvement: Reduce buffer creation and copying in ParsePackageMessage - Improvement: Don't read attachments for oplogs we already have when parsing oplog message ## v0.1.6 - Bugfix: Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid (UE-164966) ## v0.1.5 - Bugfix: Don't fail entire request if GetCacheValue from Horde fails for a single value ## v0.1.4 - Change: Bumped ZEN_SCHEMA_VERSION - this will invalidate entire local cache when deployed - Change: Make CAS storage an hidden implementation detail of CidStore, we no longer hash and do mapping to compressed hash when storing cache values - Feature: Extended zen print command to also handle CbPackage and CompressedBuffer format payloads - Feature: Added /prj/{project}/oplog/{log}/{op} endpoint to allow retrieval of an op entry by LSN. Supports returning CbObject or CbPackage format payloads - Improvement: asio: added some context to error reporting - Improvement: namespace/bucket validation now uses AsciiSet for more efficient validation - Improvement: Frontend: simplified content-type logic - Improvement: Improved message indicating no GC is scheduled - Improvement: Implement proper GetCacheValues upstream path - Improvement: Demote a number of ZEN_ERROR log calls for problems that are recoverable and handled - Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid - Bugfix: Fixed issue in CbPackage marshaling of local reference - Bugfix: Fix crash when switching Zen upstream configured via DNS when one endpoint becomes unresposive - Bugfix: Fixed issue where projects would not be discovered via DiscoverProjects due to use of stem() vs filename() - Bugfix: Use "\\\\?\\" prefixed paths on Windows and fix hardcoded path delimiters (UE-141222) - Bugfix: Safer detection of html folder when running non-bundled executable - Bugfix: Use "application/x-jupiter-inline" to fetch GetCacheValues from Horde (UE-162151) - Sentry: Added logging of sentry_init error code - Sentry: Attach log file to Sentry error reports - Sentry: Capture capture error/critical log statements as errors in Sentry - Update: Bump VCPKG version to 2022.08.15 - CI: MacOS build enable again in GitHub CI - CI: Upload debug info and source files to Sentry when release is created ## v0.1.3 -- Enable adding namespace to URI based upstream requests -- Add logging of namespace name and bucket name if we get invalid names in requests -- Updated README.md with Linux dev prerequisites -- asio: added some logging to indicate concurrency -- Fixed expired cache keys overwriting between namespaces when bucket names were the same in multiple namespaces ## v0.1.2 - Tweak bundle compression settings to streamline build - ZenCacheDiskLayer::CacheBucket::GatherReferences: Don't hold index lock while reading standalone values - hardening of ZenCacheDiskLayer::CacheBucket::PutStandaloneCacheValue - GitHub Actions: Move release job to in-house linux agent ## v0.1.1 - BlockStore (small object store) Always block GC of current write block - Make it possible to configure GC monitoring interval using `--gc-monitor-interval-seconds` - Keep "reason" from upstream response so we can present it even if the request fails without outright error - New GitHub Actions release flow - Add release flow in GitHub actions on pushed tag `v0.1.2` gives full release, `v0.1.2-pre0` gives pre-release ## 0d08450 - Fixes issue with broken Zen instances for legacy requests ## 63f50b5 - Enable FILE_SHARE_DELETE on standalone files in disk buckets - fixes Jira UE-154234 - Make sure we can properly create the block file before assigning it for use - fixes Jira UE-154438 - Horde execute compressed input blobs - Drop namespace support - Safer delete of cache buckets ## dba8b36 - Namespaces: This introduces namespaces to the zenserver but only the default ue4.ddc is supported. Clients that don't send a namespace in the request will keep old behviour, new clients that sends namespace is required to use ue4.ddc (which they currently do) - Aligned bucket naming rules with UE code base - Fix retry counter and add an extra iteration to give more time for success during contention for standalone files in cache - Make sure CacheBucket::PutStandaloneCacheValue cleans up the temp file - Restore logic where we accept failed overwrite if resulting size is the same for standlone file in cache - Correctly calculate the m_TotalSize difference when overwriting file for standalone files in cache - Fix namespace folder scanning