## - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level - Feature: Add endpoint for controlling Insights tracing - GET `/admin/trace` to query if tracing is currently running or not - POST `/admin/trace/start` to start tracing - `host=` start tracing to a trace server at ip `` - `file=` start tracing to file at path `` - POST `/admin/trace/stop` stop the currently running trace - Feature: Add `zen trace` command to control Insights tracing - `zen trace` to show the status of tracing ("enabled" or "enabled") - `zen trace --host=` start tracing to a trace server at ip `` - `zen trace --file=` start tracing to file at path `` - `zen trace --stop` stop the currently running trace - Feature: Implemented virtual file system (VFS) support for debugging and introspection purposes - `zen vfs mount ` will initialize a virtual file system at the specified mount point. The mount point should ideally not exist already as the server can delete the entirety of it at exit or in other situations. Within the mounted tree you will find directories which allow you to enumerate contents of DDC and the project store - `zen vfs unmount` will stop the VFS - `zen vfs info` can be used to check the status of the VFS - Bugfix: Do controlled shut down order of zenserver and catch exceptions thrown during shutdown - Improvement: Add names to background jobs for easier debugging - Improvement: Background jobs now temporarily sets thread name to background job name while executing - Improvement: Background jobs tracks worker thread id used while executing - Improvement: `xmake sln` can now be used on Mac as well to generate project files - Improvement: http.sys request queues are named to make it easier to find performance counters in Performance Monitor and such - Improvement: http.sys - if request rate is too high then rejected requests will get a 503 response instead of a dropped connection ## 0.2.23 - Bugfix: Respect result from FinalizeRef in Jupiter oplog upload where it requests missing attachments - Improvement: Increase timeout when doing import/export of oplogs to jupiter to 30 min per request - Improvement: Better logging/progress report on oplog export - Improvement: Ignore OOM errors in spdlog, just drop the error since we can't do anything useful if we run out of memory here - Improvement: Try to catch any exceptions in spdlog error handling to avoid abort termination of process - Improvement: Block cache access/write log from writing to log if disk is low on free space ## 0.2.22 - Bugfix: Under heavy load, an http.sys async response handler could end up deleting the HTTP transaction object before the issuing call had completed. This is now fixed - Improvement: More tracing scopes in zenserver ## 0.2.21 - Feature: New http endpoint for background jobs `/admin/jobs` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back - Feature: Incremental oplog export for block-base target (Cloud/File). If a base is given it will download an existing oplog (excluding attachments) and try to reuse existing block references in that oplog. - `--basename` option for file based `oplog-export` - `--basekey` option for cloud based (Jupiter) `oplog-export` - Feature: Added `--cache-write-log` and `--cache-access-log` command line option to enable/disable cache write/access logs - Feature: Added `--http-threads`, `--httpsys-async-work-threads`, `--httpsys-enable-request-logging` and `--httpsys-enable-async-response` command line options to zenserver - Feature: More statistics for Cache, Project Store and Cid Store - Cache: `requestcount`, `badrequestcount`, `writes` - Project Store: `requestcount` - Cid Store: `cidhits`, `cidmisses`, `cidwrites` - Bugfix: Make sure cache logging thread does not crash on errors - Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle - Bugfix: Issue warning instead of assert on bad data in cid store - Bugfix: Don't index out of string_view range when parsing URI in httpsys - Improvement: Sorting attachments in oplog blocks based on Op key to group op attachments together - Improvement: Don't split attachments associated with the same op across oplog blocks - Improvement: 25% faster oplog op reading, only read and parse op data of latest op for particular key speeding up reading of oplog with old oplog data ## 0.2.20 - Feature: `zen up` command has two new command line options - `--config ` tells zenserver to start with a specific config file - `--owner-pid ` tells zenserver to start with a owning process id - Feature: `zen attach` command to add additional owning processes to a running zenserver instance - `--owner-pid ` adds pid to running zenserver instance list of owning processes - Feature: `--write-config` command line option for zenserver - `--write-config ` path to a file which will contain a lua config file for zenserver combining all command line options and optional lua config files - Bugfix: Only write disk usage log if disk writes are allowed (disk space is not cirtically low) - Improvement: `zen up` command will check if zenserver is currently running before starting up a new instance - Improvement: Add retry logic when creating oplog temp files and cas block files - Improvement: Large attachments fetched from Jupiter while doing oplog-import now streams to disk and are moved in place ## 0.2.19 - Bugfix: Fix deadlock in project store garbage collection - Bugfix: Fix zen command executable not being able to access shared status memory (`zen status`, `zen down` etc fails) on MacOs - Bugfix: All options given on command line now overrides lua config file settings - Improvement: All options available from command line can now be configured in the lua config file (with a few exceptions such as `owner-pid`, `install` and `uninstall`) ## 0.2.18 - Feature: Add `--embedloosefiles` option to `oplog-export` which adds loose files to the export, removing need to call `oplog-snapshot` - Bugfix: Fix construction order in OpenProcessCache to avoid crash in OpenProcessCache::GcWorker - Bugfix: Retain `ServerPath` in oplog when performing `oplog-snapshot`. This is a short-term fix for current incompatibility with the UE cooker. - Bugfix: Fix OpenProcessCache state error causing assert/error - Bugfix: Make sure to reset cache logging worker thread event to avoid busy-looping looking for more work - Improvement: Make sure we have disk space available to do GC and use reserve up front if need be - Improvement: We now build the Linux target using the UE toolchain to be compliant with the VFX platform that UE uses for Linux. ## 0.2.17 - Feature: Add `oplog-mirror` command to Zen command line tool. It can be used to export the contents of an oplog as files. Currently it will export all files, filtering options will be added at a later time - Feature: Add `--force-update` option to Zen command line tool `project-create` to update or create a project store project. It will update meta information about the project without affecting existing oplogs if project exists. - Feature: Add `--force-update` option to Zen command line tool `oplog-update` to update or create a project store oplog. It will update meta information about the project without affecting existing oplog data if oplog exists. - Feature: Zen command line tool `project-delete` to delete a project store project and all its oplogs. - `--project` Project name (id) - Feature: Zen command line tool `oplog-delete` to delete a project store oplog and the oplog data. - `--project` Project name (id) - `--oplog` Oplog name (id) - Bugfix: Make sure to check oplog op attachments when gathering references for GC - Bugfix: Reduce log level of RLIMIT message on Mac/Linux to avoid it interfering with parsing of stdout from `zen version` command - Bugfix: Make sure we close our trace session properly at exit when trace is enabled - Improvement: Use buffered file reading when replaying oplog - Improvement: Add endpoint in project store to update the information in a project without deleting the stored data/oplog - Improvement: Add endpoint in project store to update the information in a project without deleting the stored data/oplog - Improvement: Add oplog op content to error result if attachment is missing when doing `oplog-export` - Improvement: Windows: Cache process handles for FormatPackageMessage reducing function execution time from 100+us to ~1 us - Improvement: Skip upstream logic early if we have no upstream endpoints - Improvement: Cachestore logging of CbObjects are now async - Improvement: Use better hashing algorithm for instance pointers when using shared lock in IoBufferExtendedCore::Materialize - Improvement: Use tsl/robin-map/robin-set in compactcas and projectstore for 30% faster GC ## 0.2.16 - Feature: Add more stats for `stats/prj` - Project: read/write/delete count - Oplog: read/write/delete count - Chunk: hit/miss/write count - Op: hit/miss/write count - BadRequest count - Bugfix: Allow oplog file mapping where ServerPath is missing if a attachment hash is specified - Bugfix: Make sure we always write "data" attachment hash for snapshotted oplog entries - Bugfix: Fixed expiry limit for GC of project/oplogs - Improvement: Add `response.text` to output in log when jupiter request fails - Improvement: Only hash jupiter oplog ref once when uploading - Improvement: Increase request timeout when uploading to Jupiter to 3 min (to handle very large attachments) - Improvement: Fix issues with latest fmt vcpk dependency (10.0.0) and sentry-native for linux ## 0.2.15 - Feature: Add `--assume-http2` option to cloud style import/export command to use a HTTP/2 endpoint without without HTTP/1.1 upgrade - Bugfix: Make sure out of memory condition does not terminate http-asio service thread. UE-191531 - Bugfix: `oplog-import` with `--file` source now sends the oplog folder correctly to zenserver - Bugfix: If `oplog-export` fails while creating blocks, wait for background jobs to finish before aborting to avoid crash - Bugfix: If `GetChunkInfo` in project store finds a chunk in the wrong format, return a readable error instead of ASSERT - Bugfix: If checking for state_marker throws exception, exit gracefully rather than throw exception - Improvement: More details in zenserver logfile if jupiter operation fails ## 0.2.14 - Feature: Added `zen serve` command for establishing a link to a directory tree for use with staged UE builds and the `-Mount=` option - Bugfix: Make sure to validate return pointer when calling Memory::Alloc in all locations - Bugfix: Log error instead of hard crash if GC scheduler thread throws exception - Improvement: In oplog import/export, try to resolve access token via env variable on zen command side as first option, with resolve on zenserver side as second option. Resolves [UE-189978](https://jira.it.epicgames.com/browse/UE-189978) - Improvement: Keep reason and status code when parsing responses from jupiter remote requests - Improvement: Add additional context for errors when importing/exporting oplogs - Improvement: Added `ZenServerInstance::SpawnServerAndWait` and improved logic around process termination when using `ZenServerInstance::AttachToRunningServer` - Improvement: When uploading compressed blob to jupiter, use streaming reading of source file if it is a "whole file" - a large attachment. ## 0.2.13 - Feature: Project store now has a `snapshot` RPC on oplogs which may be used to inline any files referenced by name into Zen store. This makes the oplog transportable - Feature: Zen command line tool `oplog-snapshot` which may be used to inline any files referenced by name into Zen store. This makes the oplog transportable - `--project` Project name (id) - `--oplog` Oplog name (id) - Feature: Session Id and Request Id are now logged in log for cache put/get operations - Bugfix: Prevent destructors in ProjectStore::Project, ScopedActivityBase and FileMapping from throwing exceptions to avoid abort termination - Bugfix: Zen CLI command help now include descriptions for positional arguments - Bugfix: Correctly prefix auth token when using a bare token in project oplog import/export - Bugfix: Make sure GetEnvVariable can handle values that are longer than 1023 characters - Improvement: Throw exception with information on failed memory allocation instead of calling ZEN_ASSERT - Improvement: Added support for streaming decompression - Improvement: Added zenserver.exe and zen.exe/zen.pdb to Sentry debug information upload to populate unwind information - Improvement: Front-end can now be served from a development directory in release mode as well as debug if there's no zipfs attached - Improvement: Increased retry logic in diskcachelayer when we are denied moving a temporary file into place - Improvement: Named some additional background threads for better debug / sentry reporting - Update: Bump CI VCPKG version to 2023.04.15 and xmake to 2.7.9 (was 2022.08.15 and 2.6.5) ## 0.2.12 - Feature: zenserver/zen: Added zen command line command `scrub` which can be used to trigger a data scrubbing pass which traverses all stored data and verifies its integrity. If any checksum mismatches or structural errors are found the content is dropped. For now this does not provide much feedback in the console, but the zenserver logs will contain information about the operation - Feature: zen: added zen `bench` command which has an option to empty Windows standby lists. This effectively empties the system (disk) cache, which can be useful when performing benchmarks since this puts the system in a more consistent state - Feature: zen: added zen `copy` command which can be used to perform copy-on-write copies of files and directories on supported file systems (i.e ReFS on Windows). This is useful when working with test datasets where you want to avoid tests modifying the original test data - Feature: zenserver: Add command line option `--gc-projectstore-duration-seconds` to control GC life time of project store data - Bugfix: Improve error handling when processing requests in http asio - Bugfix: Error out if `test` is passed to zenserver in release builds (tests are only compiled in for debug) - Bugfix: Gracefully exit with error code if problems arise during startup (used to cause abort termination) - Bugfix: Project oplog delete fixed so it works even right after server startup, before the oplog has been instantiated in memory - Bugfix: Corrected argument name in oplog-export file target RPC message - Improvement: Change state_marker detection (deletion of DDC folder) log to WARN, it is not an error but useful information in the log output - Improvement: Added logging when bad chunks are detected in `BlockStore` - Improvement: `zen::SetCurrentThreadName` now also sets trace (Insights) thread name - Improvement: All thread pool threads now have names - Improvement: zenserver now emits session information to trace (Insights) for a better session browser experience - Improvement: Add more trace instrumentation - Improvement: Eliminated ATL header dependency - Improvement: If no `-hosturl=...` parameter is passed to zen CLI commands we use the current session state from shared memory to pick an instance to communicate with - Improvement: Better option validation in zen command line parsing ## 0.2.11 - Feature: Gracefully exit if Ctrl-C is pressed - Feature: Structured cache now writes an activity log to `logs/z$` which may be used to understand client interactions better. Enabled by default for the time being - Bugfix: Return error code on exit as set by application - Bugfix: Fix crash at startup if dead process handles are detected in ZenServerState - Bugfix: Fixed assert/error when running block store GC and a block to GC does not exist - Bugfix: GC could mix up locations of cache bucket items causing it to return the wrong item for a specific key. All cache buckets from previous versions will be wiped to remove inconsistent state - Improvement: Log details about file and read operation when it fails inside IoBuffer::Materialize() ## 0.2.10 - Feature: zenserver now writes a state_marker file in the root of the data directory. Deleting this file will cause zenserver to exit. This is used to detect if user is deleting the data folder while zenserver is running - Feature: Disk writes are now blocked early and return an insufficient storage error if free disk space falls below the `--low-diskspace-threshold` value - Feature: zenserver: Add command line option `--sentry-allow-personal-info` to allow personally identifiable information in sentry reports, disabled by default - Feature: Age-based GC of oplogs in project store - Improvement: Failing to write index snapshots or access times are now considered a warning rather than error - Bugfix: Validate that block store entries points inside valid blocks when initializing - Bugfix: Close down http server gracefully when exiting even while requests are still being processed - Bugfix: Flush snapshot for filecase on flush/exit - Bugfix: Fix log of size found when scanning for files in filecas ## 0.2.9 - Bugfix: Treat reading outside of block store file as a not found error. We may encounter truncated blocks due to earlier abnormal termination of zenserver or disk failures ## 0.2.8 - Feature: ASSERTs triggered at runtime are sent directly to Sentry with callstack if sentry is enabled - Bugfix: Verify that there are blocks to GC for block store garbage collect (void division by zero) - Bugfix: Write log error and flush log before reporting error to Sentry/error logger - Bugfix: Log ERROR in scope guard if function throws exception, throwing exception causes application abort ## 0.2.7 - Bugfix: Safely handle missing blocks when doing garbage collection in block store data - Bugfix: Only strip uri accept type suffix if it can be parsed to a known type - Bugfix: Keep system error code on Windows when file mapping fails and propagate to log/exception - Bugfix: Catch any errors throw in HttpAsioServer() destructor and log error. ## 0.2.6 - Strip __FILE__ macro names in logging to only include the file name as to not expose file paths of the machine building the executable - Bugfix: Reporting the correct callstack to sentry on ERROR/CRITICAL failure. - Extend sentry message with triggering file/line/function ## 0.2.5 - Feature: Zen command line tool `rpc-record-start` to record all RPC requests to the structured cache - `--path` Recording file path where the rpc requests will be stored - Feature: Zen command line tool `rpc-record-stop` stop the currently active RPC request recording started with `rpc-record-start` - Feature: Zen command line tool `rpc-record-replay` replacy a RPC request recording created with `rpc-record-start` - `--path` Recording file path where the rpc requests are stored - `--numthreads` Number of worker threads to use while replaying the RPC requests - `--numproc` Number of worker processes to run, if more than one new processes will be spawn with `` workers each - `--offset` Offset into request playback to start at - `--stride` The stride to use when selecting requests to playback - `--onhost` Replay the recording inside the zenserver bypassing http overhead - `--showmethodstats` Show statistics of which RPC methods are used - `--forceallowlocalrefs` Force the requests to allow local references (file path/file handle) - `--disablelocalrefs` Force disable local references in request (file path/file handle) - `--forceallowlocalhandlerefs` Force the requests to allow local references via duplicated file handles for requests that allow local refs - `--disablelocalhandlerefs` Force disable local references via duplicated file handles in requests - `--forceallowpartiallocalref` Force the requests to allow local references for files that are not saved as whole files for requests that allow local refs - `--disablepartiallocalrefs` Force disable local references for files that are not saved as whole files for requests that allow local refs - Feature: Zen command line tool `cache-stats` to give stats result about the zen cache - Feature: Zen command line tool `project-stats` to give stats result about the zen project store - Feature: Zen command line tool `cache-details` to give detail result about the zen cache, defaults to overview information about the cache - `--namespace` Get information about cache values in a namespace - `--bucket` Get information about cache values limited to a specific bucket in a namespace - `--valuekey` Get information about a cache value in a specific bucket in a namespace, valuekey is specified as IoHash hex string - `--details` Get detailed information about each cache record - `--attachmentdetails` Get detailed information about each attachments for each cache record - `--csv` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: Zen command line tool `project-details` to give detail result about the zen project store, defaults to overview information about the project store - `--project` The project id to get information about - `--oplog` The oplog id to get information about - `--opid` The op Oid to get information about - `--details` Get detailed information about the op - `--opdetails` Extract the entire op information (not available in CSV output) - `--attachmentdetails` Get detailed information about each attachments for each op - `--csv` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: New project store stats endpoint `/stats/prj` to get stats info for zen project store - Feature: New project store details endpoints `/prj/details$`, `/prj/details$/{project}`, `/prj/details$/{project}/{oplog}`, `/prj/details$/{project}/{oplog}/{op}` to give detail result about the zen project store, defaults to overview information about the project store items - `details=true` Get detailed information about the op - `opdetails=true` Extract the entire op information - `attachmentdetails=true` Get detailed information about each attachments for each op - `csv=true` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: New cache detail endpoints `/z$/details$`, `/z$/details$/{namespace}`, `/z$/details$/{namespace}/{bucket}`, `/z$/details$/{namespace}/{bucket}/{key}` has been added - `details=true` Get detailed information about each cache record - `attachmentdetails=true` Get detailed information about each attachments for each cache record - `csv=true` Format the response as a comma delimited CSV file. If not specified it defaults to CbObject but can auto-format to json - Feature: `--junit` switch to `xmake test` to generate junit style reports of tests. - Feature: CI build on GitHub now uploads junit test reports as artifact to the check for PR validation and mainline validation - Feature: Payloads from zenserver can now be sent using duplicated file handles if caller requests provides client ProcessId (Windows only). - Feature: Add `--port` option to zen down command to shut down servers on different base ports - Bugfix: Make sure async responses are sent async correctly in httpsys - Bugfix: Don't delete manifest file in cas root when initializing a new filecas folder - Bugfix: Sentry does not like UNC paths, so strip the prefix before passing them to sentry - Bugfix: Make sure zen down command uses the correct port for shutdown event - Improvement: FileCas now keeps an up to date index of all the entries improving performance when getting cache misses on large payloads - Improvement: Structured cache now keeps RawHash and RawSize in memory avoiding materialization of cache values before sending response - Changed: Exit with failure code on port conflict rather than reporting crash to Sentry - Changed: removed catch2 support for now since it does not handle multithreaded tests - Bugfix: fixed bug in dashboard content serving (see PR #255) ## 0.2.4 - Bugfix: Don't send empty http responses with content type set to Text. Fixes UE-177895 ## 0.2.3 - Feature: Add support for "packagedata" mapping in oplog entries - Feature: Zen command line tool `project-create` to create a project store project - `--project` Project name (id) - `--rootdir` Absolute path to root directory (optional) - `--enginedir` Absolute path to engine root directory (optional) - `--projectdir` Absolute path to project directory (optional) - `--projectfile` Absolute path to .uproject file (optional) - Feature: Zen command line tool `oplog-create` to create a project store oplog - `--project` Project name (id) - `--oplog` Oplog name (id) - `--gcpath` Absolute path to oplog lifetime marker file (optional) - Feature: Build scripts and tooling to build zen compliant with VFX reference platform CY2022/2021 matching UE linux builds - Feature: added `xmake sln` task which replaces `generate_projects.bat` - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: ```json { "method" : "", "params" : { "maxblocksize": "", "maxchunkembedsize": "", "file" : { "path" : "", "name" : "" }, "cloud" : { "url" : "", "namespace" : "", "bucket" : "", "key" : "", "openid-provider" : "", "access-token" : "", "access-token-env": ", "disableblocks" : "", "disabletempblocks" : "" }, "zen" : { "url" : "", "project" : "", "oplog" : "" } } } ``` - `"method"`supported methods are `"export"` and `"import"` to import/export an oplog - `"params"` container for parameters - `"maxblocksize"` - Optional. The maximum size of a block of attachments, default 134217728 (128 Mb) (export only) - `"maxchunkembedsize"` - Optional. The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments, default 1048576 (1Mb) (export only) - `"force"` - Optional. Boolean flag to indicate weather attachments should be uploaded/downloaded disregarding prior existance - External location types are "file" (File system), "cloud" (UE Cloud Storage service) or "zen" (Zen server instance), provide one of those as remote location. - `"file"` - Optional. Indicates remote location is the local file system - `"path"` - File system path folder to export to / import from - `"name"` - File name of oplog output, written into - `"cloud"` - Optional. Indicates remote location is UE Cloud Storage service - `"url"` - Jupiter service endpoint url - `"namespace"` - Name of namespace to store data to - `"bucket"` - Name of bucket to store data to - `"key"` - IoHash key to the stored oplog container - `"openid-provider"` - Optional. Name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for - `"access-token"` - Optional. JWT access token to authenticate with - `"access-token-env"` - Optional. Name of environment variable that holds an JWT access token to authenticate with - `"disableblocks"` - Optional. Disable creation of attachments blocks - "true"/"false" (export only) - `"disabletempblocks"` - Optional. Disable creation of attachments temp blocks forcing upload before oplog container - "true"/"false" (export only) - `"zen"` - Optional. Indicates remote location is a Zen server instance - `"url"` - Zen server instance url - `"project"` - The remote project name (id) - `"oplog" - The remote oplog name (id) - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. ```json { "method" : "getchunks", "chunks" : [ "", ] } ``` - CbPackage RPC format for `putchunks` method, attachments are stored in CidStore ```json { "method" : "putchunks", } ``` - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - `--project` Project name (id) - `--oplog` Project name (id) - `--maxblocksize` The maximum size of a block of attachments (optional) - `--maxchunkembedsize` The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments (optional) - `--force` Force upload/download of attachments even if they already exist. - `--file` File system path folder to export to / import from - `--name` File name of oplog output, written into `--file` path - `--disableblocks` Disable block creation and save all attachments individually - `--forcetempblocks` Force creation of temp attachment blocks - `--cloud` Jupiter service endpoint to export to / import from - `namespace` Name of namespace to store data to - `bucket` Name of bucket to store data to - `key` Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket) - `openid-provider` Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name - `access-token` Optional JWT access token to authenticate with - `access-token-env` - Optional name of environment variable that holds an JWT access token to authenticate with - `disableblocks` Disable block creation and save all attachments individually - `disabletempblocks` Disable temp block creation and upload blocks without waiting for oplog container to be uploaded - `--zen` Zen server instance url to export to / import from - `--target-project` The remote project name (id) (optional, defaults to same as `project`) - `--taret-oplog` The remote oplog name (id) (optional, defaults to same as `olplog`) - `--clean` Delete and create a new oplog before starting export - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint. - `--project` Project name (id) - `--oplog` Project name (id) - `--force` Force upload/download of attachments even if they already exist. - `--file` File system path folder to export to / import from - `--name` File name of oplog output, written into `--file` path - `--cloud` Jupiter service endpoint to export to / import from - `namespace` Name of namespace to store data to - `bucket` Name of bucket to store data to - `key` Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket) - `openid-provider` Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name - `access-token` Optional JWT access token to authenticate with - `access-token-env` - Optional name of environment variable that holds an JWT access token to authenticate with - `--zen` Zen server instance url to export to / import from - `--source-project` The remote project name (id) (optional, defaults to same as `project`) - `--source-oplog` The remote oplog name (id) (optional, defaults to same as `olplog`) - `--clean` Delete and create a new oplog before starting import - Improvement: Faster oplog replay - reduces time to open an existing oplog - Improvement: Clearer error messages and logging when requests to project store fails - Changed: Removed remnants of old mesh experiment - Changed: Remove obsolete export-project command - Changed: Removed remnants import-project command - Changed: Removed unused remote build scripts - Changed: Removed very old and invalid TODO.md - Changed: Removed some deprecated scripts ## 0.2.2 - Feature: Added info (GET) endpoints for structured cache - `/z$` - get a list of namespaces and global info - `/z$/{namespace}` - get list of buckets in a namespace and namespace related info - `/z$/{namespace}/{bucket}` - get bucket info - Feature: Added project store oplog info: `markerpath`, `totalsize`, `opcount`, `expired` on GET requests for oplog - Feature: Added project store project info: `expired` on GET requests for project - Feature :Added project store root route `/prj` which is identical to `/prj/list` - Feature: Zen command line tool `cache-info` to show cache, namespace or bucket info - Feature: Zen command line tool `project-info` to show store, project or oplog info - Feature: Zen command line tool `project-drop` to drop project or oplog - Feature: Zen command line tool `gc` to trigger a GC run - Feature: Zen command line tool `gc-info` to check status of GC - Feature: Added version information to zenserver and zen command line tool executables - Bugfix: Don't log "time to next GC" if time to next GC is not set - Improvement: Don't wait for GC monitor interval before doing first GC check - Improvement: Zen command line tool now fails on any unrecognized arguments - Improvement: Zen command line tool now displays extra help for all sub-commands - Improvement: Host address can now be configured for zen command line tool `drop` command - Improvement: Added precommit xmake task `xmake precommit` to run precommit checks - Changed: Default GC interval set to 1 hour - Changed: Default GC cache duration set to 2 weeks - Changed: Removed HttpLaunchService and related code - Changed: Removed dead/experimental file system related code - Changed: Removed faux vfs config option ## 0.2.1 - Feature: Oplog level GC in project store. If gc marker file path is given by UE, oplogs will be GC:d when marker file is deleted (and GC is triggered) - Bugfix: Index handling for cache large object store was broken resulting in log always being played back - Bugfix: Make sure to flush cache store on call to flush on service and exit - Improvement: Don't write index snapshots if no new entries has been added to log ## 0.2.0 - Feature: Recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request - `/z$/exec$/start-recording?` - `/z$/exec$/stop-recording` - `/z$/exec$/replay-recording?&` - Feature: Disk size triggered GC, a soft disk usage limit for cache data - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging - Improvement: Much improved performance, between 2x to 9x improvement under heavy load (excluding http service overhead). See https://github.com/EpicGames/zen/pull/200 for details - Bugfix: Always store records or oplog entries before storing attachments to avoid GC finding unreferenced chunks i CidStore - Bugfix: Updated Zen `drop` command to support namespaces - Bugfix: Use ZEN_CONSOLE for output to console in Zen commands - Bugfix: Zen `status` command now shows info about found Zen instances - Bugfix: Zen `top` command now shows session id string - Bugfix: On Windows platforms explicitly set the special `SO_EXCLUSIVEADDRUSE` flag as `SO_REUSEADDR` still allows shared use of sockets - Bugfix: Fix logging of number of entries read from caslog at startup - Bugfix: Fix asio http handling of very large/malformed headers and handle mismatching content size - Changed: Reduced GC `INFO` spam by converting to `DEBUG` log messages - Changed: Use Iso8601 format for logging start and end message ## 0.1.9 - Feature: Adds two command to Zen command tool to export/import project store oplogs with attachments - `export-project [oplogs...]` - `import-project [oplogs...]` - Feature: Adds command to query Zen version, specify `host-name` url to query running service version, otherwise you get zen command version. `detailed` option gives you long form version. - `version [host-name] [detailed]` - Feature: New service endpoint to query Zen server version, add `?detailed=true` to get long form version - `/health/version` - Feature: Configure OpenID providers from cmd line and Lua cfg - Feature: Added zen command line executable to release distribution - Bugfix: Fix double reporting of disk usage for namespaces - Bugfix: Fix double garbage collection analisys and garbage collection execution of namespaces - Improvement: Improve tracking of used disk space for filecas and compactcas - Improvement: Add tracking of used disk space for project store - Improvement: Bumped limit for storing cache values as separate files to reduce number of loose files - Improvement: Optimizations when handling compressed buffer (less materialization and reading of headers) - Improvement: Send attachments as file references if the IoBuffer we find represents a complete file and `AcceptFlags` in RPC request allows it. - Improvement: Don't reserve full block size for block store files at creation ## v0.1.8 - Change: Responding with new wire format for RPC requests requires the requestor to add a `Accept` field in the request. This is to allow compatability with older clients for shared instances. - Improvement: Fixed concurrency issues in project store - project and oplog lifetime issues. - Improvement: Don't open oplogs until we require use of them. - Cleanup: Remove rocksdb experimental code. - Feature: Add GC to projects store. Checks path to project file in UE side to determine when a project may be GCd on the Zen side. ## v0.1.7 - Change: All RPC responses are now formatted using dedicated wire format, Zen server has fallback to enable compatability with legacy upstreams - Feature: Adding a `.json` extension to the `--abslog` option will make zenserver log in json format to file - Feature: Create release in Sentry and use `sentry_options_set_release` to associate the executable - Bugfix: CompactBinary: Fixed LoadCompactBinary to gracefully handle read failures and sizes larger than the archive. From http1s://p4-swarm.epicgames.net/changes/21983905 - Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid - Bugfix: Handle edge case when trying to materialize a IoBuffer of zero size via memory mapping - Improvement: Logging: don't do formatting of messages the will not be logged - Improvement: Logging: Timing and upstream source information in upstream logging when debug level logging is enabled - Improvement: Reduce buffer creation and copying in ParsePackageMessage - Improvement: Don't read attachments for oplogs we already have when parsing oplog message ## v0.1.6 - Bugfix: Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid (UE-164966) ## v0.1.5 - Bugfix: Don't fail entire request if GetCacheValue from Horde fails for a single value ## v0.1.4 - Change: Bumped ZEN_SCHEMA_VERSION - this will invalidate entire local cache when deployed - Change: Make CAS storage an hidden implementation detail of CidStore, we no longer hash and do mapping to compressed hash when storing cache values - Feature: Extended zen print command to also handle CbPackage and CompressedBuffer format payloads - Feature: Added /prj/{project}/oplog/{log}/{op} endpoint to allow retrieval of an op entry by LSN. Supports returning CbObject or CbPackage format payloads - Improvement: asio: added some context to error reporting - Improvement: namespace/bucket validation now uses AsciiSet for more efficient validation - Improvement: Frontend: simplified content-type logic - Improvement: Improved message indicating no GC is scheduled - Improvement: Implement proper GetCacheValues upstream path - Improvement: Demote a number of ZEN_ERROR log calls for problems that are recoverable and handled - Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid - Bugfix: Fixed issue in CbPackage marshaling of local reference - Bugfix: Fix crash when switching Zen upstream configured via DNS when one endpoint becomes unresposive - Bugfix: Fixed issue where projects would not be discovered via DiscoverProjects due to use of stem() vs filename() - Bugfix: Use "\\\\?\\" prefixed paths on Windows and fix hardcoded path delimiters (UE-141222) - Bugfix: Safer detection of html folder when running non-bundled executable - Bugfix: Use "application/x-jupiter-inline" to fetch GetCacheValues from Horde (UE-162151) - Sentry: Added logging of sentry_init error code - Sentry: Attach log file to Sentry error reports - Sentry: Capture capture error/critical log statements as errors in Sentry - Update: Bump VCPKG version to 2022.08.15 - CI: MacOS build enable again in GitHub CI - CI: Upload debug info and source files to Sentry when release is created ## v0.1.3 -- Enable adding namespace to URI based upstream requests -- Add logging of namespace name and bucket name if we get invalid names in requests -- Updated README.md with Linux dev prerequisites -- asio: added some logging to indicate concurrency -- Fixed expired cache keys overwriting between namespaces when bucket names were the same in multiple namespaces ## v0.1.2 - Tweak bundle compression settings to streamline build - ZenCacheDiskLayer::CacheBucket::GatherReferences: Don't hold index lock while reading standalone values - hardening of ZenCacheDiskLayer::CacheBucket::PutStandaloneCacheValue - GitHub Actions: Move release job to in-house linux agent ## v0.1.1 - BlockStore (small object store) Always block GC of current write block - Make it possible to configure GC monitoring interval using `--gc-monitor-interval-seconds` - Keep "reason" from upstream response so we can present it even if the request fails without outright error - New GitHub Actions release flow - Add release flow in GitHub actions on pushed tag `v0.1.2` gives full release, `v0.1.2-pre0` gives pre-release ## 0d08450 - Fixes issue with broken Zen instances for legacy requests ## 63f50b5 - Enable FILE_SHARE_DELETE on standalone files in disk buckets - fixes Jira UE-154234 - Make sure we can properly create the block file before assigning it for use - fixes Jira UE-154438 - Horde execute compressed input blobs - Drop namespace support - Safer delete of cache buckets ## dba8b36 - Namespaces: This introduces namespaces to the zenserver but only the default ue4.ddc is supported. Clients that don't send a namespace in the request will keep old behviour, new clients that sends namespace is required to use ue4.ddc (which they currently do) - Aligned bucket naming rules with UE code base - Fix retry counter and add an extra iteration to give more time for success during contention for standalone files in cache - Make sure CacheBucket::PutStandaloneCacheValue cleans up the temp file - Restore logic where we accept failed overwrite if resulting size is the same for standlone file in cache - Correctly calculate the m_TotalSize difference when overwriting file for standalone files in cache - Fix namespace folder scanning