## - Bugfix: Verify that there are blocks to GC for block store garbage collect (void division by zero) - Bugfix: Write log error and flush log before reporting error to Sentry/error logger - Bugfix: Log ERROR in scope guard if function throws exception, throwing exception causes application abort ## 0.2.7 - Bugfix: Safely handle missing blocks when doing garbage collection in block store data - Bugfix: Only strip uri accept type suffix if it can be parsed to a known type - Bugfix: Keep system error code on Windows when file mapping fails and propagate to log/exception - Bugfix: Catch any errors throw in HttpAsioServer() destructor and log error. ## 0.2.6 - Strip __FILE__ macro names in logging to only include the file name as to not expose file paths of the machine building the executable - Bugfix: Reporting the correct callstack to sentry on ERROR/CRITICAL failure. - Extend sentry message with triggering file/line/function ## 0.2.5 - Feature: Zen command line tool `rpc-record-start` to record all RPC requests to the structured cache - `--path` Recording file path where the rpc requests will be stored - Feature: Zen command line tool `rpc-record-stop` stop the currently active RPC request recording started with `rpc-record-start` - Feature: Zen command line tool `rpc-record-replay` replacy a RPC request recording created with `rpc-record-start` - `--path` Recording file path where the rpc requests are stored - `--numthreads` Number of worker threads to use while replaying the RPC requests - `--numproc` Number of worker processes to run, if more than one new processes will be spawn with `` workers each - `--offset` Offset into request playback to start at - `--stride` The stride to use when selecting requests to playback - `--onhost` Replay the recording inside the zenserver bypassing http overhead - `--showmethodstats` Show statistics of which RPC methods are used - `--forceallowlocalrefs` Force the requests to allow local references (file path/file handle) - `--disablelocalrefs` Force disable local references in request (file path/file handle) - `--forceallowlocalhandlerefs` Force the requests to allow local references via duplicated file handles for requests that allow local refs - `--disablelocalhandlerefs` Force disable local references via duplicated file handles in requests - `--forceallowpartiallocalref` Force the requests to allow local references for files that are not saved as whole files for requests that allow local refs - `--disablepartiallocalrefs` Force disable local references for files that are not saved as whole files for requests that allow local refs - Feature: Zen command line tool `cache-stats` to give stats result about the zen cache - Feature: Zen command line tool `project-stats` to give stats result about the zen project store - Feature: Zen command line tool `cache-details` to give detail result about the zen cache, defaults to overview information about the cache - `--namespace` Get information about cache values in a namespace - `--bucket` Get information about cache values limited to a specific bucket in a namespace - `--valuekey` Get information about a cache value in a specific bucket in a namespace, valuekey is specified as IoHash hex string - `--details` Get detailed information about each cache record - `--attachmentdetails` Get detailed information about each attachments for each cache record - `--csv` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: Zen command line tool `project-details` to give detail result about the zen project store, defaults to overview information about the project store - `--project` The project id to get information about - `--oplog` The oplog id to get information about - `--opid` The op Oid to get information about - `--details` Get detailed information about the op - `--opdetails` Extract the entire op information (not available in CSV output) - `--attachmentdetails` Get detailed information about each attachments for each op - `--csv` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: New project store stats endpoint `/stats/prj` to get stats info for zen project store - Feature: New project store details endpoints `/prj/details$`, `/prj/details$/{project}`, `/prj/details$/{project}/{oplog}`, `/prj/details$/{project}/{oplog}/{op}` to give detail result about the zen project store, defaults to overview information about the project store items - `details=true` Get detailed information about the op - `opdetails=true` Extract the entire op information - `attachmentdetails=true` Get detailed information about each attachments for each op - `csv=true` Format the output as a comma delimited CSV file. If not specified it defaults to JSon style response. - Feature: New cache detail endpoints `/z$/details$`, `/z$/details$/{namespace}`, `/z$/details$/{namespace}/{bucket}`, `/z$/details$/{namespace}/{bucket}/{key}` has been added - `details=true` Get detailed information about each cache record - `attachmentdetails=true` Get detailed information about each attachments for each cache record - `csv=true` Format the response as a comma delimited CSV file. If not specified it defaults to CbObject but can auto-format to json - Feature: `--junit` switch to `xmake test` to generate junit style reports of tests. - Feature: CI build on GitHub now uploads junit test reports as artifact to the check for PR validation and mainline validation - Feature: Payloads from zenserver can now be sent using duplicated file handles if caller requests provides client ProcessId (Windows only). - Feature: Add `--port` option to zen down command to shut down servers on different base ports - Bugfix: Make sure async responses are sent async correctly in httpsys - Bugfix: Don't delete manifest file in cas root when initializing a new filecas folder - Bugfix: Sentry does not like UNC paths, so strip the prefix before passing them to sentry - Bugfix: Make sure zen down command uses the correct port for shutdown event - Improvement: FileCas now keeps an up to date index of all the entries improving performance when getting cache misses on large payloads - Improvement: Structured cache now keeps RawHash and RawSize in memory avoiding materialization of cache values before sending response - Changed: Exit with failure code on port conflict rather than reporting crash to Sentry - Changed: removed catch2 support for now since it does not handle multithreaded tests - Bugfix: fixed bug in dashboard content serving (see PR #255) ## 0.2.4 - Bugfix: Don't send empty http responses with content type set to Text. Fixes UE-177895 ## 0.2.3 - Feature: Add support for "packagedata" mapping in oplog entries - Feature: Zen command line tool `project-create` to create a project store project - `--project` Project name (id) - `--rootdir` Absolute path to root directory (optional) - `--enginedir` Absolute path to engine root directory (optional) - `--projectdir` Absolute path to project directory (optional) - `--projectfile` Absolute path to .uproject file (optional) - Feature: Zen command line tool `oplog-create` to create a project store oplog - `--project` Project name (id) - `--oplog` Oplog name (id) - `--gcpath` Absolute path to oplog lifetime marker file (optional) - Feature: Build scripts and tooling to build zen compliant with VFX reference platform CY2022/2021 matching UE linux builds - Feature: added `xmake sln` task which replaces `generate_projects.bat` - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: ```json { "method" : "", "params" : { "maxblocksize": "", "maxchunkembedsize": "", "file" : { "path" : "", "name" : "" }, "cloud" : { "url" : "", "namespace" : "", "bucket" : "", "key" : "", "openid-provider" : "", "access-token" : "", "access-token-env": ", "disableblocks" : "", "disabletempblocks" : "" }, "zen" : { "url" : "", "project" : "", "oplog" : "" } } } ``` - `"method"`supported methods are `"export"` and `"import"` to import/export an oplog - `"params"` container for parameters - `"maxblocksize"` - Optional. The maximum size of a block of attachments, default 134217728 (128 Mb) (export only) - `"maxchunkembedsize"` - Optional. The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments, default 1048576 (1Mb) (export only) - `"force"` - Optional. Boolean flag to indicate weather attachments should be uploaded/downloaded disregarding prior existance - External location types are "file" (File system), "cloud" (UE Cloud Storage service) or "zen" (Zen server instance), provide one of those as remote location. - `"file"` - Optional. Indicates remote location is the local file system - `"path"` - File system path folder to export to / import from - `"name"` - File name of oplog output, written into - `"cloud"` - Optional. Indicates remote location is UE Cloud Storage service - `"url"` - Jupiter service endpoint url - `"namespace"` - Name of namespace to store data to - `"bucket"` - Name of bucket to store data to - `"key"` - IoHash key to the stored oplog container - `"openid-provider"` - Optional. Name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for - `"access-token"` - Optional. JWT access token to authenticate with - `"access-token-env"` - Optional. Name of environment variable that holds an JWT access token to authenticate with - `"disableblocks"` - Optional. Disable creation of attachments blocks - "true"/"false" (export only) - `"disabletempblocks"` - Optional. Disable creation of attachments temp blocks forcing upload before oplog container - "true"/"false" (export only) - `"zen"` - Optional. Indicates remote location is a Zen server instance - `"url"` - Zen server instance url - `"project"` - The remote project name (id) - `"oplog" - The remote oplog name (id) - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. ```json { "method" : "getchunks", "chunks" : [ "", ] } ``` - CbPackage RPC format for `putchunks` method, attachments are stored in CidStore ```json { "method" : "putchunks", } ``` - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - `--project` Project name (id) - `--oplog` Project name (id) - `--maxblocksize` The maximum size of a block of attachments (optional) - `--maxchunkembedsize` The maximum size of an attachment to be put in a block, larger attachments will be stored as usual attachments (optional) - `--force` Force upload/download of attachments even if they already exist. - `--file` File system path folder to export to / import from - `--name` File name of oplog output, written into `--file` path - `--disableblocks` Disable block creation and save all attachments individually - `--forcetempblocks` Force creation of temp attachment blocks - `--cloud` Jupiter service endpoint to export to / import from - `namespace` Name of namespace to store data to - `bucket` Name of bucket to store data to - `key` Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket) - `openid-provider` Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name - `access-token` Optional JWT access token to authenticate with - `access-token-env` - Optional name of environment variable that holds an JWT access token to authenticate with - `disableblocks` Disable block creation and save all attachments individually - `disabletempblocks` Disable temp block creation and upload blocks without waiting for oplog container to be uploaded - `--zen` Zen server instance url to export to / import from - `--target-project` The remote project name (id) (optional, defaults to same as `project`) - `--taret-oplog` The remote oplog name (id) (optional, defaults to same as `olplog`) - `--clean` Delete and create a new oplog before starting export - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint. - `--project` Project name (id) - `--oplog` Project name (id) - `--force` Force upload/download of attachments even if they already exist. - `--file` File system path folder to export to / import from - `--name` File name of oplog output, written into `--file` path - `--cloud` Jupiter service endpoint to export to / import from - `namespace` Name of namespace to store data to - `bucket` Name of bucket to store data to - `key` Key to the stored oplog container (If omitted a default key will be generated based on project/oplog/namespace/bucket) - `openid-provider` Optional name of openid provider used to authenticate with, requires that the zen server instance has been provided with a oids refresh token for the provider name - `access-token` Optional JWT access token to authenticate with - `access-token-env` - Optional name of environment variable that holds an JWT access token to authenticate with - `--zen` Zen server instance url to export to / import from - `--source-project` The remote project name (id) (optional, defaults to same as `project`) - `--source-oplog` The remote oplog name (id) (optional, defaults to same as `olplog`) - `--clean` Delete and create a new oplog before starting import - Improvement: Faster oplog replay - reduces time to open an existing oplog - Improvement: Clearer error messages and logging when requests to project store fails - Changed: Removed remnants of old mesh experiment - Changed: Remove obsolete export-project command - Changed: Removed remnants import-project command - Changed: Removed unused remote build scripts - Changed: Removed very old and invalid TODO.md - Changed: Removed some deprecated scripts ## 0.2.2 - Feature: Added info (GET) endpoints for structured cache - `/z$` - get a list of namespaces and global info - `/z$/{namespace}` - get list of buckets in a namespace and namespace related info - `/z$/{namespace}/{bucket}` - get bucket info - Feature: Added project store oplog info: `markerpath`, `totalsize`, `opcount`, `expired` on GET requests for oplog - Feature: Added project store project info: `expired` on GET requests for project - Feature :Added project store root route `/prj` which is identical to `/prj/list` - Feature: Zen command line tool `cache-info` to show cache, namespace or bucket info - Feature: Zen command line tool `project-info` to show store, project or oplog info - Feature: Zen command line tool `project-drop` to drop project or oplog - Feature: Zen command line tool `gc` to trigger a GC run - Feature: Zen command line tool `gc-info` to check status of GC - Feature: Added version information to zenserver and zen command line tool executables - Bugfix: Don't log "time to next GC" if time to next GC is not set - Improvement: Don't wait for GC monitor interval before doing first GC check - Improvement: Zen command line tool now fails on any unrecognized arguments - Improvement: Zen command line tool now displays extra help for all sub-commands - Improvement: Host address can now be configured for zen command line tool `drop` command - Improvement: Added precommit xmake task `xmake precommit` to run precommit checks - Changed: Default GC interval set to 1 hour - Changed: Default GC cache duration set to 2 weeks - Changed: Removed HttpLaunchService and related code - Changed: Removed dead/experimental file system related code - Changed: Removed faux vfs config option ## 0.2.1 - Feature: Oplog level GC in project store. If gc marker file path is given by UE, oplogs will be GC:d when marker file is deleted (and GC is triggered) - Bugfix: Index handling for cache large object store was broken resulting in log always being played back - Bugfix: Make sure to flush cache store on call to flush on service and exit - Improvement: Don't write index snapshots if no new entries has been added to log ## 0.2.0 - Feature: Recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request - `/z$/exec$/start-recording?` - `/z$/exec$/stop-recording` - `/z$/exec$/replay-recording?&` - Feature: Disk size triggered GC, a soft disk usage limit for cache data - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging - Improvement: Much improved performance, between 2x to 9x improvement under heavy load (excluding http service overhead). See https://github.com/EpicGames/zen/pull/200 for details - Bugfix: Always store records or oplog entries before storing attachments to avoid GC finding unreferenced chunks i CidStore - Bugfix: Updated Zen `drop` command to support namespaces - Bugfix: Use ZEN_CONSOLE for output to console in Zen commands - Bugfix: Zen `status` command now shows info about found Zen instances - Bugfix: Zen `top` command now shows session id string - Bugfix: On Windows platforms explicitly set the special `SO_EXCLUSIVEADDRUSE` flag as `SO_REUSEADDR` still allows shared use of sockets - Bugfix: Fix logging of number of entries read from caslog at startup - Bugfix: Fix asio http handling of very large/malformed headers and handle mismatching content size - Changed: Reduced GC `INFO` spam by converting to `DEBUG` log messages - Changed: Use Iso8601 format for logging start and end message ## 0.1.9 - Feature: Adds two command to Zen command tool to export/import project store oplogs with attachments - `export-project [oplogs...]` - `import-project [oplogs...]` - Feature: Adds command to query Zen version, specify `host-name` url to query running service version, otherwise you get zen command version. `detailed` option gives you long form version. - `version [host-name] [detailed]` - Feature: New service endpoint to query Zen server version, add `?detailed=true` to get long form version - `/health/version` - Feature: Configure OpenID providers from cmd line and Lua cfg - Feature: Added zen command line executable to release distribution - Bugfix: Fix double reporting of disk usage for namespaces - Bugfix: Fix double garbage collection analisys and garbage collection execution of namespaces - Improvement: Improve tracking of used disk space for filecas and compactcas - Improvement: Add tracking of used disk space for project store - Improvement: Bumped limit for storing cache values as separate files to reduce number of loose files - Improvement: Optimizations when handling compressed buffer (less materialization and reading of headers) - Improvement: Send attachments as file references if the IoBuffer we find represents a complete file and `AcceptFlags` in RPC request allows it. - Improvement: Don't reserve full block size for block store files at creation ## v0.1.8 - Change: Responding with new wire format for RPC requests requires the requestor to add a `Accept` field in the request. This is to allow compatability with older clients for shared instances. - Improvement: Fixed concurrency issues in project store - project and oplog lifetime issues. - Improvement: Don't open oplogs until we require use of them. - Cleanup: Remove rocksdb experimental code. - Feature: Add GC to projects store. Checks path to project file in UE side to determine when a project may be GCd on the Zen side. ## v0.1.7 - Change: All RPC responses are now formatted using dedicated wire format, Zen server has fallback to enable compatability with legacy upstreams - Feature: Adding a `.json` extension to the `--abslog` option will make zenserver log in json format to file - Feature: Create release in Sentry and use `sentry_options_set_release` to associate the executable - Bugfix: CompactBinary: Fixed LoadCompactBinary to gracefully handle read failures and sizes larger than the archive. From http1s://p4-swarm.epicgames.net/changes/21983905 - Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid - Bugfix: Handle edge case when trying to materialize a IoBuffer of zero size via memory mapping - Improvement: Logging: don't do formatting of messages the will not be logged - Improvement: Logging: Timing and upstream source information in upstream logging when debug level logging is enabled - Improvement: Reduce buffer creation and copying in ParsePackageMessage - Improvement: Don't read attachments for oplogs we already have when parsing oplog message ## v0.1.6 - Bugfix: Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid (UE-164966) ## v0.1.5 - Bugfix: Don't fail entire request if GetCacheValue from Horde fails for a single value ## v0.1.4 - Change: Bumped ZEN_SCHEMA_VERSION - this will invalidate entire local cache when deployed - Change: Make CAS storage an hidden implementation detail of CidStore, we no longer hash and do mapping to compressed hash when storing cache values - Feature: Extended zen print command to also handle CbPackage and CompressedBuffer format payloads - Feature: Added /prj/{project}/oplog/{log}/{op} endpoint to allow retrieval of an op entry by LSN. Supports returning CbObject or CbPackage format payloads - Improvement: asio: added some context to error reporting - Improvement: namespace/bucket validation now uses AsciiSet for more efficient validation - Improvement: Frontend: simplified content-type logic - Improvement: Improved message indicating no GC is scheduled - Improvement: Implement proper GetCacheValues upstream path - Improvement: Demote a number of ZEN_ERROR log calls for problems that are recoverable and handled - Bugfix: Use bucket/key to get inline value in upstream for chunks without a chunkid - Bugfix: Fixed issue in CbPackage marshaling of local reference - Bugfix: Fix crash when switching Zen upstream configured via DNS when one endpoint becomes unresposive - Bugfix: Fixed issue where projects would not be discovered via DiscoverProjects due to use of stem() vs filename() - Bugfix: Use "\\\\?\\" prefixed paths on Windows and fix hardcoded path delimiters (UE-141222) - Bugfix: Safer detection of html folder when running non-bundled executable - Bugfix: Use "application/x-jupiter-inline" to fetch GetCacheValues from Horde (UE-162151) - Sentry: Added logging of sentry_init error code - Sentry: Attach log file to Sentry error reports - Sentry: Capture capture error/critical log statements as errors in Sentry - Update: Bump VCPKG version to 2022.08.15 - CI: MacOS build enable again in GitHub CI - CI: Upload debug info and source files to Sentry when release is created ## v0.1.3 -- Enable adding namespace to URI based upstream requests -- Add logging of namespace name and bucket name if we get invalid names in requests -- Updated README.md with Linux dev prerequisites -- asio: added some logging to indicate concurrency -- Fixed expired cache keys overwriting between namespaces when bucket names were the same in multiple namespaces ## v0.1.2 - Tweak bundle compression settings to streamline build - ZenCacheDiskLayer::CacheBucket::GatherReferences: Don't hold index lock while reading standalone values - hardening of ZenCacheDiskLayer::CacheBucket::PutStandaloneCacheValue - GitHub Actions: Move release job to in-house linux agent ## v0.1.1 - BlockStore (small object store) Always block GC of current write block - Make it possible to configure GC monitoring interval using `--gc-monitor-interval-seconds` - Keep "reason" from upstream response so we can present it even if the request fails without outright error - New GitHub Actions release flow - Add release flow in GitHub actions on pushed tag `v0.1.2` gives full release, `v0.1.2-pre0` gives pre-release ## 0d08450 - Fixes issue with broken Zen instances for legacy requests ## 63f50b5 - Enable FILE_SHARE_DELETE on standalone files in disk buckets - fixes Jira UE-154234 - Make sure we can properly create the block file before assigning it for use - fixes Jira UE-154438 - Horde execute compressed input blobs - Drop namespace support - Safer delete of cache buckets ## dba8b36 - Namespaces: This introduces namespaces to the zenserver but only the default ue4.ddc is supported. Clients that don't send a namespace in the request will keep old behviour, new clients that sends namespace is required to use ue4.ddc (which they currently do) - Aligned bucket naming rules with UE code base - Fix retry counter and add an extra iteration to give more time for success during contention for standalone files in cache - Make sure CacheBucket::PutStandaloneCacheValue cleans up the temp file - Restore logic where we accept failed overwrite if resulting size is the same for standlone file in cache - Correctly calculate the m_TotalSize difference when overwriting file for standalone files in cache - Fix namespace folder scanning