| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Added global zenserver option `--cache-bucket-limit-overwrites` controlling Whether to require policy flag pattern before allowing overwrites or not. Default `false` = overwrites always allowed
- Feature: Add per bucket cache configuration option `limitoverwrites` (Lua options file only)
cache = {
bucket = { -- This is the default for all namespaces
limitoverwrites = true
},
buckets = { -- Here you can add matching per bucket name (matches accross namespaces)
iostorecompression = {
limitoverwrites = false
},
},
}
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Feature: Added `--sentry-environment` to `zen` and `zenserver`
- Feature: Added `--sentry-debug` to `zen` and `zenserver`
- Feature: Added environment variable parsing for the following options:
- `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted)
- `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug`
- `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info`
- `UE_ZEN_SENTRY_DSN`: `--sentry-dsn`
- `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
|
| | | |
|
| | |
| |
| |
| |
| | |
add env-variable parsing for sentry option
|
| | |
| |
| |
| |
| | |
reporting endpoint (#427)
moved sentry database path to temporary directory for zen commandline
|
| | |
| |
| |
| | |
Feature: Add per bucket cache configuration (Lua options file only)
Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
|
| | | |
|
| | |
| |
| | |
- Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
- **EXPERIMENTAL** `zen builds`
- Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata
- Feature: New `/builds` endpoint for caching build blobs and blob metadata
- `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs
- `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs
- `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs
- `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future.
To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks.
This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`:
* `mimalloc` - mimalloc (default, same as before)
* `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows.
* `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows.
* `ansi` - fallback to default system allocator
|
| |
|
|
|
|
| |
- Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default.
- Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default.
- Added more details in post GC log.
- Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
|
| |
|
|
|
|
|
| |
- Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled.
- Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace
- A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled
- A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces
- A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
|
| |
|
| |
* zen command - add options to control meta data cache when triggering gc
|
| |
|
|
|
| |
Added option `gc-attachment-passes` to zenserver
Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file
Fixed project store not properly sorting references found during lock phase
|
| |
|
|
| |
- Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false`
- Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
|
| |
|
|
|
| |
added `--detach` option to zenserver. When this is passed in with a false value, we do not create a new process group in order to behave more as expected when running with `xmake run zenserver`. Without this change the zenserver process does not receive any signals and won't exit when xmake does, causing processes to linger in the background.
The default behaviour (when run from UE) is unchanged.
|
| |
|
|
|
|
| |
* add support for indenting callstack output
* Explicitly catch option-parse error and reduce log spam on bad parameters
* add command line to sentry error reports
* log command line at startup
|
| |
|
| |
* add option to force gcv2 to run single threaded
|
| |
|
| |
Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
|
| |
|
| |
* remove reference caching
|
| |
|
|
|
| |
this also adds a central, shared folder for storing information which may be found by any instance on the host. The directory is currently located alongside the default install and state directory.
Initially this is used to store a collection of known `root_manifest` locations and a copy of the latest manifest version which allow us to find all known locations where zen state is present.
|
| |
|
|
|
|
| |
- Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks)
- Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command
- Improvement: Implemented oplog scrubbing (previously was a no-op)
- Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
|
| |
|
|
|
|
|
| |
with these changes it is possible to configure loggers on the command line. For instance:
`xmake run zenserver --log-trace=http_requests,http`
will configure the system so that the `http_request` and `http` loggers are set to TRACE level
|
| |
|
|
|
| |
* added --powercycle option. when this is passed in the zenserver process will shut down immediately after initialization is complete. This is primarily useful when benchmarking init/cleanup but could also be used to verify/clean up disk state
* moved EmptyStandbyList code to make it accessible to more commands
|
| |
|
|
|
|
|
|
|
|
|
| |
- Feature: Added new options to zenserver for GC V2
- `--gc-compactblock-threshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
- `--gc-verbose` GCV2 - enable more verbose output when running a GC pass
- Feature: Added new options to `zen gc` command for GC V2
- `--compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
- `--verbose` GCV2 - enable more verbose output when running a GC pass
- Feature: Added new parameters for endpoint `admin/gc` (PUT)
- `compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
- `verbose` GCV2 - enable more verbose output when running a GC pass
|
| |
|
|
|
| |
this introduces a --snapshot-dir command line option to zenserver which specifies a directory which will be propagated to the persistence root directory on start-up.
This is most powerful with file systems which support block cloning, such as ReFS on Windows. This allows even very large state snapshots to be used repeatedly without having to worry about mutating the original dataset on disk. When using ReFS the state copy for even large state directories can be very fast since the duration is primarily proportional to the number of files in the tree rather than the size of the files being cloned. The storage requirements are also minimal as all data will be handled in a copy-on-write manner.
|
| |
|
| |
- Feature: New garbage collection implementation, still in evaluation mode. Enabled by `--gc-v2` command line option
|
| |
|
|
|
| |
added support for reporting metrics via statsd style UDP messaging, which is supported by many monitoring solution providers
this change adds reporting only of three cache related metrics (hit/miss/put) but this should be extended to include more metrics after additional evaluation
|
| |
|
|
| |
- Feature: Added `--cache-memlayer-sizethreshold` option to zenserver to control at which size cache entries get cached in memory
- Changed: Merged cache memory layer with cache disk layer to reduce memory and cpu overhead
|
| |
|
| |
when specified with `--clean`, the data directory will be wiped clean at startup
|
| |
|
|
| |
Changes the default port without altering config for shared instances.
|
| |
|
|
|
| |
* removed legacy compute code, which will be replaced with a new implementation in the future
* also updated references to Jupiter storage
|
| |
|
|
|
| |
- Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC
- disabled by default, enable with `--cache-reference-cache-enabled`
- Improvement: Faster collection of referenced CId content in project store
|
| | |
|
| |
|
|
| |
added --quiet option for zenserver to suppress default logging to stdout
|
| |
|
| |
removed websocket support since it is not used right now and is unlikely to be used in the future
|
| |
|
|
|
|
|
|
| |
- Feature: Limit the size ZenCacheMemoryLayer may use
- `--cache-memlayer-targetfootprint` option to set which size (in bytes) it should be limited to, zero to have it unbounded
- `--cache-memlayer-maxage` option to set how long (in seconds) cache items should be kept in the memory cache
Do more "standard" GC rather than clearing everything.
Tries to purge memory on Get/Put on the fly if exceeding limit - not sure if we should have a polling thread instead of adding overhead to Get/Put (however light it may be).
|
| |
|
|
|
|
| |
- Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store
- Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false"
- Add `--skipcid` option to `zen gc` command, defaults to false
- Add `--gc-lightweight-interval-seconds` option to zenserver
|
| |
|
|
| |
* Added `--http-threads`, `--httpsys-async-work-threads`, `--httpsys-enable-request-logging` and `--httpsys-enable-async-response` command line options to zenserver
* remove unused CreateHttpSysServer
|
| |
|
| |
* add `cache-write-log` and `cache-access-log´ configuration options
|
| |
|
|
|
|
| |
* Add `--gc-projectstore-duration-seconds` option
* Cleanup lua gc options parsing
* Remove dead configuration values
* changelog
|
| |
|
|
| |
* - Feature: Disk writes are now blocked early and return an insufficient storage error if free disk space falls below the `--low-diskspace-threshold` value
* Never keep an entry in m_ChunkBlocks that points to a nullptr
|
| |
|
|
| |
* add ip and username to sentry reports if allowed in settings
* add --sentry-allow-personal-info command line options to zenserver
|
|
|
* moved source directories into `/src`
* updated bundle.lua for new `src` path
* moved some docs, icon
* removed old test trees
|