| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
| |
this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver
this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes
to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system
all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Added `--security-config-path` option to zenserver to configure security settings
- Expects a path to a .json file
- Default is an empty path resulting in no extra security settings and legacy behavior
- Current support is a top level filter of incoming http requests restricted to the `password` type
- `password` type will check the `Authorization` header and match it to the selected authorization strategy
- Currently the security settings is very basic and configured to a fixed username+password at startup
{
"http" {
"root": {
"filter": {
"type": "password",
"config": {
"password": {
"username": "<username>",
"password": "<password>"
},
"protect-machine-local-requests": false,
"unprotected-uris": [
"/health/",
"/health/info",
"/health/version"
]
}
}
}
}
}
|
| |
|
|
|
|
|
| |
* add system metrics output to top command
* removed unnecessary xmake directives
* file system API/comment tweaks
* fixed out-of-range access in httpserver test
* updated ZenServer base API to allow customization by mode
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder.
It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build.
* Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist
* BLAKE3 is now in-tree at `thirdparty/blake3`
* cpr is now in-tree at `thirdparty/cpr`
* cxxopts is now in-tree at `thirdparty/cxxopts`
* fmt is now in-tree at `thirdparty/fmt`
* robin-map is now in-tree at `thirdparty/robin-map`
* ryml is now in-tree at `thirdparty/ryml`
* sol2 is now in-tree at `thirdparty/sol2`
* spdlog is now in-tree at `thirdparty/spdlog`
* utfcpp is now in-tree at `thirdparty/utfcpp`
* xmake package repo definitions is in `repo`
* implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS
* the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux
* crashpad
* bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
|
| |
|
| |
this breaks out the configuration logic to allow multiple applications to share common configuration and initialization logic whilst customizing chosen aspects of the process
|
| |
|
|
| |
this change moves common base service code into `zenserver.(cpp|h)` and storage server code into `zenstorageserver.(cpp|h)` to more clearly separate concerns but also to make way for another service
|
| |
|
|
|
| |
Separates storage server code and generic server code into two classes.
This is a change to prepare for different services to be implemented using the same framework, into the same executable
|
| |
|
| |
* move zen vfs implementation to zenstore
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
- Refactor so we can have more than one cas store for project store and cache.
- Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore
- Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub).
- Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize)
- Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
|
| |
|
|
|
|
| |
- Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage
- **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed
- Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
|
| |
|
|
|
|
|
|
|
| |
- **EXPERIMENTAL** `zen builds`
- Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata
- Feature: New `/builds` endpoint for caching build blobs and blob metadata
- `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs
- `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs
- `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs
- `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
|
| |
|
|
| |
* move openprocess cache to central location
* enable openprocesscache in projectstore so "getchunks" can send filehandles when requested
|
| |
|
| |
Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
|
| |
|
|
|
|
|
|
| |
this change addresses some TSAN warnings for improved robustness and less TSAN noise
- Added dedicated timer for EnqueueStateExitFlagTimer
- Made log formatter `fullformatter` output consistent time stamps across threads
- Made Linux/Mac event implementation TSAN clean
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: `zen up` command improvements
- --`port` allows you to specify a base port when starting an instance
- --`base-dir` allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable
- Feature: `zen down`
- --`port` allows you to specify a base port when shutting down an instance
- --`base-dir` allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable
- --`force` if regular shutdown fails it tries to find a running zenserver.exe process and terminate it
- If it fails to attach to the running server it now waits for it to exit when setting the RequestExit shared memory flag
- Improvement: zenserver now checks the RequestExit flag in the shared memory and exist gracefully if it is set
- Improvement: When adding a sponsor process to a running zenserver instance, we wait for it to be picked up from the shared memory section to determine success/fail
|
| |
|
|
|
| |
- Improvement: Delay exiting due to no sponsor processes by one second to handle race conditions
- Improvement: Safer IsProcessRunning check
- Improvement: make sure we can RequestApplicationExit safely from any thread
|
| |
|
|
|
|
|
| |
- Improvement: Add zenhttp-test and zenutil-test
- Improvement: Moved cachepolicy test to cachepolicy.cpp
- Improvement: Renamed cachestore tests from z$ to cachestore
- Improvement: Moved test linking so test for a lib is linked by <lib>-test
- Improvement: Removed HttpRequestParseRelativeUri in httpstructuredcache.cpp and use the one in cacherequests.h instead
|
| |
|
|
|
|
| |
* moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore
* move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp
* introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
|
| | |
|
| |
|
|
|
|
| |
- Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks)
- Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command
- Improvement: Implemented oplog scrubbing (previously was a no-op)
- Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
|
| | |
|
| |
|
|
|
| |
* added --powercycle option. when this is passed in the zenserver process will shut down immediately after initialization is complete. This is primarily useful when benchmarking init/cleanup but could also be used to verify/clean up disk state
* moved EmptyStandbyList code to make it accessible to more commands
|
| |
|
|
|
|
|
|
|
| |
this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future.
As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link.
Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future.
The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
|
| |
|
|
|
| |
added support for reporting metrics via statsd style UDP messaging, which is supported by many monitoring solution providers
this change adds reporting only of three cache related metrics (hit/miss/put) but this should be extended to include more metrics after additional evaluation
|
| | |
|
| |
|
| |
separating the http server implementations into a directory and moved diagsvcs into zenserver since it's somewhat hard-coded for it
|
| |
|
|
|
| |
* removed legacy compute code, which will be replaced with a new implementation in the future
* also updated references to Jupiter storage
|
| |
|