aboutsummaryrefslogtreecommitdiff
path: root/src/zen/zen.cpp
Commit message (Collapse)AuthorAgeFilesLines
* "unlimited" line length when using plain progress (#347)Dan Engelbrecht2025-04-041-7/+19
|
* progress bar improvements (#346)Dan Engelbrecht2025-04-041-18/+82
| | | | * hide ETA until at least 5% is complete * dynamically adjust progres output length
* fix quoted command lines arguments (#306)Dan Engelbrecht2025-03-121-0/+18
| | | Handling of quotes and quotes with leading backslash for command line parsing - UE-231677
* ProgressBar improvements (#305)Stefan Boberg2025-03-121-2/+41
| | | | | * changed ProgressBar so it doesn't use printf. printf by default is very slow on Windows due to weird buffering behaviour. During a 2 minute build download I profiled 35 CPU seconds inside printf * changed so ProgressBar uses plain output mode if stdout is not a console/tty
* improved block gen logic (#302)Dan Engelbrecht2025-03-121-0/+3
| | | | | | | - Improvement: Reduced memory usage during upload and part upload validation - Improvement: Reduced I/O usage during upload and download - Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild - Improvement: More trace scopes for build upload operations - Bugfix: Fixed crash during download when trying to write outside a file range
* Build command tweaks (#301)Stefan Boberg2025-03-111-33/+20
| | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* Add trace support for zen CLI command (#296)Stefan Boberg2025-03-051-0/+54
| | | | - This change adds support for `--trace`, `--tracehost` and `--tracefile` command arguments to enable and control tracing to Insights - It also adds profiling scopes primarily to build download command related code
* improve error handling (#289)Dan Engelbrecht2025-02-281-1/+1
| | | | | | | | * clearer errors * quicker abort * handle deleted local files * simplify parallellwork error handling * don't finish progress on destructor - gives wrong impression * graceful ctrl-c handling
* builds upload command (#278)Dan Engelbrecht2025-02-261-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-4/+16
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* Insights-compatible memory tracking (#214)Stefan Boberg2024-11-251-7/+1
| | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-151-1/+3
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* project details cmd resolve prj oplog (#209)Dan Engelbrecht2024-11-061-2/+2
| | | | | * auto-resolve project name if only one project exist * always try to resolve project and oplog name in project-details command * project-details -> project-op-details
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-061-0/+101
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* cache get command (#183)Dan Engelbrecht2024-10-031-0/+2
| | | | * move TryParseObjectId and TryParseIoHash to Oid::TryParse and IoHash::TryParse respectively * zen cache-get command
* cache generate command to create large data sets for testing (#159)Dan Engelbrecht2024-09-181-0/+2
| | | * add CacheGenerateCommand
* oplog cmd improvements (#152)Dan Engelbrecht2024-09-111-4/+0
| | | | | | - Improvement: Removed redundant commands `project-delete` and `oplog-delete`. Use already existing `project-drop` instead. - Improvement: zen oplog commands `project-drop`, `project-info`, `oplog-create`, `oplog-import`, `oplog-mirror` can now help resolve partial project and oplog identifiers - Improvement: zen `oplog-mirror` command now has new filter options to control which files are realized to disk: `--key` for op key, `--file` for file path matching and `--chunk` for chunk id matching - Improvement: `project-drop` command defaults to `--dry-run=true` and will only delete the target if `--dry-run=false` is added to the command line to avoid accidental delete
* workspaces review feedbackDan Engelbrecht2024-05-291-4/+4
|
* workspace shares (#84)Dan Engelbrecht2024-05-291-45/+83
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* improved assert (#37)Dan Engelbrecht2024-04-041-5/+5
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* Actually thow exception if we can't parse the jobid when starting async job ↵Dan Engelbrecht2024-01-311-1/+7
| | | | | | (#645) Return system error code on exception throw in zen command Clean up HttpClient::Response::ErrorMessage to remove redundant ": "
* Adding an info command to display a top-level summary of disk space etc (#602)Stefan Boberg2023-12-121-0/+3
| | | | | this also adds a central, shared folder for storing information which may be found by any instance on the host. The directory is currently located alongside the default install and state directory. Initially this is used to store a collection of known `root_manifest` locations and a copy of the latest manifest version which allow us to find all known locations where zen state is present.
* gc stop command (#569)v0.2.36-pre2Dan Engelbrecht2023-11-271-0/+2
| | | | | - Feature: New endpoint `/admin/gc-stop` to cancel a running garbage collect operation - Feature: Added `zen gc-stop` command to cancel a running garbage collect operation - Bugfix: GCv2 - make sure to discover all projects and oplogs before checking for expired data
* lined up rpc command descriptions with namesStefan Boberg2023-11-221-3/+3
|
* zen run command (#552)Stefan Boberg2023-11-211-73/+111
| | | | | | | | | | initial version -- this is primarily intended to be used for running stress tests and/or benchmarks example usage: `zen run -n 10 -- zenserver-test` `zen run -n 10 -- zenserver-test --ts=core.assert` run zenserver-test 10 times (testing only the `core.assert` test suite) `zen run --time 600 --basepath=d:\test_dir\test1 -- zenserver-test` keeps spawning new instances for 10 minutes (600 seconds)
* removed zen runtests command (#551)Stefan Boberg2023-11-161-29/+0
|
* remove dependency on cxxopts exception types (#542)Stefan Boberg2023-11-151-21/+21
| | | | | | changed options parsing so that we don't depend on cxxopts exception types this makes it possible to use any cxxopts-version including beyond 3.0.0
* spdlog implementation hiding (#498)Stefan Boberg2023-11-061-4/+5
| | | | | | | | | this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future. As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link. Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future. The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
* zen copy-state command to copy a zenserver data directory without the bulk ↵Dan Engelbrecht2023-11-061-0/+2
| | | | | data (#510) * zen copy-state command to copy a zenserver data directory without the bulk data
* add `flush` command and more gc status info (#483)Dan Engelbrecht2023-10-181-0/+2
| | | | | | - Feature: New endpoint `/admin/flush ` to flush all storage - CAS, Cache and ProjectStore - Feature: New command `zen flush` to flush all storage - CAS, Cache and ProjectStore - Improved: Command `zen gc-status` now gives details about storage, when last GC occured, how long until next GC etc - Changed: Cache access and write log are disabled by default
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-09-261-15/+19
|\
| * Add runtime status/control of logging (#419)Dan Engelbrecht2023-09-221-0/+2
| | | | | | | | | | | | | | | | | | | | | | - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
| * Collect all zen admin-related commands into admin.h/.cpp (#418)Dan Engelbrecht2023-09-221-16/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * move commands in scrub.h/cpp to admin_cmd.h/cpp * move job command into admin_cmd.h/.cpp * admin -> admin_cmd * bench -> bench_cmd * cache -> cache_cmd * copy -> copy_cmd * dedup -> dedup_cmd * hash -> hash_cmd * print -> print_cmd * projectstore -> projectstore_cmd * rpcreplay -> rpcreplay_cmd * serve -> serve_cmd * status -> status_cmd * top -> top_cmd * trace -> trace_cmd * up -> up_cmd * version -> version_cmd
| * add trace command to enable/disable tracing at runtime (#416)Dan Engelbrecht2023-09-221-0/+3
| | | | | | | | | | * add trace command to enable/disable tracing at runtime * rework tracing init/start/stop * changelog
* | sort commands for cleaner mergesStefan Boberg2023-09-261-2/+2
|/
* VFS implementation for local storage service (#396)Stefan Boberg2023-09-201-0/+3
| | | currently, only Windows (using Projected File System) is supported
* job queue and async oplog-import/export (#395)Dan Engelbrecht2023-09-131-0/+3
| | | | | | | | | | | | | | | | - Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
* add console logging to zen command (#389)Dan Engelbrecht2023-09-081-1/+15
| | | | properly set trace log level if IsVerbose add log category to http client
* multithread file realization in oplog-mirror (#388)Dan Engelbrecht2023-09-081-1/+1
| | | | convert project store commands to use http client use MakeCbObjectPayload everywhere in project store commands
* add `--write-config` to zenserver options (#382)Dan Engelbrecht2023-09-011-0/+2
| | | | | | | | | | - Feature: `zen up` command has two new command line options - `--config <file_path>` tells zenserver to start with a specific config file - `--owner-pid <pid>` tells zenserver to start with a owning process id - Feature: `zen attach` command to add additional owning processes to a running zenserver instance - `--owner-pid <pid>` adds pid to running zenserver instance list of owning processes - Feature: `--write-config` command line option for zenserver - `--write-config <file_path>` path to a file which will contain a lua config file for zenserver combining all command line options and optional lua config files - Improvement: `zen up` command will check if zenserver is currently running before starting up a new instance
* oplog mirror support (#367)Stefan Boberg2023-08-211-0/+2
| | | | feature: added oplog-mirror command. this can be invoked to export oplog contents to corresponding files
* add update/delete endpoint for project and oplog (#353)Dan Engelbrecht2023-08-181-0/+4
| | | | | | | | | | | | | * add update endpoint for project store project * add update endpoint for oplog * changelog * Zen command line tool `project-update` Zen command line tool `project-delete` Zen command line tool `oplog-update` Zen command line tool `oplog-delete` * add --force-update option to project/oplog create remove project/oplog update commnad
* update vcpkg dependencies (#356)Dan Engelbrecht2023-08-111-1/+1
| | | | | | * bump vcpkg version * fmt lib 10 fixes * xmake dependencies (with linux workarounds) * changelog
* add requested item in oplog remote op (#340)Dan Engelbrecht2023-08-011-1/+1
| | | * add more context for oplog import/export errors
* file share support (#328)Stefan Boberg2023-06-161-4/+36
| | | | | | | | | | | | this change adds a serve command to the zen CLI. This can be used to establish links to a set of files which may be served to clients via the project store interface: ```cmd> zen serve Lyra/WindowsClient d:\temp_share\StagedBuilds\WindowsClient``` with the appropriate changes in UE you may then start an instance of the runtime and have it load all files via the remote file connection: ``` Lyra\Binaries\LyraClient.exe ../../../Lyra/Lyra.uproject -pak -basedir=D:\temp_share\StagedBuilds\WindowsClient/Lyra/Binaries/Win64 -Mount=Lyra/WindowsClient ```
* add oplog snapshot command (#331)Dan Engelbrecht2023-06-121-0/+2
| | | | * add zen oplog-snapshot command line tool * changelog
* fix for issue with missing options in helpStefan Boberg2023-05-251-0/+2
| | | | | positional options were not included in the `--help` output previously, now they are!
* sorted the commands exposed from zen CLIStefan Boberg2023-05-191-8/+8
|
* added benchmark utility command `bench` (#298)Stefan Boberg2023-05-161-1/+3
| | | currently this implements a way (`zen bench --purge`) to purge the standby lists on Windows. This basically flushes the file system cache and is useful to put your system in a consistent state before running benchmarks
* Moved EnableVTMode function into zencore (#311)Stefan Boberg2023-05-161-13/+4
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-07 08:10:45 +0000