aboutsummaryrefslogtreecommitdiff
path: root/src/zenutil
Commit message (Collapse)AuthorAgeFilesLines
...
| * refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-032-47/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
| * builds download incremental (#290)Dan Engelbrecht2025-03-011-5/+7
| | | | | | | | | | * incremental download * merge rebuild state and output state building * fix writing when > 1 zero size file
| * improve error handling (#289)Dan Engelbrecht2025-02-286-15/+79
| | | | | | | | | | | | | | | | * clearer errors * quicker abort * handle deleted local files * simplify parallellwork error handling * don't finish progress on destructor - gives wrong impression * graceful ctrl-c handling
| * Zs/auth bad function fix (#287)Zousar Shaker2025-02-272-7/+5
| | | | | | | | | | | | | | | | | | * Describe fix in changelog * remove JupiterClient::m_TokenProvider --------- Co-authored-by: zousar <[email protected]> Co-authored-by: Dan Engelbrecht <[email protected]>
| * builds upload command (#278)Dan Engelbrecht2025-02-2612-18/+2678
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
| * improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-254-69/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
| * moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-127-0/+1206
| |
| * improved builds api interface in jupiter (#281)Dan Engelbrecht2025-02-122-23/+340
| | | | | | | | | | * multipart upload/download iterface in jupiter * review fixes
* | Merge remote-tracking branch 'origin/de/zen-service-command' into ↵Dan Engelbrecht2025-03-144-16/+64
|\ \ | | | | | | | | | de/zen-service-command
| * | Update Linux service type and add libsystemd dependencyLiam Mitchell2025-03-051-1/+1
| | |
| * | Implementation of service commands for Linux.Liam Mitchell2025-02-273-14/+60
| | |
| * | Linux compilation fixesLiam Mitchell2025-02-192-1/+3
| | |
* | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-236-916/+743
|\ \ \ | |/ / |/| / | |/
| * jupiter code cleanup (#276)Dan Engelbrecht2025-01-226-916/+743
| | | | | | | | | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-1711-2285/+922
|\|
| * move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-1611-2285/+922
| | | | | | | | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* | clang formatDan Engelbrecht2025-01-152-13/+14
| |
* | systemd unit file, incompleteDan Engelbrecht2025-01-152-116/+137
| |
* | clang formatDan Engelbrecht2025-01-131-39/+41
| |
* | generate unit fileDan Engelbrecht2025-01-131-3/+71
| |
* | linux service stopDan Engelbrecht2025-01-131-159/+13
| |
* | linux serviceDan Engelbrecht2025-01-131-1/+198
| |
* | clang formatDan Engelbrecht2025-01-102-65/+60
| |
* | displayname and description are windows-only propertiesDan Engelbrecht2025-01-102-4/+5
| |
* | cleanupsDan Engelbrecht2025-01-101-159/+105
| |
* | partially working service commands for macosDan Engelbrecht2025-01-101-25/+250
| |
* | use local service account for zen serviceDan Engelbrecht2025-01-101-13/+13
| |
* | remove ServiceLevelDan Engelbrecht2025-01-102-356/+25
| |
* | partial macos implementationDan Engelbrecht2025-01-092-12/+60
| |
* | mac function signature fixDan Engelbrecht2025-01-081-8/+6
| |
* | use attachconsole to send ctrl+c to running processDan Engelbrecht2025-01-081-42/+41
| |
* | add ServiceLevel for service processes: User, AllUsers and ServiceDan Engelbrecht2025-01-082-38/+370
| |
* | Merge remote-tracking branch 'origin/de/zen-service-command' into ↵Dan Engelbrecht2025-01-081-35/+112
|\ \ | | | | | | | | | de/zen-service-command
| * | wipDan Engelbrecht2024-09-181-34/+108
| | |
| * | Add ServiceSpec structDan Engelbrecht2024-09-182-39/+65
| | |
| * | clang formatDan Engelbrecht2024-09-181-3/+3
| | |
| * | launchd plist installDan Engelbrecht2024-09-181-4/+15
| | |
| * | more WIPDan Engelbrecht2024-09-181-432/+499
| | |
| * | wipDan Engelbrecht2024-09-181-320/+433
| | |
| * | zen `service` commandDan Engelbrecht2024-09-182-0/+431
| | |
* | | Add ServiceSpec structDan Engelbrecht2025-01-082-39/+65
| | |
* | | clang formatDan Engelbrecht2025-01-081-3/+3
| | |
* | | launchd plist installDan Engelbrecht2025-01-081-4/+15
| | |
* | | more WIPDan Engelbrecht2025-01-081-432/+499
| | |
* | | wipDan Engelbrecht2025-01-081-320/+433
| | |
* | | zen `service` commandDan Engelbrecht2025-01-082-0/+431
| |/ |/|
* | more memory tagging and fixes (#263)Stefan Boberg2024-12-164-2/+23
| | | | | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* | added support for dynamic LLM tags (#245)Stefan Boberg2024-12-023-0/+35
| | | | | | | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* | Insights-compatible memory tracking (#214)Stefan Boberg2024-11-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* | some fixes to timestamp caching in logging (#219)Stefan Boberg2024-11-182-7/+13
| | | | | | * some fixes to timestamp caching in logging
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-19 20:36:46 +0000