aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
* | | | | | | fixes for log timestamps (#304)Stefan Boberg2025-03-129-3/+379
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add GetTimeSinceProcessStart returning time since process start. implemented using https://github.com/maxliani/GetTimeSinceProcessStart/tree/main * fix fractions when using epoch mode. Previously it would show the fraction from the absolute time stamp and not relative to epoch * used GetTimeSinceProcessStart to offset the epoch so that it represents the process spawn time
* | | | | | | 5.6.1-pre0v5.6.1-pre0Dan Engelbrecht2025-03-121-1/+1
| | | | | | |
* | | | | | | fix mac/linux builds command (#303)Dan Engelbrecht2025-03-122-15/+22
| | | | | | | | | | | | | | | | | | | | | * fix linux/mac version of GetModificationTickFromPath and CopyFile
* | | | | | | improved block gen logic (#302)Dan Engelbrecht2025-03-128-458/+606
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Reduced memory usage during upload and part upload validation - Improvement: Reduced I/O usage during upload and download - Improvement: Faster block regeneration when uploading in response to PutBuild/FinalizeBuild - Improvement: More trace scopes for build upload operations - Bugfix: Fixed crash during download when trying to write outside a file range
* | | | | | | async find blocks (#300)Dan Engelbrecht2025-03-112-42/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * put/get build and find blocks while scanning local folder when uploading * changelog * remove redundant move
* | | | | | | Build command tweaks (#301)Stefan Boberg2025-03-118-59/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* | | | | | | pick up existing cache (#299)Dan Engelbrecht2025-03-1010-888/+1655
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run - Improvement: Faster abort during stream compression - Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write - Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails
* | | | | | | partial block fetch (#298)Dan Engelbrecht2025-03-079-402/+913
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
* | | | | | | reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-0627-199/+380
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* | | | | | | 5.6.0v5.6.0Dan Engelbrecht2025-03-061-1/+1
| | | | | | |
* | | | | | | 5.6.0-pre0v5.6.0-pre0Dan Engelbrecht2025-03-051-1/+1
| | | | | | |
* | | | | | | Add trace support for zen CLI command (#296)Stefan Boberg2025-03-055-37/+212
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - This change adds support for `--trace`, `--tracehost` and `--tracefile` command arguments to enable and control tracing to Insights - It also adds profiling scopes primarily to build download command related code
* | | | | | | streaming compress (#295)Dan Engelbrecht2025-03-054-38/+292
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Validate hash of decompressed data inline with streaming decompression - Improvement: Do streaming compression of large blobs to improve memory and I/O performance
* | | | | | | do direct update of stats numbers (#294)Dan Engelbrecht2025-03-041-24/+13
| | | | | | |
* | | | | | | stream decompress (#293)Dan Engelbrecht2025-03-044-57/+338
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * clean up latency parameters and slow down rate updates * add DecompressToStream
* | | | | | | limit and validate responses before logging the text (#292)Stefan Boberg2025-03-045-1/+111
| | | | | | | | | | | | | | | | | | | | | Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
* | | | | | | refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-036-552/+919
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
* | | | | | | builds download incremental (#290)Dan Engelbrecht2025-03-014-629/+635
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * incremental download * merge rebuild state and output state building * fix writing when > 1 zero size file
* | | | | | | improve error handling (#289)Dan Engelbrecht2025-02-288-974/+1036
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * clearer errors * quicker abort * handle deleted local files * simplify parallellwork error handling * don't finish progress on destructor - gives wrong impression * graceful ctrl-c handling
* | | | | | | 5.5.20v5.5.20Dan Engelbrecht2025-02-271-1/+1
| | | | | | |
* | | | | | | changelogDan Engelbrecht2025-02-271-1/+1
| | | | | | |
* | | | | | | 5.5.20-pre0v5.5.20-pre0Dan Engelbrecht2025-02-271-1/+1
| | | | | | |
* | | | | | | Zs/auth bad function fix (#287)Zousar Shaker2025-02-273-7/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Describe fix in changelog * remove JupiterClient::m_TokenProvider --------- Co-authored-by: zousar <[email protected]> Co-authored-by: Dan Engelbrecht <[email protected]>
* | | | | | | Fix bug causing crash if large file was exact multiple of 256KB when using ↵Dan Engelbrecht2025-02-262-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | zen builds command (#286)
* | | | | | | clang-format fixStefan Boberg2025-02-261-1/+1
| | | | | | |
* | | | | | | added <chrono> include to zentest-appstub.cppStefan Boberg2025-02-261-0/+1
| | | | | | |
* | | | | | | 5.5.19v5.5.19Dan Engelbrecht2025-02-261-1/+1
| | | | | | |
* | | | | | | 5.5.19-pre0v5.5.19-pre0Dan Engelbrecht2025-02-261-1/+1
| | | | | | |
* | | | | | | builds upload command (#278)Dan Engelbrecht2025-02-2624-61/+9169
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* | | | | | | improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-2521-271/+399
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* | | | | | | 5.5.18v5.5.18Dan Engelbrecht2025-02-241-1/+1
| | | | | | |
* | | | | | | 5.5.18-pre0v5.5.18-pre0Dan Engelbrecht2025-02-241-1/+1
| | | | | | |
* | | | | | | strip leading path separator when creating workspace shares (#285)Dan Engelbrecht2025-02-242-1/+16
| |/ / / / / |/| | | | |
* | | | | | move WriteToTempFile to basicfile.h (#283)Dan Engelbrecht2025-02-123-63/+111
| | | | | | | | | | | | | | | | | | add helper constructors to BasicFile
* | | | | | moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-1214-243/+334
| | | | | |
* | | | | | improved builds api interface in jupiter (#281)Dan Engelbrecht2025-02-125-163/+518
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * multipart upload/download iterface in jupiter * review fixes
* | | | | | Fix workspace shares reply array (#280)Tomasz Obrębski2025-02-031-3/+3
| |_|_|_|/ |/| | | | | | | | | | | | | | Workspace shares were sent in the '/ws' reply as repeating objects 'shares' instead of a 'shares' array of objects
| * | | | Change to PutResult structurezousar2025-06-248-166/+258
| | | | | | | | | | | | | | | | | | | | Result structure contains status and a string message (may be empty)
| * | | | Establish TODOs and unit test for rejected PUT propagationzousar2025-03-243-2/+78
| | | | |
| * | | | Control overwrite enforcement with a config settingzousar2025-03-026-47/+154
| | | | |
| * | | | Move utility methods in cachedisklayerzousar2025-02-261-38/+38
| | | | | | | | | | | | | | | | | | | | Value comparison methods moved to more appropriate area in file.
| * | | | Enforce Overwrite Prevention According To Cache Policyzousar2025-02-267-130/+311
| | | | | | | | | | | | | | | | | | | | Overwrite with differing value should be denied if QueryLocal is not present and StoreLocal is present. Overwrite with equal value should succeed regardless of policy flags.
| * | | | Expand and fix unit tests for overwrite behaviorzousar2025-02-261-7/+82
| | | | |
| * | | | Add unit test for overwrite behaviorzousar2025-02-181-0/+102
|/ / / /
* | | | handle special backslash followed by quote for paths (#279)Dan Engelbrecht2025-01-222-4/+17
| | | |
* | | | Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-2216-131/+290
| | | | | | | | | | | | | | | | | | | | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* | | | jupiter code cleanup (#276)Dan Engelbrecht2025-01-2216-1102/+1007
| |_|/ |/| | | | | | | | | | | | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* | | add missing statsd.enabled and gc.projectstore.duration.seconds (#275)Dan Engelbrecht2025-01-162-10/+16
| | | | | | | | | | | | | | | - Bugfix: Fix parsing of workspace options in Lua config - Bugfix: Add missing Lua option for option `--gc-projectstore-duration-seconds` - Bugfix: Add missing Lua mapping option to `--statsd` command line option
* | | export oplog to zen target url (#274)Dan Engelbrecht2025-01-163-4/+9
| | | | | | | | | | | | * fix oplog target url for oplog export to remote zenserver * verify that chunking is allowed before chunking loose files
* | | move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-1642-53/+54
| | | | | | | | | | | | | | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-21 07:35:54 +0000