aboutsummaryrefslogtreecommitdiff
path: root/src/zenremotestore
Commit message (Collapse)AuthorAgeFilesLines
...
* builds multipart upload (#722)Dan Engelbrecht2026-01-206-800/+1960
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds upload` now support structure manifest input for `--manifest-path` when the path has a `.json` extension - The structured manifest supports splitting a build into multiple parts { "parts": { "default" : { "partId": "f939f3939939fff3f3202", # optional - used to control the id of each part "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* fix init of buildpart id in zen oplog-download (#711)Dan Engelbrecht2026-01-151-1/+1
|
* structured output for builds ls (#709)Dan Engelbrecht2026-01-141-3/+12
| | | | * make ResolveBuildStore respect Verbose flag * add structured output to zen builds ls command
* added options to configure exclude folders and extensions to zen build ↵Dan Engelbrecht2026-01-134-150/+222
| | | | | | | | commands (#706) * added `--exclude-folders` to `zen upload`, `zen download` and `zen diff` added `--exclude-extensions` to `zen upload` and `zen diff` excluded folder names are now matched by folder name in subfolders in addition to root level folders * allow multiple token separators
* fixed block generation after bug introduced in PR #704 (#707)Dan Engelbrecht2026-01-131-12/+15
|
* various optimizations (#704)Dan Engelbrecht2026-01-092-22/+126
| | | | | | | | | - Improvement: Validate chunk hashes when dechunking files in oplog import - Improvement: Use stream decompression when dechunking files - Improvement: When assembling blocks for oplog export, make sure we keep under/at block size limit - Improvement: Make cancelling of oplog import more responsive - Improvement: Use decompress to composite to avoid allocating a new memory buffer for uncompressed chunks during oplog import - Improvement: Reduce memory buffer size and allocate it on demand when writing multiple chunks to block store - Improvement: Reduce lock contention when fetching/checking existence of chunks in block store
* read build state files into memory instead of relying on memory mapping (#702)Dan Engelbrecht2026-01-081-2/+7
|
* optimize scavenge part 2 (#699)Dan Engelbrecht2026-01-082-148/+109
| | | | * optimize scavenge discovery by iterate over chunks only instead of iterating through all chunks in the scavenged files * refactor scavenge lookup
* Merge branch 'main' into zs/limit-overwrite-defaultZousar Shaker2025-12-194-169/+241
|\
| * optimize scavenge (#697)Dan Engelbrecht2025-12-194-169/+241
| | | | | | | | * optimize FindScavengeContent * optimize GetValidFolderContent
* | Ensure upstream put propagation includes overwritezousar2025-12-193-6/+20
|/ | | | When changing the default limit-overwrite behavior, a unit test surfaced a bug where an put of data with overwrite cache policy would not get propagated via zen's built-in upstream mechanism with a matching overwrite cache policy to the upstream. This change ensures that it does and leaves the unit test configured to exercise this scenario.
* fix scavenge source path match (again) (#694)Dan Engelbrecht2025-12-171-11/+12
| | | | * fix check for scavenged path matching target path
* add boost-worker oplog import export options (#693)Dan Engelbrecht2025-12-165-16/+154
| | | | | | | | | | - Feature: `zen oplog-export`, `zen oplog-import` and `zen oplog-download` now has options to boost workers - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive - Improvement: Refactored boost options for `zen builds` operations `upload`, `download`, `diff`, `prime-cache`, `fetch-blob` and `validate-part` - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive
* remove found chunks as they are found in blocks (#691)Dan Engelbrecht2025-12-151-7/+18
|
* oplog download size (#690)Dan Engelbrecht2025-12-159-295/+627
| | | | - Bugfix: Upload of oplogs could reference multiple blocks for the same chunk causing redundant downloads of blocks - Improvement: Use the improved block reuse selection function from zen builds upload in zen oplog-export to reduce oplog download size
* show download source data (#689)Dan Engelbrecht2025-12-1211-2/+163
| | | * show source stats for jupiter/cache
* remove direct console output from code that is used from service mode (#688)Dan Engelbrecht2025-12-116-28/+145
|
* fix buffer memory in builds cache (#682)Dan Engelbrecht2025-12-101-18/+68
| | | | * add --zen-cache-upload option to zen oplog-import command * fix buildstoragecache to not hold on to possibly materialized buffers
* safer lambda captures (#633)Dan Engelbrecht2025-12-081-8/+10
|
* exists must be called before equivalent (#678)Dan Engelbrecht2025-12-051-1/+1
|
* add checks to protect against access violation due to failed disk read (#675)Dan Engelbrecht2025-12-041-0/+5
| | | * add checkes to protect against access violation due to failed disk read
* use scope guards to secure that BasicFile::Detach is called even on ↵Dan Engelbrecht2025-12-011-1/+1
| | | | exception (#672)
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-274-45/+17
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* remove bad assert (#670)Dan Engelbrecht2025-11-271-4/+0
|
* update state when wildcard (#657)Dan Engelbrecht2025-11-249-665/+1475
| | | * add --append option and improve state handling when using downloads for `zen builds download`
* changelog (#661)Dan Engelbrecht2025-11-241-1/+1
|
* increase timeout and add retry when testing host connectivity (#659)Dan Engelbrecht2025-11-214-8/+8
|
* loose chunk filtering bug when using wildcards (#654)Dan Engelbrecht2025-11-183-0/+475
| | | | | | | | | | * fix filtering of loose chunks when downloading with a filter add tests * changelog * move InlineRemoveUnusedHashes * remove extra braces
* control cache upload (#646)Dan Engelbrecht2025-11-126-14/+23
| | | * add option to enable/disable upload to builds cache
* Change curl defaults on MacOS (#645)Stefan Boberg2025-11-117-19/+86
| | | | | | | * changed curl config to match the default from vcpkg (i.e `CURL_CA_FALLBACK=ON`) * disables use of Secure Transport for Mac since it's deprecated * Also worked around an issue (with `CURL_CA_BUNDLE`) where cross compiling curl on Mac would not configure curl in the same way as when compiling natively. This meant builds would not download on ARM macs when the CI build machine architecture was x86. The workaround should be redundant if we upgrade to 8.17 and use Apple SecTrust for cert validation. This should happen soon. * Also added various verbose logging to facilitate trouble shooting
* add `--boost-worker-memory` option to zen builds (#639)Dan Engelbrecht2025-11-102-4/+5
|
* fix parsing of assume http2 from server discovery (#643)Dan Engelbrecht2025-11-101-1/+1
|
* add check that we have enough free space to complete the builds download (#640)Dan Engelbrecht2025-11-102-6/+56
| | | * add check that we have enough free space to complete the builds download
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-3/+3
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* get oplog attachments (#622)Dan Engelbrecht2025-11-074-18/+1054
| | | * add support for downloading individual attachments from an oplog
* move progress bar to separate file (#638)Dan Engelbrecht2025-11-073-72/+2
| | | * move progress bar to separate file
* remotestore op refactorings (#637)Dan Engelbrecht2025-11-066-555/+913
| | | | | | * broke out BuildLogOutput to separate file * refactored out GetBlockDescriptions * log progress improvements * refactorings to accomodate oplog download operations
* merge 5.7.8 minimal release (#635)Dan Engelbrecht2025-11-061-0/+2
| | | merge back all the cherry-picks
* compress partial file chunks in blocks if appropriate (#632)v5.7.9-pre8v5.7.9-pre7v5.7.9-pre6v5.7.9-pre5v5.7.9-pre4v5.7.9-pre3v5.7.9-pre11v5.7.9-pre1v5.7.9-pre0v5.7.8-pre11v5.7.8-pre10v5.7.8Dan Engelbrecht2025-11-041-13/+17
| | | | | - Improvement: Hide compress/download rates when uploading build when complete - Improvement: Compress partial file chunks inside blocks if appropriate - Bugfix: Don't display "Validating" progress title during download if validation is disabled
* refactor MergeChunkedFolderContents to not reallocate ↵Dan Engelbrecht2025-11-041-9/+35
| | | | Result.ChunkedContent.ChunkOrders for each path added (#626)
* abort build upload if we fail to finalize a build part (#623)Dan Engelbrecht2025-11-032-14/+55
| | | * abort build upload if we fail to finalize a build part
* fix clean directory and make them use effective threading where appropriate ↵v5.7.8-pre5v5.7.8-pre3v5.7.8-pre2Dan Engelbrecht2025-11-033-28/+55
| | | | | | (#625) fix retry logic so it does not immediately sleep if file does not exist make sure we don't try to delete target folder files if we have already wiped it
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-0110-55/+64
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-281-1/+1
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* fixed progress bar when scanning changed local files (#608)Dan Engelbrecht2025-10-241-0/+12
| | | * fixed progress bar when scanning changed local files
* add host discovery and zen cache support for oplog import (#601)Dan Engelbrecht2025-10-2313-108/+500
| | | * add host discovery and zen cache support for oplog import
* add `zen builds prime-cache` command (#598)Dan Engelbrecht2025-10-224-0/+322
|
* make validation of completed sequences in builds download optional (#596)Dan Engelbrecht2025-10-222-2527/+2576
|
* fix jupiter payload validation with allow redirect (#594)Dan Engelbrecht2025-10-201-5/+8
| | | * don't validate compressed buffer payload if a range is requested
* updated chunking strategy (#589)Dan Engelbrecht2025-10-207-414/+518
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-25 03:04:53 +0000