aboutsummaryrefslogtreecommitdiff
path: root/src/zenremotestore/builds/buildstorageoperations.cpp
Commit message (Collapse)AuthorAgeFilesLines
* add support in http client to accept multi-range responses (#788)Dan Engelbrecht12 days1-74/+78
| | | * add support in http client to accept multi-range responses
* use partial blocks for oplog import (#780)Dan Engelbrecht2026-02-241-28/+17
| | | | | Feature: Add --allow-partial-block-requests to zen oplog-import Improvement: zen oplog-import now uses partial block requests to reduce download size Improvement: Use latency to Cloud Storage host and Zen Cache host when calculating partial block requests
* move partial chunk block anailsys to chunkblock.h/cpp (#767)Dan Engelbrecht2026-02-201-628/+254
|
* fix builds download indexing timer (#769)Dan Engelbrecht2026-02-201-7/+0
| | | * fix build download indexing timer log
* convert ZEN_ASSERTs to exception to handle corrupt data gracefully (#760)Dan Engelbrecht2026-02-181-12/+56
| | | * convert ZEN_ASSERTs to exception to handle corrupt data gracefully
* fix target folder cleanup in builds download (#746)Dan Engelbrecht2026-02-061-12/+9
|
* only disable backlog scheduling when downloaded payload is not on disk (#741)Dan Engelbrecht2026-02-021-6/+12
|
* allow download specification for zen builds download (#734)Dan Engelbrecht2026-01-271-95/+67
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds download` now supports `--download-spec-path` to determine what content to download from a build - The unstructured format expects one line per file relative to the root with '/' as a path delimiter - The structured format uses JSon format and the `--download-spec-path` must have extension `.json` to enable structured input { "parts": { "default" : { "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* avoid big ioworker backlog (#733)Dan Engelbrecht2026-01-261-35/+40
| | | | | | * add ability to override scheduling mode in ParallelWork * Don't increase buffering size when copying from local cache with --boost-worker-memory enabled * Rework scheduling writes of downloaded data to reduce memory usage
* builds scanning cache (#727)v5.7.19-pre0Dan Engelbrecht2026-01-231-48/+136
| | | | - Feature: Added `--chunking-cache-path` option to `zen builds upload` and `zen builds diff` - Path to cache for chunking information of scanned files. Default is empty resulting in no caching
* make sure new blocks generated by a part is accessible to following parts (#732)Dan Engelbrecht2026-01-231-79/+88
|
* builds multipart upload (#722)Dan Engelbrecht2026-01-201-789/+1817
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds upload` now support structure manifest input for `--manifest-path` when the path has a `.json` extension - The structured manifest supports splitting a build into multiple parts { "parts": { "default" : { "partId": "f939f3939939fff3f3202", # optional - used to control the id of each part "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* optimize scavenge part 2 (#699)Dan Engelbrecht2026-01-081-148/+98
| | | | * optimize scavenge discovery by iterate over chunks only instead of iterating through all chunks in the scavenged files * refactor scavenge lookup
* optimize scavenge (#697)Dan Engelbrecht2025-12-191-118/+56
| | | | * optimize FindScavengeContent * optimize GetValidFolderContent
* fix scavenge source path match (again) (#694)Dan Engelbrecht2025-12-171-11/+12
| | | | * fix check for scavenged path matching target path
* oplog download size (#690)Dan Engelbrecht2025-12-151-190/+17
| | | | - Bugfix: Upload of oplogs could reference multiple blocks for the same chunk causing redundant downloads of blocks - Improvement: Use the improved block reuse selection function from zen builds upload in zen oplog-export to reduce oplog download size
* remove direct console output from code that is used from service mode (#688)Dan Engelbrecht2025-12-111-9/+16
|
* safer lambda captures (#633)Dan Engelbrecht2025-12-081-8/+10
|
* exists must be called before equivalent (#678)Dan Engelbrecht2025-12-051-1/+1
|
* use scope guards to secure that BasicFile::Detach is called even on ↵Dan Engelbrecht2025-12-011-1/+1
| | | | exception (#672)
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-271-1/+4
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* remove bad assert (#670)Dan Engelbrecht2025-11-271-4/+0
|
* update state when wildcard (#657)Dan Engelbrecht2025-11-241-166/+95
| | | * add --append option and improve state handling when using downloads for `zen builds download`
* control cache upload (#646)Dan Engelbrecht2025-11-121-9/+9
| | | * add option to enable/disable upload to builds cache
* add `--boost-worker-memory` option to zen builds (#639)Dan Engelbrecht2025-11-101-4/+4
|
* add check that we have enough free space to complete the builds download (#640)Dan Engelbrecht2025-11-101-6/+54
| | | * add check that we have enough free space to complete the builds download
* move progress bar to separate file (#638)Dan Engelbrecht2025-11-071-1/+1
| | | * move progress bar to separate file
* remotestore op refactorings (#637)Dan Engelbrecht2025-11-061-465/+498
| | | | | | * broke out BuildLogOutput to separate file * refactored out GetBlockDescriptions * log progress improvements * refactorings to accomodate oplog download operations
* compress partial file chunks in blocks if appropriate (#632)v5.7.9-pre8v5.7.9-pre7v5.7.9-pre6v5.7.9-pre5v5.7.9-pre4v5.7.9-pre3v5.7.9-pre11v5.7.9-pre1v5.7.9-pre0v5.7.8-pre11v5.7.8-pre10v5.7.8Dan Engelbrecht2025-11-041-13/+17
| | | | | - Improvement: Hide compress/download rates when uploading build when complete - Improvement: Compress partial file chunks inside blocks if appropriate - Bugfix: Don't display "Validating" progress title during download if validation is disabled
* abort build upload if we fail to finalize a build part (#623)Dan Engelbrecht2025-11-031-13/+53
| | | * abort build upload if we fail to finalize a build part
* fix clean directory and make them use effective threading where appropriate ↵v5.7.8-pre5v5.7.8-pre3v5.7.8-pre2Dan Engelbrecht2025-11-031-17/+6
| | | | | | (#625) fix retry logic so it does not immediately sleep if file does not exist make sure we don't try to delete target folder files if we have already wiped it
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-011-7/+15
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* add `zen builds prime-cache` command (#598)Dan Engelbrecht2025-10-221-0/+274
|
* make validation of completed sequences in builds download optional (#596)Dan Engelbrecht2025-10-221-2527/+2573
|
* updated chunking strategy (#589)Dan Engelbrecht2025-10-201-23/+21
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
* missing update of m_DiskStats.WriteByteCount in WriteSequenceChunkToCache (#583)Dan Engelbrecht2025-10-161-13/+3
|
* builds download progress include validate (#582)Dan Engelbrecht2025-10-161-37/+79
| | | * take validation into account for progress feedback when downloading builds
* fix log progress quotes (#580)Dan Engelbrecht2025-10-161-5/+14
| | | | * add quotes to message in log-progress mode * improved log progress for builds upload and download
* refactor builds cmd part4 (#579)Dan Engelbrecht2025-10-161-719/+709
| | | * move lambdas to class functions
* move builds state functions to buildsavedstate.h/cpp (#577)Dan Engelbrecht2025-10-151-183/+33
|
* move file i/o related files to separate file and remove duplicated code (#576)Dan Engelbrecht2025-10-151-502/+193
|
* refactor builds cmd part3 (#573)Dan Engelbrecht2025-10-141-170/+554
| | | | * move lambdas to member functions * add BuildsOperationValidateBuildPart
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-19/+2410
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-0/+4742
Move builds download code from builds_cmd.cpp to remotestorelib
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 07:57:22 +0000