aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-2012-136/+165
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-1/+1
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-122-105/+91
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-083-550/+323
|\
| * add the correct set of references hashes in batched inline mode (#459)Dan Engelbrecht2025-08-061-3/+3
| |
| * refactor blobstore (#458)Dan Engelbrecht2025-08-062-548/+321
| | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
* | precommitzousar2025-08-071-9/+4
| |
* | Avoid committing chunks for batch rejected putszousar2025-08-071-42/+35
| | | | | | | | Previously rejected puts would put the chunks, but not write them to the index, which was wrong.
* | Moving put rejections to happen in batch handlingzousar2025-08-052-41/+101
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-08-051-11/+40
|\|
| * add hardening for legacy cache bucket manifests (#454)Dan Engelbrecht2025-08-041-11/+40
| |
* | xmake precommitzousar2025-06-242-7/+7
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-2426-1873/+4320
|\|
| * make sure we unregister from GC before we drop bucket/namespaces (#443)Dan Engelbrecht2025-06-192-1/+4
| |
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-164-172/+213
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
| * missing chunks bugfix (#424)Dan Engelbrecht2025-06-097-84/+479
| | | | | | | | | | | | | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
| * pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-054-7/+14
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
| * fix cachbucket mem hit count (#415)Dan Engelbrecht2025-06-021-4/+7
| | | | | | | | | | * Don't count a miss twice for memory stats if the entry can't be found * changelog
| * add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-305-17/+130
| | | | | | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
| * faster oplog validate (#408)Dan Engelbrecht2025-05-304-10/+69
| | | | | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
| * frequent disk space check (#407)Dan Engelbrecht2025-05-271-25/+84
| | | | | | | | * check low disk space condition more frequently and trigger GC when low water mark is reached * show waited time when waiting for zenserver instance to exit
| * unblock cache bucket drop (#406)Dan Engelbrecht2025-05-264-54/+124
| | | | | | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
| * handle exception with batch work (#401)Dan Engelbrecht2025-05-194-134/+147
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
| * make sure tests initialize trace so we don't end up allocating tons of ↵Stefan Boberg2025-05-151-0/+2
| | | | | | | | memory for no reason (#397)
| * keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-126-104/+81
| | | | | | | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
| * enable per bucket config (#388)Dan Engelbrecht2025-05-122-2/+22
| | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| * tweak iterate block parameters (#390)Dan Engelbrecht2025-05-122-34/+50
| | | | | | * tweak block iteration chunk sizes
| * flush cas log file (#387)Dan Engelbrecht2025-05-092-46/+31
| | | | | | * make sure we remove the cas log file when writing full index at startup
| * make RemoveExpiredData and PreCache serial to reduce CPU overhead / lock ↵Dan Engelbrecht2025-05-071-85/+61
| | | | | | | | | | contention (#385) * make RemoveExpiredData and PreCache serial to reduce CPU overhead / lock contention
| * optimize block store CompactBlocks (#384)Dan Engelbrecht2025-05-071-14/+19
| | | | | | | | | | - Improvement: Optimize block compact reducing memcpy operations - Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer - Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
| * optimize cache bucket state writing (#382)Dan Engelbrecht2025-05-061-42/+59
| | | | | | * optimize cache bucket snapshot and sidecar writing
| * make OOD and OOM in gc non critical (#381)Dan Engelbrecht2025-05-051-27/+218
| | | | | | | | * oom and ood exceptions in GC are now treated as warnings instead of errors
| * silence Out Of Disk errors to sentry (#378)Dan Engelbrecht2025-05-051-48/+56
| | | | | | | | | | * block writing GC state/info if disk is full * fix if/else on error while writing gc state
| * iterate chunks crash fix (#376)Dan Engelbrecht2025-05-024-76/+314
| | | | | | * Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
| * cbobject validation (#377)Dan Engelbrecht2025-05-021-10/+31
| | | | | | | | * validate incoming CbObject to cache when receiving a package * validate records when fetched from store in cache before parsing them
| * replace local equal_to_2 with eastl impl (#368)Stefan Boberg2025-04-251-16/+2
| |
| * predicate to enable compiling with later EASTL version (#367)Stefan Boberg2025-04-241-1/+1
| |
| * blobstore size limit (#342)Dan Engelbrecht2025-04-042-13/+255
| | | | | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
| * build store save access times (#341)v5.6.3-pre0Dan Engelbrecht2025-04-033-30/+304
| | | | | | | | | | * save payload size in log for buildstore * read/write access times and manifest for buldstore * use retry when removing temporary files
| * use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-13/+0
| | | | | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
| * reduce disk io during gc (#335)Dan Engelbrecht2025-04-011-20/+8
| | | | | | * do cache bucket flush/write snapshot as part of compact to reduce disk I/O
| * long filename support (#330)Dan Engelbrecht2025-03-319-117/+117
| | | | | | - Bugfix: Long file paths now works correctly on Windows
| * optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-5/+26
| | | | | | | | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
| * zen build cache service (#318)Dan Engelbrecht2025-03-2611-46/+1738
| | | | | | | | | | | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
| * reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-067-108/+143
| | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
| * improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
| * moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-124-997/+0
| |
* | Change to PutResult structurezousar2025-06-247-134/+201
| | | | | | | | Result structure contains status and a string message (may be empty)
* | Establish TODOs and unit test for rejected PUT propagationzousar2025-03-241-0/+1
| |
* | Control overwrite enforcement with a config settingzousar2025-03-022-1/+3
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 01:25:38 +0000