aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* more iterate chunk logging (#516)Dan Engelbrecht2025-09-262-12/+56
| | | | | * add log warnings when we can't read payloads in cas when we thing we should have them * fix misleading option help
* Improvement to Incomplete Result Iterationzousar2025-09-251-2/+1
| | | | From review feedback
* Report Incomplete Records To Clientzousar2025-09-241-5/+39
| | | | When requesting partial records, report back when a record is incomplete via an "Incomplete" array of bools that is a sibling to the "Result" array for batch/rpc operations, or via the HttpResponseCode::PartialContent status code for individual record requests.
* Adjust the responses from PUT commandszousar2025-09-233-14/+9
| | | | | - Ensure that text responses are in a field named "Message" - Change the record response to be named "Record" instead of "Object"
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-223-14/+15
|\
| * more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-222-12/+13
| | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
| * issue error on second retry, not first attempt (#503)Dan Engelbrecht2025-09-221-2/+2
| |
* | Change batch put responses for client reportingzousar2025-09-195-36/+79
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* revise exception vs error (#495)Dan Engelbrecht2025-09-151-16/+16
| | | | | - Change BadAlloc exceptions in GC to warnings - Add explict ASSERT exception catch in http plugin request processing - Make exceptions handled in http request processing to warnings
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-109-360/+416
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-042-19/+27
| | | * add validation of compact binary payloads before reading them
* oplog memory usage reduction (#482)Dan Engelbrecht2025-09-041-0/+2
| | | | | | - Improvement: For projectstore oplog GET operation, only read basic information and release it if the oplog is not already open to reduce memory usage when listing oplogs in web UI - Improvement: Reduce memory usage for oplog Op address lookup Refactored Oplog::EState -> Oplog ::EMode and make sure we open data files in read-only mode when EMode::kBasicReadOnly is used.
* revert multi-cid store (#475)Dan Engelbrecht2025-08-263-44/+20
|
* clean up trace options parsing (#473)Dan Engelbrecht2025-08-221-0/+2
| | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-2012-136/+165
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-1/+1
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-122-105/+91
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-083-550/+323
|\
| * add the correct set of references hashes in batched inline mode (#459)Dan Engelbrecht2025-08-061-3/+3
| |
| * refactor blobstore (#458)Dan Engelbrecht2025-08-062-548/+321
| | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
* | precommitzousar2025-08-071-9/+4
| |
* | Avoid committing chunks for batch rejected putszousar2025-08-071-42/+35
| | | | | | | | Previously rejected puts would put the chunks, but not write them to the index, which was wrong.
* | Moving put rejections to happen in batch handlingzousar2025-08-052-41/+101
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-08-051-11/+40
|\|
| * add hardening for legacy cache bucket manifests (#454)Dan Engelbrecht2025-08-041-11/+40
| |
* | xmake precommitzousar2025-06-242-7/+7
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-2426-1873/+4320
|\|
| * make sure we unregister from GC before we drop bucket/namespaces (#443)Dan Engelbrecht2025-06-192-1/+4
| |
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-164-172/+213
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
| * missing chunks bugfix (#424)Dan Engelbrecht2025-06-097-84/+479
| | | | | | | | | | | | | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
| * pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-054-7/+14
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
| * fix cachbucket mem hit count (#415)Dan Engelbrecht2025-06-021-4/+7
| | | | | | | | | | * Don't count a miss twice for memory stats if the entry can't be found * changelog
| * add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-305-17/+130
| | | | | | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
| * faster oplog validate (#408)Dan Engelbrecht2025-05-304-10/+69
| | | | | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
| * frequent disk space check (#407)Dan Engelbrecht2025-05-271-25/+84
| | | | | | | | * check low disk space condition more frequently and trigger GC when low water mark is reached * show waited time when waiting for zenserver instance to exit
| * unblock cache bucket drop (#406)Dan Engelbrecht2025-05-264-54/+124
| | | | | | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
| * handle exception with batch work (#401)Dan Engelbrecht2025-05-194-134/+147
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
| * make sure tests initialize trace so we don't end up allocating tons of ↵Stefan Boberg2025-05-151-0/+2
| | | | | | | | memory for no reason (#397)
| * keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-126-104/+81
| | | | | | | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
| * enable per bucket config (#388)Dan Engelbrecht2025-05-122-2/+22
| | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| * tweak iterate block parameters (#390)Dan Engelbrecht2025-05-122-34/+50
| | | | | | * tweak block iteration chunk sizes
| * flush cas log file (#387)Dan Engelbrecht2025-05-092-46/+31
| | | | | | * make sure we remove the cas log file when writing full index at startup
| * make RemoveExpiredData and PreCache serial to reduce CPU overhead / lock ↵Dan Engelbrecht2025-05-071-85/+61
| | | | | | | | | | contention (#385) * make RemoveExpiredData and PreCache serial to reduce CPU overhead / lock contention
| * optimize block store CompactBlocks (#384)Dan Engelbrecht2025-05-071-14/+19
| | | | | | | | | | - Improvement: Optimize block compact reducing memcpy operations - Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer - Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
| * optimize cache bucket state writing (#382)Dan Engelbrecht2025-05-061-42/+59
| | | | | | * optimize cache bucket snapshot and sidecar writing
| * make OOD and OOM in gc non critical (#381)Dan Engelbrecht2025-05-051-27/+218
| | | | | | | | * oom and ood exceptions in GC are now treated as warnings instead of errors
| * silence Out Of Disk errors to sentry (#378)Dan Engelbrecht2025-05-051-48/+56
| | | | | | | | | | * block writing GC state/info if disk is full * fix if/else on error while writing gc state
| * iterate chunks crash fix (#376)Dan Engelbrecht2025-05-024-76/+314
| | | | | | * Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
| * cbobject validation (#377)Dan Engelbrecht2025-05-021-10/+31
| | | | | | | | * validate incoming CbObject to cache when receiving a package * validate records when fetched from store in cache before parsing them
| * replace local equal_to_2 with eastl impl (#368)Stefan Boberg2025-04-251-16/+2
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-23 04:44:41 +0000