aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/cache
Commit message (Collapse)AuthorAgeFilesLines
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-035-5/+673
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-031-1/+2
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-1/+1
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* more cbobject validations (#527)Dan Engelbrecht2025-09-291-5/+19
| | | - Improvement: Add additional validations when reading disk cache records to get references in GC
* GetCacheChunk value request respects RawHash (#518)Zousar Shaker2025-09-291-1/+3
| | | When requesting a value via the GetCacheChunks rpc method, if the request specified a raw hash, only return a hit if the raw hash matches what was in the cache.
* Improvement to Incomplete Result Iterationzousar2025-09-251-2/+1
| | | | From review feedback
* Report Incomplete Records To Clientzousar2025-09-241-5/+39
| | | | When requesting partial records, report back when a record is incomplete via an "Incomplete" array of bools that is a sibling to the "Result" array for batch/rpc operations, or via the HttpResponseCode::PartialContent status code for individual record requests.
* Adjust the responses from PUT commandszousar2025-09-233-14/+9
| | | | | - Ensure that text responses are in a field named "Message" - Change the record response to be named "Record" instead of "Object"
* Change batch put responses for client reportingzousar2025-09-193-33/+75
| | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-102-48/+63
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-5/+3
| | | * add validation of compact binary payloads before reading them
* revert multi-cid store (#475)Dan Engelbrecht2025-08-261-37/+17
|
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-203-68/+82
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-121-91/+81
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-081-2/+2
|\
| * add the correct set of references hashes in batched inline mode (#459)Dan Engelbrecht2025-08-061-3/+3
| |
* | precommitzousar2025-08-071-9/+4
| |
* | Avoid committing chunks for batch rejected putszousar2025-08-071-42/+35
| | | | | | | | Previously rejected puts would put the chunks, but not write them to the index, which was wrong.
* | Moving put rejections to happen in batch handlingzousar2025-08-051-33/+87
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-08-051-11/+40
|\|
| * add hardening for legacy cache bucket manifests (#454)Dan Engelbrecht2025-08-041-11/+40
| |
* | xmake precommitzousar2025-06-241-4/+4
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-243-308/+530
|\|
| * make sure we unregister from GC before we drop bucket/namespaces (#443)Dan Engelbrecht2025-06-192-1/+4
| |
| * graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-35/+45
| | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
| * missing chunks bugfix (#424)Dan Engelbrecht2025-06-091-14/+70
| | | | | | | | | | | | | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
| * pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-051-4/+8
| | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
| * fix cachbucket mem hit count (#415)Dan Engelbrecht2025-06-021-4/+7
| | | | | | | | | | * Don't count a miss twice for memory stats if the entry can't be found * changelog
| * add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-301-2/+22
| | | | | | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
| * unblock cache bucket drop (#406)Dan Engelbrecht2025-05-262-40/+110
| | | | | | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
| * handle exception with batch work (#401)Dan Engelbrecht2025-05-191-15/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
| * keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-121-26/+21
| | | | | | | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
| * enable per bucket config (#388)Dan Engelbrecht2025-05-121-2/+19
| | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| * tweak iterate block parameters (#390)Dan Engelbrecht2025-05-121-32/+45
| | | | | | * tweak block iteration chunk sizes
| * optimize cache bucket state writing (#382)Dan Engelbrecht2025-05-061-42/+59
| | | | | | * optimize cache bucket snapshot and sidecar writing
| * iterate chunks crash fix (#376)Dan Engelbrecht2025-05-021-1/+1
| | | | | | * Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
| * cbobject validation (#377)Dan Engelbrecht2025-05-021-10/+31
| | | | | | | | * validate incoming CbObject to cache when receiving a package * validate records when fetched from store in cache before parsing them
| * replace local equal_to_2 with eastl impl (#368)Stefan Boberg2025-04-251-16/+2
| |
| * predicate to enable compiling with later EASTL version (#367)Stefan Boberg2025-04-241-1/+1
| |
| * reduce disk io during gc (#335)Dan Engelbrecht2025-04-011-20/+8
| | | | | | * do cache bucket flush/write snapshot as part of compact to reduce disk I/O
| * long filename support (#330)Dan Engelbrecht2025-03-311-24/+24
| | | | | | - Bugfix: Long file paths now works correctly on Windows
| * reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-063-89/+113
| | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* | Change to PutResult structurezousar2025-06-243-94/+150
| | | | | | | | Result structure contains status and a string message (may be empty)
* | Establish TODOs and unit test for rejected PUT propagationzousar2025-03-241-0/+1
| |
* | Control overwrite enforcement with a config settingzousar2025-03-021-1/+2
| |
* | Move utility methods in cachedisklayerzousar2025-02-261-38/+38
| | | | | | | | Value comparison methods moved to more appropriate area in file.
* | Enforce Overwrite Prevention According To Cache Policyzousar2025-02-263-91/+234
|/ | | | Overwrite with differing value should be denied if QueryLocal is not present and StoreLocal is present. Overwrite with equal value should succeed regardless of policy flags.
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-222-2/+2
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-161-1/+1
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Miscellaneous minor LLM fixes (#268)v5.5.17-pre0Stefan Boberg2024-12-171-0/+2
| | | | | | | With this change, LLM tags are assigned using the name,parent tuple rather than just by name only. This allows tag hierarchies like `cache/store` and `project/store` which would previously get collapsed into the first pair seen when registering the `store` tag. This PR also adds some more LLM tag annotations to more accurately associate memory allocations with subsystems In addition, this PR also tweaks the frequency of timer marker events to increase the resolution in Insights and avoid some cases of Insights deciding that marker events are too far apart since we don't allocate as frequently as UE tends to.
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-20 11:20:47 +0000