aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/cache
Commit message (Collapse)AuthorAgeFilesLines
* don't read chunks into memory during cache batch fetch unless we may cache ↵Dan Engelbrecht2024-10-091-1/+2
| | | | | them in memory (#188) * Don't read chunks into memory during cache batch fetch unless we may cache them in memory
* remove gc v1 (#121)Dan Engelbrecht2024-10-032-787/+17
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc block size target max size (#180)Dan Engelbrecht2024-10-021-4/+0
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* optimize gc reference sort (#179)Dan Engelbrecht2024-10-011-4/+3
| | | | | - Do a single call to mempcy when fetching attachments from the meta store in GC - Use small lambda when calling std::sort in FilterReferences (enables inlining of the comparision function) - Use a single function for < and == comparision in KeepUnusedReferences
* gc command attachment options (#176)Dan Engelbrecht2024-09-301-3/+4
| | | * zen command - add options to control meta data cache when triggering gc
* optimize startup time (#175)Dan Engelbrecht2024-09-301-8/+15
| | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
* reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-271-101/+125
| | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-252-3/+3
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* Added namespace qualifier (optional) for z$ rpc requests (#166)Stefan Boberg2024-09-231-1/+16
| | | This change adds support for a namespace-qualified RPC endpoint for z$ at `/z$/<namespace>/$rpc` which may be used to validate RPC requests by URL inspection. The old scheme is still supported.
* gc unused refactor (#165)Dan Engelbrecht2024-09-232-41/+37
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* unblock PreCache (#164)Dan Engelbrecht2024-09-202-6/+8
| | | Don't lock disk cache buckets from writing when scanning records for attachment references
* gc performance improvements (#160)Dan Engelbrecht2024-09-171-51/+78
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* trace scopes improvementsDan Engelbrecht2024-09-102-9/+10
|
* don't add batch overhead if we are only going to put one cache valueDan Engelbrecht2024-09-101-5/+19
|
* only add mem cached items to list to fill metadata if we don't have metadata ↵Dan Engelbrecht2024-09-101-40/+39
| | | | (#151)
* fix race condition in zenserver during batched fetch (#149)Dan Engelbrecht2024-09-091-1/+92
| | | * fix race condition in zenserver duing batched fetch
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-041-19/+31
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-042-1/+24
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-031-0/+2
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-301-16/+113
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-232-168/+208
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-2/+2
| | | | blocking client requests (#134)
* remove bad assert when payload is memcached but metadata is not set (#130)Dan Engelbrecht2024-08-221-1/+0
|
* don't throw exception if sidecar file is missing (#126)Dan Engelbrecht2024-08-191-1/+14
| | | | * Don't throw exception if sidecar file is missing. * Log reasons for rejecting sidecar file
* If we fail to get compression info for a partial request - log and report a ↵Dan Engelbrecht2024-08-191-6/+18
| | | | | miss (#122) * If we fail to get compression info for a partial request - log and report a miss
* don't try to memcache the empty buffer if invalid format (#110)Dan Engelbrecht2024-08-131-1/+3
|
* validate cbobject before iterating for attachments to avoid crash on ↵Dan Engelbrecht2024-08-061-7/+10
| | | | malformed data (#101)
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-132-126/+414
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-043-88/+583
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* cache optimizations (#88)Dan Engelbrecht2024-05-302-2/+10
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-171-60/+46
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* safer partial requests (#82)Dan Engelbrecht2024-05-171-3/+2
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* fix argument order when logging missed partial chunk (#81)Dan Engelbrecht2024-05-141-8/+11
| | | | | | * fix argument order when logging missed partial chunk * make partial value a temporary variable helps with debugging as the warning on bad range can tell the original size * add asserts
* return as missing if out of range chunk (#78)Dan Engelbrecht2024-05-081-20/+24
| | | * treat invalid partial chunk request as a miss
* check partial chunk result (#73)Dan Engelbrecht2024-05-031-9/+26
| | | | * validate result when getting partial chunk and warn on failure * changelog
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-021-1/+6
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-2/+2
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-023-115/+425
| | | - Improvement: Batch scope for put of cache values
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-261-4/+4
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* iterate cas chunks (#59)Dan Engelbrecht2024-04-241-13/+14
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-0/+2
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-221-39/+52
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* improved assert (#37)Dan Engelbrecht2024-04-042-13/+13
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-031-14/+15
| | | * Validate size of found chunks in cas/cache
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-281-1/+1
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* re-enable partial cache chunks (#21)v5.4.2-pre9v5.4.2-pre12v5.4.2-pre11v5.4.2-pre10Dan Engelbrecht2024-03-221-19/+28
| | | | * Separate chunk raw hash from section hash (how to find the fragment attachment) * fix partial get cache value tests
* disable partial getcachechunk responses (#19)Dan Engelbrecht2024-03-221-1/+7
|
* add support for responding with partial cache chunks (#11)Dan Engelbrecht2024-03-211-3/+33
| | | * add support for responding with partial cache chunks
* clean up test linking (#4)Dan Engelbrecht2024-03-142-59/+13
| | | | | | | - Improvement: Add zenhttp-test and zenutil-test - Improvement: Moved cachepolicy test to cachepolicy.cpp - Improvement: Renamed cachestore tests from z$ to cachestore - Improvement: Moved test linking so test for a lib is linked by <lib>-test - Improvement: Removed HttpRequestParseRelativeUri in httpstructuredcache.cpp and use the one in cacherequests.h instead
* http request parser safety (#664)Dan Engelbrecht2024-03-121-2/+2
| | | | * make sure we don't add more headers than we support * don't capture for loop variables by reference for async work
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-22 10:58:17 +0000