aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/cache/cachedisklayer.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+15
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-251-2/+3
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-151-10/+13
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* memory leak fix (GetBatchHandle) (#215)Stefan Boberg2024-11-081-0/+4
| | | This fixes a memory leak which would cause stale handles to accumulate until process shutdown. Each cache get operation would therefore leak some memory.
* bucket size queries (#203)Dan Engelbrecht2024-10-211-12/+54
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* don't read chunks into memory during cache batch fetch unless we may cache ↵Dan Engelbrecht2024-10-091-1/+2
| | | | | them in memory (#188) * Don't read chunks into memory during cache batch fetch unless we may cache them in memory
* remove gc v1 (#121)Dan Engelbrecht2024-10-031-504/+0
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc block size target max size (#180)Dan Engelbrecht2024-10-021-4/+0
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* optimize gc reference sort (#179)Dan Engelbrecht2024-10-011-4/+3
| | | | | - Do a single call to mempcy when fetching attachments from the meta store in GC - Use small lambda when calling std::sort in FilterReferences (enables inlining of the comparision function) - Use a single function for < and == comparision in KeepUnusedReferences
* gc command attachment options (#176)Dan Engelbrecht2024-09-301-3/+4
| | | * zen command - add options to control meta data cache when triggering gc
* optimize startup time (#175)Dan Engelbrecht2024-09-301-8/+15
| | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
* reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-271-101/+125
| | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-2/+2
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* gc unused refactor (#165)Dan Engelbrecht2024-09-231-27/+27
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* unblock PreCache (#164)Dan Engelbrecht2024-09-201-5/+7
| | | Don't lock disk cache buckets from writing when scanning records for attachment references
* gc performance improvements (#160)Dan Engelbrecht2024-09-171-51/+78
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* trace scopes improvementsDan Engelbrecht2024-09-101-7/+8
|
* only add mem cached items to list to fill metadata if we don't have metadata ↵Dan Engelbrecht2024-09-101-40/+39
| | | | (#151)
* fix race condition in zenserver during batched fetch (#149)Dan Engelbrecht2024-09-091-1/+92
| | | * fix race condition in zenserver duing batched fetch
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-041-19/+31
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-0/+15
| | | - Improvement: Move GC logging in callback functions into "gc" context
* meta info store (#75)Dan Engelbrecht2024-08-301-16/+113
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-231-162/+202
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-2/+2
| | | | blocking client requests (#134)
* remove bad assert when payload is memcached but metadata is not set (#130)Dan Engelbrecht2024-08-221-1/+0
|
* don't throw exception if sidecar file is missing (#126)Dan Engelbrecht2024-08-191-1/+14
| | | | * Don't throw exception if sidecar file is missing. * Log reasons for rejecting sidecar file
* don't try to memcache the empty buffer if invalid format (#110)Dan Engelbrecht2024-08-131-1/+3
|
* validate cbobject before iterating for attachments to avoid crash on ↵Dan Engelbrecht2024-08-061-7/+10
| | | | malformed data (#101)
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-131-126/+179
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-041-23/+337
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* cache optimizations (#88)Dan Engelbrecht2024-05-301-0/+8
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-171-60/+46
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-2/+2
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-021-6/+207
| | | - Improvement: Batch scope for put of cache values
* iterate cas chunks (#59)Dan Engelbrecht2024-04-241-13/+14
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-0/+2
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* improved assert (#37)Dan Engelbrecht2024-04-041-11/+11
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-031-14/+15
| | | * Validate size of found chunks in cas/cache
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-281-1/+1
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* http request parser safety (#664)Dan Engelbrecht2024-03-121-2/+2
| | | | * make sure we don't add more headers than we support * don't capture for loop variables by reference for async work
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-281-9/+31
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* improved block store logging and more gcv2 tests (#659)Dan Engelbrecht2024-02-271-1/+2
| | | | * improved gc/blockstore logging * more gcv2 tests
* remove reference caching (#658)Dan Engelbrecht2024-02-271-503/+101
| | | * remove reference caching
* hashing fixes (#657)Dan Engelbrecht2024-02-261-87/+3
| | | | | * move structuredcachestore tests to zenstore-test * Don't materialize entire files when hashing if it is a large files * rewrite CompositeBuffer::Mid to never materialize buffers
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-201-1/+1
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-191-0/+4397
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 21:45:56 +0000