aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/cache
Commit message (Collapse)AuthorAgeFilesLines
* gc performance improvements (#160)Dan Engelbrecht2024-09-171-51/+78
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* trace scopes improvementsDan Engelbrecht2024-09-102-9/+10
|
* don't add batch overhead if we are only going to put one cache valueDan Engelbrecht2024-09-101-5/+19
|
* only add mem cached items to list to fill metadata if we don't have metadata ↵Dan Engelbrecht2024-09-101-40/+39
| | | | (#151)
* fix race condition in zenserver during batched fetch (#149)Dan Engelbrecht2024-09-091-1/+92
| | | * fix race condition in zenserver duing batched fetch
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-041-19/+31
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-042-1/+24
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-031-0/+2
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-301-16/+113
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-232-168/+208
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-2/+2
| | | | blocking client requests (#134)
* remove bad assert when payload is memcached but metadata is not set (#130)Dan Engelbrecht2024-08-221-1/+0
|
* don't throw exception if sidecar file is missing (#126)Dan Engelbrecht2024-08-191-1/+14
| | | | * Don't throw exception if sidecar file is missing. * Log reasons for rejecting sidecar file
* If we fail to get compression info for a partial request - log and report a ↵Dan Engelbrecht2024-08-191-6/+18
| | | | | miss (#122) * If we fail to get compression info for a partial request - log and report a miss
* don't try to memcache the empty buffer if invalid format (#110)Dan Engelbrecht2024-08-131-1/+3
|
* validate cbobject before iterating for attachments to avoid crash on ↵Dan Engelbrecht2024-08-061-7/+10
| | | | malformed data (#101)
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-132-126/+414
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-043-88/+583
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* cache optimizations (#88)Dan Engelbrecht2024-05-302-2/+10
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-171-60/+46
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* safer partial requests (#82)Dan Engelbrecht2024-05-171-3/+2
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* fix argument order when logging missed partial chunk (#81)Dan Engelbrecht2024-05-141-8/+11
| | | | | | * fix argument order when logging missed partial chunk * make partial value a temporary variable helps with debugging as the warning on bad range can tell the original size * add asserts
* return as missing if out of range chunk (#78)Dan Engelbrecht2024-05-081-20/+24
| | | * treat invalid partial chunk request as a miss
* check partial chunk result (#73)Dan Engelbrecht2024-05-031-9/+26
| | | | * validate result when getting partial chunk and warn on failure * changelog
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-021-1/+6
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-2/+2
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-023-115/+425
| | | - Improvement: Batch scope for put of cache values
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-261-4/+4
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* iterate cas chunks (#59)Dan Engelbrecht2024-04-241-13/+14
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-0/+2
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-221-39/+52
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* improved assert (#37)Dan Engelbrecht2024-04-042-13/+13
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-031-14/+15
| | | * Validate size of found chunks in cas/cache
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-281-1/+1
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* re-enable partial cache chunks (#21)v5.4.2-pre9v5.4.2-pre12v5.4.2-pre11v5.4.2-pre10Dan Engelbrecht2024-03-221-19/+28
| | | | * Separate chunk raw hash from section hash (how to find the fragment attachment) * fix partial get cache value tests
* disable partial getcachechunk responses (#19)Dan Engelbrecht2024-03-221-1/+7
|
* add support for responding with partial cache chunks (#11)Dan Engelbrecht2024-03-211-3/+33
| | | * add support for responding with partial cache chunks
* clean up test linking (#4)Dan Engelbrecht2024-03-142-59/+13
| | | | | | | - Improvement: Add zenhttp-test and zenutil-test - Improvement: Moved cachepolicy test to cachepolicy.cpp - Improvement: Renamed cachestore tests from z$ to cachestore - Improvement: Moved test linking so test for a lib is linked by <lib>-test - Improvement: Removed HttpRequestParseRelativeUri in httpstructuredcache.cpp and use the one in cacherequests.h instead
* http request parser safety (#664)Dan Engelbrecht2024-03-121-2/+2
| | | | * make sure we don't add more headers than we support * don't capture for loop variables by reference for async work
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-281-9/+31
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* improved block store logging and more gcv2 tests (#659)Dan Engelbrecht2024-02-272-11/+3
| | | | * improved gc/blockstore logging * more gcv2 tests
* remove reference caching (#658)Dan Engelbrecht2024-02-272-638/+185
| | | * remove reference caching
* hashing fixes (#657)Dan Engelbrecht2024-02-262-88/+75
| | | | | * move structuredcachestore tests to zenstore-test * Don't materialize entire files when hashing if it is a large files * rewrite CompositeBuffer::Mid to never materialize buffers
* respond with BadRequest result instead of throwing exception on bad request ↵Dan Engelbrecht2024-02-051-7/+25
| | | | input (#648)
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-203-2/+1642
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-192-0/+6837
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-29 03:14:04 +0000