aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* add cmake build config to be able to use cliondi/add_cmake_devDmytro Ivanov2024-10-021-1/+1
|
* gc block size target max size (#180)Dan Engelbrecht2024-10-024-42/+67
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* Porject -> ProjectStefan Boberg2024-10-021-1/+1
|
* optimize gc reference sort (#179)Dan Engelbrecht2024-10-012-37/+40
| | | | | - Do a single call to mempcy when fetching attachments from the meta store in GC - Use small lambda when calling std::sort in FilterReferences (enables inlining of the comparision function) - Use a single function for < and == comparision in KeepUnusedReferences
* use alternate IoHash comparision function (#177)v5.5.8-pre5Dan Engelbrecht2024-09-301-4/+23
| | | * Use alternate IoHash comparision function - reduces KeepUnusedReferences execution time by ~20%
* gc command attachment options (#176)Dan Engelbrecht2024-09-304-17/+48
| | | * zen command - add options to control meta data cache when triggering gc
* optimize startup time (#175)Dan Engelbrecht2024-09-304-35/+26
| | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
* reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-272-104/+129
| | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-256-45/+166
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* exception safety when writing block (#168)Dan Engelbrecht2024-09-251-10/+8
| | | | * make sure we always clear writing block from m_ActiveWriteBlocks even if we have an exception
* Added namespace qualifier (optional) for z$ rpc requests (#166)Stefan Boberg2024-09-232-1/+17
| | | This change adds support for a namespace-qualified RPC endpoint for z$ at `/z$/<namespace>/$rpc` which may be used to validate RPC requests by URL inspection. The old scheme is still supported.
* gc unused refactor (#165)Dan Engelbrecht2024-09-237-85/+324
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* made fmt formatter format function const (#162)Stefan Boberg2024-09-201-1/+1
| | | this appears to be required as of fmt v11
* unblock PreCache (#164)Dan Engelbrecht2024-09-203-7/+9
| | | Don't lock disk cache buckets from writing when scanning records for attachment references
* gc performance improvements (#160)Dan Engelbrecht2024-09-175-96/+117
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* trace scopes improvementsDan Engelbrecht2024-09-103-9/+11
|
* don't add batch overhead if we are only going to put one cache valueDan Engelbrecht2024-09-101-5/+19
|
* only add mem cached items to list to fill metadata if we don't have metadata ↵Dan Engelbrecht2024-09-101-40/+39
| | | | (#151)
* fix race condition in zenserver during batched fetch (#149)Dan Engelbrecht2024-09-091-1/+92
| | | * fix race condition in zenserver duing batched fetch
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-042-22/+38
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-046-2/+38
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-034-4/+8
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-304-30/+284
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-232-168/+208
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-223-5/+5
| | | | blocking client requests (#134)
* remove bad assert when payload is memcached but metadata is not set (#130)Dan Engelbrecht2024-08-221-1/+0
|
* close payload file if size mismatch for file cas (#128)Dan Engelbrecht2024-08-201-2/+2
|
* don't throw exception if sidecar file is missing (#126)Dan Engelbrecht2024-08-191-1/+14
| | | | * Don't throw exception if sidecar file is missing. * Log reasons for rejecting sidecar file
* if disk space is low, set the last gc time to avoid spamming retries (#124)Dan Engelbrecht2024-08-191-0/+2
| | | * if disk space is low, set the last gc time to avoid spamming retries
* If we fail to get compression info for a partial request - log and report a ↵Dan Engelbrecht2024-08-191-6/+18
| | | | | miss (#122) * If we fail to get compression info for a partial request - log and report a miss
* prevent new block in gc (#118)Dan Engelbrecht2024-08-151-1/+1
| | | * make sure we don't reset write-pos for new block for each block iterated
* improved logging removing unimportant information (#116)Dan Engelbrecht2024-08-141-24/+33
|
* hardening and reduced spam from GC on failure (#112)Dan Engelbrecht2024-08-142-143/+232
| | | | * Retry writing GC state if it fails to handle transient problems * If GC operation fails demote errors to warnings on consecutive fails
* don't try to memcache the empty buffer if invalid format (#110)Dan Engelbrecht2024-08-131-1/+3
|
* Skip chunk in block stores when iterating a block if the location is out of ↵Dan Engelbrecht2024-08-121-2/+7
| | | | range (#109)
* add gc single threaded option (#104)Dan Engelbrecht2024-08-072-5/+13
| | | * add option to force gcv2 to run single threaded
* stop exceptions from leaking on threaded work (#102)Dan Engelbrecht2024-08-062-26/+46
| | | | * catch exceptions in threaded work * don't abort all project file/chunk info fetch for single failure
* validate cbobject before iterating for attachments to avoid crash on ↵Dan Engelbrecht2024-08-061-7/+10
| | | | malformed data (#101)
* changelog (#99)Dan Engelbrecht2024-08-061-3/+20
| | | Skip and report invalid configurations for workspaces instead of crashing
* don't assert that we have moved bytes if source block is zero size (#97)Dan Engelbrecht2024-06-141-1/+2
| | | | * don't assert that we have moved bytes if source block is zero size * handle invalid session ids gracefully
* workspace share path hardening (#95)Dan Engelbrecht2024-06-141-0/+45
| | | | | | * resolve relative paths for root path * block share paths that go outside of root path * fix test using invalid share_path * validate that root path is absolute
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-136-175/+571
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-112-141/+102
| | | | * fix alias request capture * use single config file for workspaces
* workspace share aliases (#91)Dan Engelbrecht2024-06-042-4/+136
| | | | | | | - Add `zen workspace-share` `--root-path` option - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If `--workspace` is omitted, an id will be generated from the `--root-path` parameter - Add `/ws/share/{alias}/` endpoint - a shortcut to `/ws/{workspace_id}/{share_id}/` based endpoints using the alias for a workspace share - Add `--alias` option to replace `--workspace` and `--share` options for `workspace-share` zen commands - Rename `zen workspace create` `folder` option to `root-path` - Rename `zen workspace create` `folder` option to `share-path`
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-047-121/+679
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* cache optimizations (#88)Dan Engelbrecht2024-05-302-2/+10
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* workspace shares (#84)Dan Engelbrecht2024-05-293-0/+1059
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-174-279/+308
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* safer partial requests (#82)Dan Engelbrecht2024-05-171-3/+2
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* fix argument order when logging missed partial chunk (#81)Dan Engelbrecht2024-05-141-8/+11
| | | | | | * fix argument order when logging missed partial chunk * make partial value a temporary variable helps with debugging as the warning on bad range can tell the original size * add asserts
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-02 04:40:00 +0000