aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
...
* only add mem cached items to list to fill metadata if we don't have metadata ↵Dan Engelbrecht2024-09-101-40/+39
| | | | (#151)
* fix race condition in zenserver during batched fetch (#149)Dan Engelbrecht2024-09-091-1/+92
| | | * fix race condition in zenserver duing batched fetch
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-042-22/+38
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-046-2/+38
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-034-4/+8
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-304-30/+284
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-232-168/+208
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-223-5/+5
| | | | blocking client requests (#134)
* remove bad assert when payload is memcached but metadata is not set (#130)Dan Engelbrecht2024-08-221-1/+0
|
* close payload file if size mismatch for file cas (#128)Dan Engelbrecht2024-08-201-2/+2
|
* don't throw exception if sidecar file is missing (#126)Dan Engelbrecht2024-08-191-1/+14
| | | | * Don't throw exception if sidecar file is missing. * Log reasons for rejecting sidecar file
* if disk space is low, set the last gc time to avoid spamming retries (#124)Dan Engelbrecht2024-08-191-0/+2
| | | * if disk space is low, set the last gc time to avoid spamming retries
* If we fail to get compression info for a partial request - log and report a ↵Dan Engelbrecht2024-08-191-6/+18
| | | | | miss (#122) * If we fail to get compression info for a partial request - log and report a miss
* prevent new block in gc (#118)Dan Engelbrecht2024-08-151-1/+1
| | | * make sure we don't reset write-pos for new block for each block iterated
* improved logging removing unimportant information (#116)Dan Engelbrecht2024-08-141-24/+33
|
* hardening and reduced spam from GC on failure (#112)Dan Engelbrecht2024-08-142-143/+232
| | | | * Retry writing GC state if it fails to handle transient problems * If GC operation fails demote errors to warnings on consecutive fails
* don't try to memcache the empty buffer if invalid format (#110)Dan Engelbrecht2024-08-131-1/+3
|
* Skip chunk in block stores when iterating a block if the location is out of ↵Dan Engelbrecht2024-08-121-2/+7
| | | | range (#109)
* add gc single threaded option (#104)Dan Engelbrecht2024-08-072-5/+13
| | | * add option to force gcv2 to run single threaded
* stop exceptions from leaking on threaded work (#102)Dan Engelbrecht2024-08-062-26/+46
| | | | * catch exceptions in threaded work * don't abort all project file/chunk info fetch for single failure
* validate cbobject before iterating for attachments to avoid crash on ↵Dan Engelbrecht2024-08-061-7/+10
| | | | malformed data (#101)
* changelog (#99)Dan Engelbrecht2024-08-061-3/+20
| | | Skip and report invalid configurations for workspaces instead of crashing
* don't assert that we have moved bytes if source block is zero size (#97)Dan Engelbrecht2024-06-141-1/+2
| | | | * don't assert that we have moved bytes if source block is zero size * handle invalid session ids gracefully
* workspace share path hardening (#95)Dan Engelbrecht2024-06-141-0/+45
| | | | | | * resolve relative paths for root path * block share paths that go outside of root path * fix test using invalid share_path * validate that root path is absolute
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-136-175/+571
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-112-141/+102
| | | | * fix alias request capture * use single config file for workspaces
* workspace share aliases (#91)Dan Engelbrecht2024-06-042-4/+136
| | | | | | | - Add `zen workspace-share` `--root-path` option - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If `--workspace` is omitted, an id will be generated from the `--root-path` parameter - Add `/ws/share/{alias}/` endpoint - a shortcut to `/ws/{workspace_id}/{share_id}/` based endpoints using the alias for a workspace share - Add `--alias` option to replace `--workspace` and `--share` options for `workspace-share` zen commands - Rename `zen workspace create` `folder` option to `root-path` - Rename `zen workspace create` `folder` option to `share-path`
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-047-121/+679
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* cache optimizations (#88)Dan Engelbrecht2024-05-302-2/+10
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* workspace shares (#84)Dan Engelbrecht2024-05-293-0/+1059
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-174-279/+308
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* safer partial requests (#82)Dan Engelbrecht2024-05-171-3/+2
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* fix argument order when logging missed partial chunk (#81)Dan Engelbrecht2024-05-141-8/+11
| | | | | | * fix argument order when logging missed partial chunk * make partial value a temporary variable helps with debugging as the warning on bad range can tell the original size * add asserts
* return as missing if out of range chunk (#78)Dan Engelbrecht2024-05-081-20/+24
| | | * treat invalid partial chunk request as a miss
* check partial chunk result (#73)Dan Engelbrecht2024-05-031-9/+26
| | | | * validate result when getting partial chunk and warn on failure * changelog
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-021-1/+6
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-023-16/+4
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-025-121/+471
| | | - Improvement: Batch scope for put of cache values
* fix get project files loop (#68)Dan Engelbrecht2024-04-303-17/+57
| | | | | - Bugfix: Remove extra loop causing GetProjectFiles for project store to find all chunks once for each chunk found - Bugfix: Don't capture ChunkIndex variable in CasImpl::IterateChunks by reference as it causes crash - Improvement: Make FileCasStrategy::IterateChunks (optionally) multithreaded (improves GetProjectFiles performance)
* oplog iterate chunks content type (#65)Dan Engelbrecht2024-04-261-3/+24
| | | | - Bugfix: Properly set content type of chunks fetch from CidStore - Improvement: Add IterateChunks(std::span<Oid>) for better performance in get oplog
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-263-9/+9
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* iterate cas chunks (#59)Dan Engelbrecht2024-04-2411-112/+268
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-245-2/+28
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-229-74/+473
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* gc v2 disk freed space fix and oplog stats report improvement (#45)Dan Engelbrecht2024-04-151-10/+15
| | | | | - Bugfix: Correctly calculate size freed/data moved from blocks in GCv2 - Improvement: Reduced details in remote store stats for oplog export/import to user - Improvement: Transfer speed for oplog export/import is now an overall number rather than average of speed per single request
* improved assert (#37)Dan Engelbrecht2024-04-046-35/+35
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-034-20/+34
| | | * Validate size of found chunks in cas/cache
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-283-4/+4
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* re-enable partial cache chunks (#21)v5.4.2-pre9v5.4.2-pre12v5.4.2-pre11v5.4.2-pre10Dan Engelbrecht2024-03-221-19/+28
| | | | * Separate chunk raw hash from section hash (how to find the fragment attachment) * fix partial get cache value tests
* disable partial getcachechunk responses (#19)Dan Engelbrecht2024-03-221-1/+7
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-22 16:27:30 +0000