aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-136-175/+571
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-112-141/+102
| | | | * fix alias request capture * use single config file for workspaces
* workspace share aliases (#91)Dan Engelbrecht2024-06-042-4/+136
| | | | | | | - Add `zen workspace-share` `--root-path` option - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If `--workspace` is omitted, an id will be generated from the `--root-path` parameter - Add `/ws/share/{alias}/` endpoint - a shortcut to `/ws/{workspace_id}/{share_id}/` based endpoints using the alias for a workspace share - Add `--alias` option to replace `--workspace` and `--share` options for `workspace-share` zen commands - Rename `zen workspace create` `folder` option to `root-path` - Rename `zen workspace create` `folder` option to `share-path`
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-047-121/+679
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* cache optimizations (#88)Dan Engelbrecht2024-05-302-2/+10
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* workspace shares (#84)Dan Engelbrecht2024-05-293-0/+1059
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-174-279/+308
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* safer partial requests (#82)Dan Engelbrecht2024-05-171-3/+2
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* fix argument order when logging missed partial chunk (#81)Dan Engelbrecht2024-05-141-8/+11
| | | | | | * fix argument order when logging missed partial chunk * make partial value a temporary variable helps with debugging as the warning on bad range can tell the original size * add asserts
* return as missing if out of range chunk (#78)Dan Engelbrecht2024-05-081-20/+24
| | | * treat invalid partial chunk request as a miss
* check partial chunk result (#73)Dan Engelbrecht2024-05-031-9/+26
| | | | * validate result when getting partial chunk and warn on failure * changelog
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-021-1/+6
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-023-16/+4
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-025-121/+471
| | | - Improvement: Batch scope for put of cache values
* fix get project files loop (#68)Dan Engelbrecht2024-04-303-17/+57
| | | | | - Bugfix: Remove extra loop causing GetProjectFiles for project store to find all chunks once for each chunk found - Bugfix: Don't capture ChunkIndex variable in CasImpl::IterateChunks by reference as it causes crash - Improvement: Make FileCasStrategy::IterateChunks (optionally) multithreaded (improves GetProjectFiles performance)
* oplog iterate chunks content type (#65)Dan Engelbrecht2024-04-261-3/+24
| | | | - Bugfix: Properly set content type of chunks fetch from CidStore - Improvement: Add IterateChunks(std::span<Oid>) for better performance in get oplog
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-263-9/+9
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* iterate cas chunks (#59)Dan Engelbrecht2024-04-2411-112/+268
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-245-2/+28
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-229-74/+473
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* gc v2 disk freed space fix and oplog stats report improvement (#45)Dan Engelbrecht2024-04-151-10/+15
| | | | | - Bugfix: Correctly calculate size freed/data moved from blocks in GCv2 - Improvement: Reduced details in remote store stats for oplog export/import to user - Improvement: Transfer speed for oplog export/import is now an overall number rather than average of speed per single request
* improved assert (#37)Dan Engelbrecht2024-04-046-35/+35
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-034-20/+34
| | | * Validate size of found chunks in cas/cache
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-283-4/+4
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* re-enable partial cache chunks (#21)v5.4.2-pre9v5.4.2-pre12v5.4.2-pre11v5.4.2-pre10Dan Engelbrecht2024-03-221-19/+28
| | | | * Separate chunk raw hash from section hash (how to find the fragment attachment) * fix partial get cache value tests
* disable partial getcachechunk responses (#19)Dan Engelbrecht2024-03-221-1/+7
|
* add support for responding with partial cache chunks (#11)Dan Engelbrecht2024-03-212-5/+36
| | | * add support for responding with partial cache chunks
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-143-1/+560
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* clean up test linking (#4)Dan Engelbrecht2024-03-143-59/+15
| | | | | | | - Improvement: Add zenhttp-test and zenutil-test - Improvement: Moved cachepolicy test to cachepolicy.cpp - Improvement: Renamed cachestore tests from z$ to cachestore - Improvement: Moved test linking so test for a lib is linked by <lib>-test - Improvement: Removed HttpRequestParseRelativeUri in httpstructuredcache.cpp and use the one in cacherequests.h instead
* fix potential partially written files (#2)Dan Engelbrecht2024-03-133-8/+36
| | | | * Make sure WriteFile() does not leave incomplete files * use TemporaryFile and MoveTemporaryIntoPlace to avoid leaving partial files on error
* http request parser safety (#664)Dan Engelbrecht2024-03-121-2/+2
| | | | * make sure we don't add more headers than we support * don't capture for loop variables by reference for async work
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-286-61/+142
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* add disk caching to block move (#661)Dan Engelbrecht2024-02-271-23/+36
| | | * add disk caching to block move
* improved block store logging and more gcv2 tests (#659)Dan Engelbrecht2024-02-276-114/+206
| | | | * improved gc/blockstore logging * more gcv2 tests
* remove reference caching (#658)Dan Engelbrecht2024-02-273-681/+193
| | | * remove reference caching
* hashing fixes (#657)Dan Engelbrecht2024-02-264-89/+77
| | | | | * move structuredcachestore tests to zenstore-test * Don't materialize entire files when hashing if it is a large files * rewrite CompositeBuffer::Mid to never materialize buffers
* respond with BadRequest result instead of throwing exception on bad request ↵Dan Engelbrecht2024-02-051-7/+25
| | | | input (#648)
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-242-80/+86
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-208-3/+1917
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-195-0/+7690
|
* Don't capture local variables in loop by reference (#623)Dan Engelbrecht2023-12-191-27/+27
| | | * Don't capture local variables in loop by reference
* fix ChunkIndexToChunkHash indexing (#621)Stefan Boberg2023-12-191-1/+1
| | | would previously index into a reserved-but-not-sized vector which is bad but not crash-inducing bad
* Fix crash bug when trying to inspect non-open block file in GC (#614)Dan Engelbrecht2023-12-182-7/+20
|
* log each block that is compacted (#613)Dan Engelbrecht2023-12-151-0/+5
|
* fix peak disk load in gc status (#608)Dan Engelbrecht2023-12-131-12/+11
| | | * MaxLoad is max load per monitor slot, not the MaxLoad for the entire graph
* skip invalid chunks when reclaiming space in block store (#607)Dan Engelbrecht2023-12-131-0/+13
| | | * skip invalid chunks when reclaiming space in block store
* improve trace (#606)Dan Engelbrecht2023-12-133-1/+12
| | | | | * Adding some more trace scopes for better visiblity * Removed spammy trace scope when replaying oplogs * Remove "::Disk" from trace scopes - redundant now that we have merge disk and memory layers
* Make sure that PathFromHandle don't hide true error when throwing exceptions ↵Dan Engelbrecht2023-12-111-1/+2
| | | | | | | | | (#600) * Make sure that PathFromHandle don't hide true error when throwing exceptions * changelog * return error info in PathFromHandle if path fails to resolve
* improved scrubbing of oplogs and filecas (#596)Stefan Boberg2023-12-116-46/+113
| | | | | | - Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks) - Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command - Improvement: Implemented oplog scrubbing (previously was a no-op) - Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
* fixed bug in CasContainerStrategy::ReadIndexFile (#595)Stefan Boberg2023-12-071-1/+3
| | | this was introduced in a recent optimization and would cause CAS items to not be found after a shutdown/restart cycle
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-24 02:48:15 +0000