aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* WIPde/improved-projectstore-batch-requestsDan Engelbrecht2024-11-116-13/+24
| | | | | | | | | allow control of size for batch iteration allow adding compositebuffers as attachments directly add batch2 httpstore api to allow batching of CAS & Oid with range requests allow responses with file handles from project store Signed-off-by: Dan Engelbrecht <[email protected]>
* workspace share security (#192)Dan Engelbrecht2024-10-232-383/+890
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* Use a smaller thread pool during pre-cache phase of GC to reduce memory ↵Dan Engelbrecht2024-10-221-7/+11
| | | | pressure (#205)
* bucket size queries (#203)Dan Engelbrecht2024-10-215-14/+106
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* validate compact binary rpc requests before trying to parse them (#200)Dan Engelbrecht2024-10-161-1/+10
|
* safer path from handle (#195)Dan Engelbrecht2024-10-161-31/+41
| | | * remove PathFromHandle that throws to give better context on failures
* don't read chunks into memory during cache batch fetch unless we may cache ↵Dan Engelbrecht2024-10-092-13/+14
| | | | | them in memory (#188) * Don't read chunks into memory during cache batch fetch unless we may cache them in memory
* remove gc v1 (#121)Dan Engelbrecht2024-10-0314-2819/+99
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc block size target max size (#180)Dan Engelbrecht2024-10-024-42/+67
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* Porject -> ProjectStefan Boberg2024-10-021-1/+1
|
* optimize gc reference sort (#179)Dan Engelbrecht2024-10-012-37/+40
| | | | | - Do a single call to mempcy when fetching attachments from the meta store in GC - Use small lambda when calling std::sort in FilterReferences (enables inlining of the comparision function) - Use a single function for < and == comparision in KeepUnusedReferences
* use alternate IoHash comparision function (#177)v5.5.8-pre5Dan Engelbrecht2024-09-301-4/+23
| | | * Use alternate IoHash comparision function - reduces KeepUnusedReferences execution time by ~20%
* gc command attachment options (#176)Dan Engelbrecht2024-09-304-17/+48
| | | * zen command - add options to control meta data cache when triggering gc
* optimize startup time (#175)Dan Engelbrecht2024-09-304-35/+26
| | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
* reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-272-104/+129
| | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-256-45/+166
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* exception safety when writing block (#168)Dan Engelbrecht2024-09-251-10/+8
| | | | * make sure we always clear writing block from m_ActiveWriteBlocks even if we have an exception
* Added namespace qualifier (optional) for z$ rpc requests (#166)Stefan Boberg2024-09-232-1/+17
| | | This change adds support for a namespace-qualified RPC endpoint for z$ at `/z$/<namespace>/$rpc` which may be used to validate RPC requests by URL inspection. The old scheme is still supported.
* gc unused refactor (#165)Dan Engelbrecht2024-09-237-85/+324
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* made fmt formatter format function const (#162)Stefan Boberg2024-09-201-1/+1
| | | this appears to be required as of fmt v11
* unblock PreCache (#164)Dan Engelbrecht2024-09-203-7/+9
| | | Don't lock disk cache buckets from writing when scanning records for attachment references
* gc performance improvements (#160)Dan Engelbrecht2024-09-175-96/+117
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* trace scopes improvementsDan Engelbrecht2024-09-103-9/+11
|
* don't add batch overhead if we are only going to put one cache valueDan Engelbrecht2024-09-101-5/+19
|
* only add mem cached items to list to fill metadata if we don't have metadata ↵Dan Engelbrecht2024-09-101-40/+39
| | | | (#151)
* fix race condition in zenserver during batched fetch (#149)Dan Engelbrecht2024-09-091-1/+92
| | | * fix race condition in zenserver duing batched fetch
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-042-22/+38
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-046-2/+38
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-034-4/+8
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-304-30/+284
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-232-168/+208
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-223-5/+5
| | | | blocking client requests (#134)
* remove bad assert when payload is memcached but metadata is not set (#130)Dan Engelbrecht2024-08-221-1/+0
|
* close payload file if size mismatch for file cas (#128)Dan Engelbrecht2024-08-201-2/+2
|
* don't throw exception if sidecar file is missing (#126)Dan Engelbrecht2024-08-191-1/+14
| | | | * Don't throw exception if sidecar file is missing. * Log reasons for rejecting sidecar file
* if disk space is low, set the last gc time to avoid spamming retries (#124)Dan Engelbrecht2024-08-191-0/+2
| | | * if disk space is low, set the last gc time to avoid spamming retries
* If we fail to get compression info for a partial request - log and report a ↵Dan Engelbrecht2024-08-191-6/+18
| | | | | miss (#122) * If we fail to get compression info for a partial request - log and report a miss
* prevent new block in gc (#118)Dan Engelbrecht2024-08-151-1/+1
| | | * make sure we don't reset write-pos for new block for each block iterated
* improved logging removing unimportant information (#116)Dan Engelbrecht2024-08-141-24/+33
|
* hardening and reduced spam from GC on failure (#112)Dan Engelbrecht2024-08-142-143/+232
| | | | * Retry writing GC state if it fails to handle transient problems * If GC operation fails demote errors to warnings on consecutive fails
* don't try to memcache the empty buffer if invalid format (#110)Dan Engelbrecht2024-08-131-1/+3
|
* Skip chunk in block stores when iterating a block if the location is out of ↵Dan Engelbrecht2024-08-121-2/+7
| | | | range (#109)
* add gc single threaded option (#104)Dan Engelbrecht2024-08-072-5/+13
| | | * add option to force gcv2 to run single threaded
* stop exceptions from leaking on threaded work (#102)Dan Engelbrecht2024-08-062-26/+46
| | | | * catch exceptions in threaded work * don't abort all project file/chunk info fetch for single failure
* validate cbobject before iterating for attachments to avoid crash on ↵Dan Engelbrecht2024-08-061-7/+10
| | | | malformed data (#101)
* changelog (#99)Dan Engelbrecht2024-08-061-3/+20
| | | Skip and report invalid configurations for workspaces instead of crashing
* don't assert that we have moved bytes if source block is zero size (#97)Dan Engelbrecht2024-06-141-1/+2
| | | | * don't assert that we have moved bytes if source block is zero size * handle invalid session ids gracefully
* workspace share path hardening (#95)Dan Engelbrecht2024-06-141-0/+45
| | | | | | * resolve relative paths for root path * block share paths that go outside of root path * fix test using invalid share_path * validate that root path is absolute
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-136-175/+571
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-112-141/+102
| | | | * fix alias request capture * use single config file for workspaces
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-20 21:26:40 +0000