aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/include
Commit message (Collapse)AuthorAgeFilesLines
* blobstore size limit (#342)Dan Engelbrecht2025-04-041-1/+20
| | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
* build store save access times (#341)v5.6.3-pre0Dan Engelbrecht2025-04-032-11/+40
| | | | | * save payload size in log for buildstore * read/write access times and manifest for buldstore * use retry when removing temporary files
* use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-13/+0
| | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
* zen build cache service (#318)Dan Engelbrecht2025-03-266-39/+239
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-063-19/+29
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-54/+0
|
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-162-2/+2
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Unity build fixes (#253)Stefan Boberg2024-12-051-1/+5
| | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-252-2/+3
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-153-26/+62
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* workspace share security (#192)Dan Engelbrecht2024-10-231-21/+62
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* bucket size queries (#203)Dan Engelbrecht2024-10-213-1/+29
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* remove gc v1 (#121)Dan Engelbrecht2024-10-034-100/+17
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc block size target max size (#180)Dan Engelbrecht2024-10-021-2/+11
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* gc command attachment options (#176)Dan Engelbrecht2024-09-302-7/+14
| | | * zen command - add options to control meta data cache when triggering gc
* optimize startup time (#175)Dan Engelbrecht2024-09-301-9/+3
| | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
* reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-271-3/+4
| | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-4/+11
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* Added namespace qualifier (optional) for z$ rpc requests (#166)Stefan Boberg2024-09-231-0/+1
| | | This change adds support for a namespace-qualified RPC endpoint for z$ at `/z$/<namespace>/$rpc` which may be used to validate RPC requests by URL inspection. The old scheme is still supported.
* gc unused refactor (#165)Dan Engelbrecht2024-09-232-18/+27
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* made fmt formatter format function const (#162)Stefan Boberg2024-09-201-1/+1
| | | this appears to be required as of fmt v11
* unblock PreCache (#164)Dan Engelbrecht2024-09-201-1/+1
| | | Don't lock disk cache buckets from writing when scanning records for attachment references
* gc performance improvements (#160)Dan Engelbrecht2024-09-172-29/+22
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-041-3/+7
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-0/+1
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-031-2/+2
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-302-7/+24
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* hardening and reduced spam from GC on failure (#112)Dan Engelbrecht2024-08-141-2/+3
| | | | * Retry writing GC state if it fails to handle transient problems * If GC operation fails demote errors to warnings on consecutive fails
* add gc single threaded option (#104)Dan Engelbrecht2024-08-071-1/+4
| | | * add option to force gcv2 to run single threaded
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-133-10/+72
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-111-4/+5
| | | | * fix alias request capture * use single config file for workspaces
* workspace share aliases (#91)Dan Engelbrecht2024-06-041-0/+15
| | | | | | | - Add `zen workspace-share` `--root-path` option - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If `--workspace` is omitted, an id will be generated from the `--root-path` parameter - Add `/ws/share/{alias}/` endpoint - a shortcut to `/ws/{workspace_id}/{share_id}/` based endpoints using the alias for a workspace share - Add `--alias` option to replace `--workspace` and `--share` options for `workspace-share` zen commands - Rename `zen workspace create` `folder` option to `root-path` - Rename `zen workspace create` `folder` option to `share-path`
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-043-19/+56
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* workspace shares (#84)Dan Engelbrecht2024-05-291-0/+102
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-171-12/+11
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* batch cache put (#67)Dan Engelbrecht2024-05-022-6/+46
| | | - Improvement: Batch scope for put of cache values
* iterate cas chunks (#59)Dan Engelbrecht2024-04-242-2/+8
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-0/+2
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-222-9/+15
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-031-1/+1
| | | * Validate size of found chunks in cas/cache
* add support for responding with partial cache chunks (#11)Dan Engelbrecht2024-03-211-2/+3
| | | * add support for responding with partial cache chunks
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-141-0/+54
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-281-0/+4
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* improved block store logging and more gcv2 tests (#659)Dan Engelbrecht2024-02-271-1/+2
| | | | * improved gc/blockstore logging * more gcv2 tests
* remove reference caching (#658)Dan Engelbrecht2024-02-271-43/+8
| | | * remove reference caching
* hashing fixes (#657)Dan Engelbrecht2024-02-262-1/+2
| | | | | * move structuredcachestore tests to zenstore-test * Don't materialize entire files when hashing if it is a large files * rewrite CompositeBuffer::Mid to never materialize buffers
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-205-1/+275
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-193-0/+853
|
* Fix crash bug when trying to inspect non-open block file in GC (#614)Dan Engelbrecht2023-12-181-0/+1
|
* improved scrubbing of oplogs and filecas (#596)Stefan Boberg2023-12-112-1/+8
| | | | | | - Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks) - Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command - Improvement: Implemented oplog scrubbing (previously was a no-op) - Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 23:49:50 +0000