aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/include
Commit message (Collapse)AuthorAgeFilesLines
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-205-42/+64
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-121-14/+10
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-081-41/+46
|\
| * refactor blobstore (#458)Dan Engelbrecht2025-08-061-41/+46
| | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
* | Moving put rejections to happen in batch handlingzousar2025-08-051-8/+14
| |
* | xmake precommitzousar2025-06-241-3/+3
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-248-153/+352
|\|
| * missing chunks bugfix (#424)Dan Engelbrecht2025-06-092-5/+8
| | | | | | | | | | | | | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
| * add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-301-2/+3
| | | | | | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
| * faster oplog validate (#408)Dan Engelbrecht2025-05-301-2/+3
| | | | | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
| * unblock cache bucket drop (#406)Dan Engelbrecht2025-05-262-14/+14
| | | | | | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
| * keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-121-5/+5
| | | | | | | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
| * enable per bucket config (#388)Dan Engelbrecht2025-05-121-0/+3
| | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| * blobstore size limit (#342)Dan Engelbrecht2025-04-041-1/+20
| | | | | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
| * build store save access times (#341)v5.6.3-pre0Dan Engelbrecht2025-04-032-11/+40
| | | | | | | | | | * save payload size in log for buildstore * read/write access times and manifest for buldstore * use retry when removing temporary files
| * use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-13/+0
| | | | | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
| * zen build cache service (#318)Dan Engelbrecht2025-03-266-39/+239
| | | | | | | | | | | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
| * reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-063-19/+29
| | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
| * moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-54/+0
| |
* | Change to PutResult structurezousar2025-06-244-40/+51
| | | | | | | | Result structure contains status and a string message (may be empty)
* | Control overwrite enforcement with a config settingzousar2025-03-021-0/+1
| |
* | Enforce Overwrite Prevention According To Cache Policyzousar2025-02-263-10/+18
|/ | | | Overwrite with differing value should be denied if QueryLocal is not present and StoreLocal is present. Overwrite with equal value should succeed regardless of policy flags.
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-162-2/+2
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Unity build fixes (#253)Stefan Boberg2024-12-051-1/+5
| | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-252-2/+3
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-153-26/+62
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* workspace share security (#192)Dan Engelbrecht2024-10-231-21/+62
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* bucket size queries (#203)Dan Engelbrecht2024-10-213-1/+29
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* remove gc v1 (#121)Dan Engelbrecht2024-10-034-100/+17
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc block size target max size (#180)Dan Engelbrecht2024-10-021-2/+11
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* gc command attachment options (#176)Dan Engelbrecht2024-09-302-7/+14
| | | * zen command - add options to control meta data cache when triggering gc
* optimize startup time (#175)Dan Engelbrecht2024-09-301-9/+3
| | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
* reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-271-3/+4
| | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-4/+11
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* Added namespace qualifier (optional) for z$ rpc requests (#166)Stefan Boberg2024-09-231-0/+1
| | | This change adds support for a namespace-qualified RPC endpoint for z$ at `/z$/<namespace>/$rpc` which may be used to validate RPC requests by URL inspection. The old scheme is still supported.
* gc unused refactor (#165)Dan Engelbrecht2024-09-232-18/+27
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* made fmt formatter format function const (#162)Stefan Boberg2024-09-201-1/+1
| | | this appears to be required as of fmt v11
* unblock PreCache (#164)Dan Engelbrecht2024-09-201-1/+1
| | | Don't lock disk cache buckets from writing when scanning records for attachment references
* gc performance improvements (#160)Dan Engelbrecht2024-09-172-29/+22
| | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
* clean cache slog files on startup (#143)Dan Engelbrecht2024-09-041-3/+7
| | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-0/+1
| | | - Improvement: Move GC logging in callback functions into "gc" context
* oplog index snapshots (#140)Dan Engelbrecht2024-09-031-2/+2
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-302-7/+24
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* hardening and reduced spam from GC on failure (#112)Dan Engelbrecht2024-08-141-2/+3
| | | | * Retry writing GC state if it fails to handle transient problems * If GC operation fails demote errors to warnings on consecutive fails
* add gc single threaded option (#104)Dan Engelbrecht2024-08-071-1/+4
| | | * add option to force gcv2 to run single threaded
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-133-10/+72
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-111-4/+5
| | | | * fix alias request capture * use single config file for workspaces
* workspace share aliases (#91)Dan Engelbrecht2024-06-041-0/+15
| | | | | | | - Add `zen workspace-share` `--root-path` option - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If `--workspace` is omitted, an id will be generated from the `--root-path` parameter - Add `/ws/share/{alias}/` endpoint - a shortcut to `/ws/{workspace_id}/{share_id}/` based endpoints using the alias for a workspace share - Add `--alias` option to replace `--workspace` and `--share` options for `workspace-share` zen commands - Rename `zen workspace create` `folder` option to `root-path` - Rename `zen workspace create` `folder` option to `share-path`
* add batching of CacheStore requests for GetCacheValues/GetCacheChunks (#90)Dan Engelbrecht2024-06-043-19/+56
| | | | | | * cache file size of block on open * add ability to control size limit for small chunk callback when iterating block * Add batch fetch of cache values in the GetCacheValues request
* workspace shares (#84)Dan Engelbrecht2024-05-291-0/+102
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 04:43:50 +0000