aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/include
Commit message (Collapse)AuthorAgeFilesLines
* Merge remote-tracking branch 'origin/main' into sb/rpc-analysisStefan Boberg2025-10-0618-383/+2103
|\
| * cacherequests helpers test only (#551)Dan Engelbrecht2025-10-036-3/+376
| | | | | | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
| * move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-031-0/+103
| | | | | | * move zen vfs implementation to zenstore
| * move projectstore to zenstore (#541)Dan Engelbrecht2025-10-022-0/+629
| |
| * Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-221-6/+6
| |\
| | * more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-221-6/+6
| | | | | | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
| * | Change batch put responses for client reportingzousar2025-09-192-3/+4
| |/ | | | | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
| * revert multi-cid store (#475)Dan Engelbrecht2025-08-262-7/+3
| |
| * per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-205-42/+64
| | | | | | | | | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
| * reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-121-14/+10
| | | | | | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
| * Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-081-41/+46
| |\
| | * refactor blobstore (#458)Dan Engelbrecht2025-08-061-41/+46
| | | | | | | | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
| * | Moving put rejections to happen in batch handlingzousar2025-08-051-8/+14
| | |
| * | xmake precommitzousar2025-06-241-3/+3
| | |
| * | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-248-153/+352
| |\|
| | * missing chunks bugfix (#424)Dan Engelbrecht2025-06-092-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
| | * add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-301-2/+3
| | | | | | | | | | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
| | * faster oplog validate (#408)Dan Engelbrecht2025-05-301-2/+3
| | | | | | | | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
| | * unblock cache bucket drop (#406)Dan Engelbrecht2025-05-262-14/+14
| | | | | | | | | | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
| | * keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-121-5/+5
| | | | | | | | | | | | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
| | * enable per bucket config (#388)Dan Engelbrecht2025-05-121-0/+3
| | | | | | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| | * blobstore size limit (#342)Dan Engelbrecht2025-04-041-1/+20
| | | | | | | | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
| | * build store save access times (#341)v5.6.3-pre0Dan Engelbrecht2025-04-032-11/+40
| | | | | | | | | | | | | | | * save payload size in log for buildstore * read/write access times and manifest for buldstore * use retry when removing temporary files
| | * use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-13/+0
| | | | | | | | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
| | * zen build cache service (#318)Dan Engelbrecht2025-03-266-39/+239
| | | | | | | | | | | | | | | | | | | | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
| | * reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-063-19/+29
| | | | | | | | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
| | * moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-54/+0
| | |
| * | Change to PutResult structurezousar2025-06-244-40/+51
| | | | | | | | | | | | Result structure contains status and a string message (may be empty)
| * | Control overwrite enforcement with a config settingzousar2025-03-021-0/+1
| | |
| * | Enforce Overwrite Prevention According To Cache Policyzousar2025-02-263-10/+18
| |/ | | | | | | Overwrite with differing value should be denied if QueryLocal is not present and StoreLocal is present. Overwrite with equal value should succeed regardless of policy flags.
| * move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-162-2/+2
| | | | | | | | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
| * Unity build fixes (#253)Stefan Boberg2024-12-051-1/+5
| | | | | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
| * caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-252-2/+3
| | | | | | | | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
| * oplog prep gc fix (#216)Dan Engelbrecht2024-11-153-26/+62
| | | | | | | | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
| * workspace share security (#192)Dan Engelbrecht2024-10-231-21/+62
| | | | | | | | | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
| * bucket size queries (#203)Dan Engelbrecht2024-10-213-1/+29
| | | | | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
| * remove gc v1 (#121)Dan Engelbrecht2024-10-034-100/+17
| | | | | | | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
| * gc block size target max size (#180)Dan Engelbrecht2024-10-021-2/+11
| | | | | | | | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
| * gc command attachment options (#176)Dan Engelbrecht2024-09-302-7/+14
| | | | | | * zen command - add options to control meta data cache when triggering gc
| * optimize startup time (#175)Dan Engelbrecht2024-09-301-9/+3
| | | | | | | | | | | | * use tsl::robin_set for BlockIndexSet don't calculate full block location when only block index is needed * don't copy visitor function * reserve space for attachments
| * reduce lock time for memcache trim (#171)Dan Engelbrecht2024-09-271-3/+4
| | | | | | | | | | | | | | - Improvement: Faster memcache trimming - Reduce calculations while holding bucket lock for memcache trim analysis to reduce contention - When trimming memcache, evict 25% more than required to reduce frequency of trimming - When trimming memcache, don't repack memcache data vector, defer that to regular garbage collection - When trimming memcache, deallocate memcache buffers when not holding exclusive lock in bucket
| * Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-4/+11
| | | | | | | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
| * Added namespace qualifier (optional) for z$ rpc requests (#166)Stefan Boberg2024-09-231-0/+1
| | | | | | This change adds support for a namespace-qualified RPC endpoint for z$ at `/z$/<namespace>/$rpc` which may be used to validate RPC requests by URL inspection. The old scheme is still supported.
| * gc unused refactor (#165)Dan Engelbrecht2024-09-232-18/+27
| | | | | | | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
| * made fmt formatter format function const (#162)Stefan Boberg2024-09-201-1/+1
| | | | | | this appears to be required as of fmt v11
| * unblock PreCache (#164)Dan Engelbrecht2024-09-201-1/+1
| | | | | | Don't lock disk cache buckets from writing when scanning records for attachment references
| * gc performance improvements (#160)Dan Engelbrecht2024-09-172-29/+22
| | | | | | | | | | | | | | | | | | | | * optimized ValidateCbUInt * optimized iohash comparision * replace unordered set/map with tsl/robin set/map in blockstore * increase max buffer size when writing cache bucket sidecar * only store meta data for files < 4Gb * faster ReadAttachmentsFromMetaData * remove memcpy call in BlockStoreDiskLocation * only write cache bucket state to disk if GC deleted anything
| * clean cache slog files on startup (#143)Dan Engelbrecht2024-09-041-3/+7
| | | | | | | | - Bugfix: If we fail to move a temporary file into place, try to re-open the file so we clean it up - Improvement: Clean up cache bucket log files at startup as we store the matching information in the index snapshot for the bucket
| * move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-0/+1
| | | | | | - Improvement: Move GC logging in callback functions into "gc" context
| * oplog index snapshots (#140)Dan Engelbrecht2024-09-031-2/+2
| | | | | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-05 15:39:34 +0000