aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/include
Commit message (Collapse)AuthorAgeFilesLines
* GC - fix handling of attachment ranges, http access token expiration, lock ↵Stefan Boberg2026-02-201-1/+1
| | | | | | | | file retry logic (#766) * GC - fix handling of attachment ranges * fix trace/log strings * fix HTTP access token expiration time logic * added missing lock retry in zenserver startup
* reduce lock time for project store gc precache and gc validate (#750)Dan Engelbrecht2026-02-111-0/+10
| | | | | * add oplog snapshot function to allow reduction of held oplog locks * release project lock when precaching each oplog
* remove ZENCORE_API completely (#718)Stefan Boberg2026-01-191-2/+2
| | | initially we had ZENCORE_API macros to potentially allow for DLL linkage. It turns out that this is not useful and the macros just contribute noise, so this change removes them completely.
* Ensure upstream put propagation includes overwritezousar2025-12-191-0/+1
| | | | When changing the default limit-overwrite behavior, a unit test surfaced a bug where an put of data with overwrite cache policy would not get propagated via zen's built-in upstream mechanism with a matching overwrite cache policy to the upstream. This change ensures that it does and leaves the unit test configured to exercise this scenario.
* Change default limit-overwrite behavior to truezousar2025-12-171-1/+1
|
* batch op not in destructor (#676)Dan Engelbrecht2025-12-042-15/+26
| | | | | * use fixed vectors for batch requests * refactor cache batch value put/get to not execute code that can throw execeptions in destructor * extend test with multi-bucket requests
* make sure we use exclusive lock in projectstore when flushing/writing ↵Dan Engelbrecht2025-12-011-2/+2
| | | | snapshot (#673)
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-274-35/+39
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* optimize blockstore flush (#614)Dan Engelbrecht2025-10-271-6/+10
| | | | | * rework block store block flushing to only happen once at end of block write outside of locks * fix warning at startup if no gc.dlog file exists
* if we are low on disk space, only run GC if it will remove any data (#603)Dan Engelbrecht2025-10-231-1/+3
| | | | * if we are low on disk space, only run GC if it will remove any data * make sure we don't treat bail of GC due to disk space as success causing 0 wait between GC passes
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-224-4/+4
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* fix builds storage stats (#590)Dan Engelbrecht2025-10-201-5/+7
| | | * restructure builds storage stats to match web-ui expectations
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-036-3/+376
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-031-0/+103
| | | * move zen vfs implementation to zenstore
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-022-0/+629
|
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-221-6/+6
|\
| * more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-221-6/+6
| | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
* | Change batch put responses for client reportingzousar2025-09-192-3/+4
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* revert multi-cid store (#475)Dan Engelbrecht2025-08-262-7/+3
|
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-205-42/+64
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-121-14/+10
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-081-41/+46
|\
| * refactor blobstore (#458)Dan Engelbrecht2025-08-061-41/+46
| | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
* | Moving put rejections to happen in batch handlingzousar2025-08-051-8/+14
| |
* | xmake precommitzousar2025-06-241-3/+3
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-06-248-153/+352
|\|
| * missing chunks bugfix (#424)Dan Engelbrecht2025-06-092-5/+8
| | | | | | | | | | | | | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
| * add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-301-2/+3
| | | | | | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
| * faster oplog validate (#408)Dan Engelbrecht2025-05-301-2/+3
| | | | | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
| * unblock cache bucket drop (#406)Dan Engelbrecht2025-05-262-14/+14
| | | | | | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
| * keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-121-5/+5
| | | | | | | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
| * enable per bucket config (#388)Dan Engelbrecht2025-05-121-0/+3
| | | | | | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
| * blobstore size limit (#342)Dan Engelbrecht2025-04-041-1/+20
| | | | | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
| * build store save access times (#341)v5.6.3-pre0Dan Engelbrecht2025-04-032-11/+40
| | | | | | | | | | * save payload size in log for buildstore * read/write access times and manifest for buldstore * use retry when removing temporary files
| * use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-13/+0
| | | | | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
| * zen build cache service (#318)Dan Engelbrecht2025-03-266-39/+239
| | | | | | | | | | | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
| * reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-063-19/+29
| | | | | | | | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
| * moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-54/+0
| |
* | Change to PutResult structurezousar2025-06-244-40/+51
| | | | | | | | Result structure contains status and a string message (may be empty)
* | Control overwrite enforcement with a config settingzousar2025-03-021-0/+1
| |
* | Enforce Overwrite Prevention According To Cache Policyzousar2025-02-263-10/+18
|/ | | | Overwrite with differing value should be denied if QueryLocal is not present and StoreLocal is present. Overwrite with equal value should succeed regardless of policy flags.
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-162-2/+2
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Unity build fixes (#253)Stefan Boberg2024-12-051-1/+5
| | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-252-2/+3
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-153-26/+62
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* workspace share security (#192)Dan Engelbrecht2024-10-231-21/+62
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* bucket size queries (#203)Dan Engelbrecht2024-10-213-1/+29
| | | - Feature: Added options --bucketsize and --bucketsizes to zen cache-info to get data sizes in cache buckets and attachments
* remove gc v1 (#121)Dan Engelbrecht2024-10-034-100/+17
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc block size target max size (#180)Dan Engelbrecht2024-10-021-2/+11
| | | | | | * If a block is small (less than half max size) we add it to blocks to compact Sort blocks when iterating over them * do compact of block stores even if no new unused are found * do compact phase even if bucket is empty
* gc command attachment options (#176)Dan Engelbrecht2024-09-302-7/+14
| | | * zen command - add options to control meta data cache when triggering gc
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 02:34:05 +0000