aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* optimize blockstore flush (#614)Dan Engelbrecht2025-10-273-35/+66
| | | | | * rework block store block flushing to only happen once at end of block write outside of locks * fix warning at startup if no gc.dlog file exists
* optimize filecas write file (#613)Dan Engelbrecht2025-10-241-16/+10
| | | * try to move file into place before trying speculative remove of target file
* optimize blockstore filesize (#612)Dan Engelbrecht2025-10-241-1/+11
| | | * since we only ever append to a block store file we don't need to actually flush the position
* fix gc disk load graph (#610)Dan Engelbrecht2025-10-241-3/+3
| | | * make sure our gc disk load graph includes the latest measurement value
* gracefully handle broken gc dlog (#606)Dan Engelbrecht2025-10-241-0/+8
| | | * if gc.dlog is corrupt, remove and restart a new log
* refactor CasContainerStrategy::IterateOneBlock to make it more readable (#607)Dan Engelbrecht2025-10-242-91/+102
|
* if we are low on disk space, only run GC if it will remove any data (#603)Dan Engelbrecht2025-10-232-90/+160
| | | | * if we are low on disk space, only run GC if it will remove any data * make sure we don't treat bail of GC due to disk space as success causing 0 wait between GC passes
* remove scope in GC that prevented GC from executing (#600)Dan Engelbrecht2025-10-221-30/+31
|
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-225-5/+5
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* fix builds storage stats (#590)Dan Engelbrecht2025-10-202-5/+10
| | | * restructure builds storage stats to match web-ui expectations
* fix gc state switching (#588)Dan Engelbrecht2025-10-171-40/+38
| | | * fix state issue in GC thread where shutting down gc did not always block gc from running
* add ability to limit concurrency (#565)Stefan Boberg2025-10-104-11/+11
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* fix missing chunk in block after gc (#560)Dan Engelbrecht2025-10-061-1/+3
| | | * make sure we use aligned write pos in blockstore compact when checking target block size
* fixed issue in compactcas.restart test due to std::vector<bool> (#559)Stefan Boberg2025-10-061-2/+5
| | | `std::vector<bool>` is a special container since it bit packs the values rather than just using an array of booleans. This means that updating it on multiple threads simultaneously is dangerous
* speed up tests (#555)Dan Engelbrecht2025-10-063-154/+209
| | | | | | | | | | | | * faster FileSystemTraversal test * faster jobqueue test * faster NamedEvent test * faster cache tests * faster basic http tests * faster blockstore test * faster cache store tests * faster compactcas tests * more responsive zenserver launch * tweak worker pool sizes in tests
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-0312-8/+1051
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-034-2/+145
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-033-1/+554
| | | * move zen vfs implementation to zenstore
* fix missing chunk (#548)Dan Engelbrecht2025-10-031-2/+12
| | | | * fix race condition where BlockStoreFile::m_CachedFileSize may be reset between check and get in FileSize()
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-035-5/+5
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-031-29/+0
|
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-025-0/+8824
|
* more cbobject validations (#527)Dan Engelbrecht2025-09-291-5/+19
| | | - Improvement: Add additional validations when reading disk cache records to get references in GC
* fix race condition in BlockStoreFile::Flush (#525)Dan Engelbrecht2025-09-291-2/+2
| | | Bugfix: Flush of blockstore file could sometimes cause an error due to a race condition
* GetCacheChunk value request respects RawHash (#518)Zousar Shaker2025-09-291-1/+3
| | | When requesting a value via the GetCacheChunks rpc method, if the request specified a raw hash, only return a hit if the raw hash matches what was in the cache.
* some bug fixes (#522)Stefan Boberg2025-09-291-1/+2
| | | | | | * fix for invalid regex in HttpBuildStoreService - triggers with most recent MSVC version * in GcScheduler don't wait for exit signal if exit has already been requested. this caused extended waits for shutdown in some automated tests on very fast machines, possibly also due to some behaviour change in condition_variable * speculative fix/workaround for issue with TLS teardown on secondary thread while main was tearing down trace
* more iterate chunk logging (#516)Dan Engelbrecht2025-09-262-12/+56
| | | | | * add log warnings when we can't read payloads in cas when we thing we should have them * fix misleading option help
* Improvement to Incomplete Result Iterationzousar2025-09-251-2/+1
| | | | From review feedback
* Report Incomplete Records To Clientzousar2025-09-241-5/+39
| | | | When requesting partial records, report back when a record is incomplete via an "Incomplete" array of bools that is a sibling to the "Result" array for batch/rpc operations, or via the HttpResponseCode::PartialContent status code for individual record requests.
* Adjust the responses from PUT commandszousar2025-09-233-14/+9
| | | | | - Ensure that text responses are in a field named "Message" - Change the record response to be named "Record" instead of "Object"
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-223-14/+15
|\
| * more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-222-12/+13
| | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
| * issue error on second retry, not first attempt (#503)Dan Engelbrecht2025-09-221-2/+2
| |
* | Change batch put responses for client reportingzousar2025-09-195-36/+79
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* revise exception vs error (#495)Dan Engelbrecht2025-09-151-16/+16
| | | | | - Change BadAlloc exceptions in GC to warnings - Add explict ASSERT exception catch in http plugin request processing - Make exceptions handled in http request processing to warnings
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-109-360/+416
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-042-19/+27
| | | * add validation of compact binary payloads before reading them
* oplog memory usage reduction (#482)Dan Engelbrecht2025-09-041-0/+2
| | | | | | - Improvement: For projectstore oplog GET operation, only read basic information and release it if the oplog is not already open to reduce memory usage when listing oplogs in web UI - Improvement: Reduce memory usage for oplog Op address lookup Refactored Oplog::EState -> Oplog ::EMode and make sure we open data files in read-only mode when EMode::kBasicReadOnly is used.
* revert multi-cid store (#475)Dan Engelbrecht2025-08-263-44/+20
|
* clean up trace options parsing (#473)Dan Engelbrecht2025-08-221-0/+2
| | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-2012-136/+165
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-1/+1
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-122-105/+91
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-083-550/+323
|\
| * add the correct set of references hashes in batched inline mode (#459)Dan Engelbrecht2025-08-061-3/+3
| |
| * refactor blobstore (#458)Dan Engelbrecht2025-08-062-548/+321
| | | | | | | | | | | | - Improvement: Refactored build store cache to use existing CidStore implementation instead of implementation specific blob storage - **CAUTION** This will clear any existing cache when updating as the manifest version and storage strategy has changed - Bugfix: BuildStorage cache return "true" for metadata existance for all blobs that had payloads regardless of actual existance for metadata
* | precommitzousar2025-08-071-9/+4
| |
* | Avoid committing chunks for batch rejected putszousar2025-08-071-42/+35
| | | | | | | | Previously rejected puts would put the chunks, but not write them to the index, which was wrong.
* | Moving put rejections to happen in batch handlingzousar2025-08-052-41/+101
| |
* | Merge branch 'main' into zs/put-overwrite-policyzousar2025-08-051-11/+40
|\|
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-22 06:59:37 +0000