aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* add append-only buffering of BlockStoreFile (#652)Dan Engelbrecht2025-11-171-9/+124
| | | | * add append-only buffering of BlockStoreFile replaces use of BasicFileWriter in Compact which bypassed cached position in BlockStore
* switch to xmake for package management (#611)Stefan Boberg2025-11-072-3/+3
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* fix clean directory and make them use effective threading where appropriate ↵v5.7.8-pre5v5.7.8-pre3v5.7.8-pre2Dan Engelbrecht2025-11-031-1/+1
| | | | | | (#625) fix retry logic so it does not immediately sleep if file does not exist make sure we don't try to delete target folder files if we have already wiped it
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-015-5/+13
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* fix use-after-free in TEST_CASE("compactcas.threadedinsert") (#620)v5.7.8-pre1Stefan Boberg2025-10-301-6/+8
|
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-281-8/+8
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* optimize blockstore flush (#614)Dan Engelbrecht2025-10-273-35/+66
| | | | | * rework block store block flushing to only happen once at end of block write outside of locks * fix warning at startup if no gc.dlog file exists
* optimize filecas write file (#613)Dan Engelbrecht2025-10-241-16/+10
| | | * try to move file into place before trying speculative remove of target file
* optimize blockstore filesize (#612)Dan Engelbrecht2025-10-241-1/+11
| | | * since we only ever append to a block store file we don't need to actually flush the position
* fix gc disk load graph (#610)Dan Engelbrecht2025-10-241-3/+3
| | | * make sure our gc disk load graph includes the latest measurement value
* gracefully handle broken gc dlog (#606)Dan Engelbrecht2025-10-241-0/+8
| | | * if gc.dlog is corrupt, remove and restart a new log
* refactor CasContainerStrategy::IterateOneBlock to make it more readable (#607)Dan Engelbrecht2025-10-242-91/+102
|
* if we are low on disk space, only run GC if it will remove any data (#603)Dan Engelbrecht2025-10-232-90/+160
| | | | * if we are low on disk space, only run GC if it will remove any data * make sure we don't treat bail of GC due to disk space as success causing 0 wait between GC passes
* remove scope in GC that prevented GC from executing (#600)Dan Engelbrecht2025-10-221-30/+31
|
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-225-5/+5
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* fix builds storage stats (#590)Dan Engelbrecht2025-10-202-5/+10
| | | * restructure builds storage stats to match web-ui expectations
* fix gc state switching (#588)Dan Engelbrecht2025-10-171-40/+38
| | | * fix state issue in GC thread where shutting down gc did not always block gc from running
* add ability to limit concurrency (#565)Stefan Boberg2025-10-104-11/+11
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* fix missing chunk in block after gc (#560)Dan Engelbrecht2025-10-061-1/+3
| | | * make sure we use aligned write pos in blockstore compact when checking target block size
* fixed issue in compactcas.restart test due to std::vector<bool> (#559)Stefan Boberg2025-10-061-2/+5
| | | `std::vector<bool>` is a special container since it bit packs the values rather than just using an array of booleans. This means that updating it on multiple threads simultaneously is dangerous
* speed up tests (#555)Dan Engelbrecht2025-10-063-154/+209
| | | | | | | | | | | | * faster FileSystemTraversal test * faster jobqueue test * faster NamedEvent test * faster cache tests * faster basic http tests * faster blockstore test * faster cache store tests * faster compactcas tests * more responsive zenserver launch * tweak worker pool sizes in tests
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-0312-8/+1051
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-034-2/+145
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-033-1/+554
| | | * move zen vfs implementation to zenstore
* fix missing chunk (#548)Dan Engelbrecht2025-10-031-2/+12
| | | | * fix race condition where BlockStoreFile::m_CachedFileSize may be reset between check and get in FileSize()
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-035-5/+5
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* move chunking code to zenremotestore lib (#545)Dan Engelbrecht2025-10-031-29/+0
|
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-025-0/+8824
|
* more cbobject validations (#527)Dan Engelbrecht2025-09-291-5/+19
| | | - Improvement: Add additional validations when reading disk cache records to get references in GC
* fix race condition in BlockStoreFile::Flush (#525)Dan Engelbrecht2025-09-291-2/+2
| | | Bugfix: Flush of blockstore file could sometimes cause an error due to a race condition
* GetCacheChunk value request respects RawHash (#518)Zousar Shaker2025-09-291-1/+3
| | | When requesting a value via the GetCacheChunks rpc method, if the request specified a raw hash, only return a hit if the raw hash matches what was in the cache.
* some bug fixes (#522)Stefan Boberg2025-09-291-1/+2
| | | | | | * fix for invalid regex in HttpBuildStoreService - triggers with most recent MSVC version * in GcScheduler don't wait for exit signal if exit has already been requested. this caused extended waits for shutdown in some automated tests on very fast machines, possibly also due to some behaviour change in condition_variable * speculative fix/workaround for issue with TLS teardown on secondary thread while main was tearing down trace
* more iterate chunk logging (#516)Dan Engelbrecht2025-09-262-12/+56
| | | | | * add log warnings when we can't read payloads in cas when we thing we should have them * fix misleading option help
* Improvement to Incomplete Result Iterationzousar2025-09-251-2/+1
| | | | From review feedback
* Report Incomplete Records To Clientzousar2025-09-241-5/+39
| | | | When requesting partial records, report back when a record is incomplete via an "Incomplete" array of bools that is a sibling to the "Result" array for batch/rpc operations, or via the HttpResponseCode::PartialContent status code for individual record requests.
* Adjust the responses from PUT commandszousar2025-09-233-14/+9
| | | | | - Ensure that text responses are in a field named "Message" - Change the record response to be named "Record" instead of "Object"
* Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-223-14/+15
|\
| * more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-222-12/+13
| | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
| * issue error on second retry, not first attempt (#503)Dan Engelbrecht2025-09-221-2/+2
| |
* | Change batch put responses for client reportingzousar2025-09-195-36/+79
|/ | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* revise exception vs error (#495)Dan Engelbrecht2025-09-151-16/+16
| | | | | - Change BadAlloc exceptions in GC to warnings - Add explict ASSERT exception catch in http plugin request processing - Make exceptions handled in http request processing to warnings
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-109-360/+416
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-042-19/+27
| | | * add validation of compact binary payloads before reading them
* oplog memory usage reduction (#482)Dan Engelbrecht2025-09-041-0/+2
| | | | | | - Improvement: For projectstore oplog GET operation, only read basic information and release it if the oplog is not already open to reduce memory usage when listing oplogs in web UI - Improvement: Reduce memory usage for oplog Op address lookup Refactored Oplog::EState -> Oplog ::EMode and make sure we open data files in read-only mode when EMode::kBasicReadOnly is used.
* revert multi-cid store (#475)Dan Engelbrecht2025-08-263-44/+20
|
* clean up trace options parsing (#473)Dan Engelbrecht2025-08-221-0/+2
| | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-2012-136/+165
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-1/+1
| | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* reduce lock contention when checking for disk cache put reject (#465)Dan Engelbrecht2025-08-122-105/+91
| | | | keep rawsize and rawhash if available when using batch for inline puts keep rawsize and rawhash of input value if we have calculated it for validation already
* Merge branch 'main' into zs/put-overwrite-policyZousar Shaker2025-08-083-550/+323
|\
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-23 02:49:46 +0000