aboutsummaryrefslogtreecommitdiff
path: root/src/zenremotestore
Commit message (Collapse)AuthorAgeFilesLines
* Fix long path handling for project storezousar2 days1-0/+1
|
* revise oplog block arrangement (#842)Dan Engelbrecht4 days5-2385/+4769
| | | | | - Improvement: Fixed issue where oplog upload could create blocks larger than the max limit (64Mb) Refactored remoteprojectstore.cpp to use ParallelWork and exceptions for error handling.
* added streaming download of payloads http client Post (#824)Dan Engelbrecht9 days1-1/+2
| | | | | | * added streaming download of payloads in cpr client ::Post * curlclient Post streaming download * case sensitivity fixes for http headers * move over missing functionality from crpclient to httpclient
* improved oplog import progress reporting (#825)Dan Engelbrecht9 days1-121/+196
|
* block scavenge of other downloads that uses an older state file (#822)Dan Engelbrecht9 days3-3/+22
|
* HttpClient using libcurl, Unix Sockets for HTTP. HTTPS support (#770)Stefan Boberg10 days4-13/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main goal of this change is to eliminate the cpr back-end altogether and replace it with the curl implementation. I would expect to drop cpr as soon as we feel happy with the libcurl back-end. That would leave us with a direct dependency on libcurl only, and cpr can be eliminated as a dependency. ### HttpClient Backend Overhaul - Implemented a new **libcurl-based HttpClient** backend (`httpclientcurl.cpp`, ~2000 lines) as an alternative to the cpr-based one - Made HttpClient backend **configurable at runtime** via constructor arguments and `-httpclient=...` CLI option (for zen, zenserver, and tests) - Extended HttpClient test suite to cover multipart/content-range scenarios ### Unix Domain Socket Support - Added Unix domain socket support to **httpasio** (server side) - Added Unix domain socket support to **HttpClient** - Added Unix domain socket support to **HttpWsClient** (WebSocket client) - Templatized `HttpServerConnectionT<SocketType>` and `WsAsioConnectionT<SocketType>` to handle TCP, Unix, and SSL sockets uniformly via `if constexpr` dispatch ### HTTPS Support - Added **preliminary HTTPS support to httpasio** (for Mac/Linux via OpenSSL) - Added **basic HTTPS support for http.sys** (Windows) - Implemented HTTPS test for httpasio - Split `InitializeServer` into smaller sub-functions for http.sys ### Other Notable Changes - Improved **zenhttp-test stability** with dynamic port allocation - Enhanced port retry logic in http.sys (handles ERROR_ACCESS_DENIED) - Fatal signal/exception handlers for backtrace generation in tests - Added `zen bench http` subcommand to exercise network + HTTP client/server communication stack
* updated chunk–block analyser (#818)Dan Engelbrecht10 days4-491/+1225
| | | | | * create oplogs as they are imported * Improved logic for partial block analisys * unit tests for ChunkBlockAnalyser
* Dashboard overhaul, compute integration (#814)Stefan Boberg11 days2-3/+3
| | | | | | | | | | - **Frontend dashboard overhaul**: Unified compute/main dashboards into a single shared UI. Added new pages for cache, projects, metrics, sessions, info (build/runtime config, system stats). Added live-update via WebSockets with pause control, sortable detail tables, themed styling. Refactored compute/hub/orchestrator pages into modular JS. - **HTTP server fixes and stats**: Fixed http.sys local-only fallback when default port is in use, implemented root endpoint redirect for http.sys, fixed Linux/Mac port reuse. Added /stats endpoint exposing HTTP server metrics (bytes transferred, request rates). Added WebSocket stats tracking. - **OTEL/diagnostics hardening**: Improved OTLP HTTP exporter with better error handling and resilience. Extended diagnostics services configuration. - **Session management**: Added new sessions service with HTTP endpoints for registering, updating, querying, and removing sessions. Includes session log file support. This is still WIP. - **CLI subcommand support**: Added support for commands with subcommands in the zen CLI tool, with improved command dispatch. - **Misc**: Exposed CPU usage/hostname to frontend, fixed JS compact binary float32/float64 decoding, limited projects displayed on front page to 25 sorted by last access, added vscode:// link support. Also contains some fixes from TSAN analysis.
* add fallback for zencache multirange (#816)Dan Engelbrecht11 days17-1285/+2133
| | | | | | | | | | | * clean up BuildStorageResolveResult to allow capabilities * add check for multirange request capability * add MaxRangeCountPerRequest capabilities * project export tests * add InMemoryBuildStorageCache * progress and logging improvements * fix ElapsedSeconds calculations in fileremoteprojectstore.cpp * oplogs/builds test script
* Eliminate spdlog dependency (#773)Stefan Boberg11 days3-23/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes the vendored spdlog library (~12,000 lines) and replaces it with a purpose-built logging system in zencore (~1,800 lines). The new implementation provides the same functionality with fewer abstractions, no shared_ptr overhead, and full control over the logging pipeline. ### What changed **New logging core in zencore/logging/:** - LogMessage, Formatter, Sink, Logger, Registry - core abstractions matching spdlog's model but simplified - AnsiColorStdoutSink - ANSI color console output (replaces spdlog stdout_color_sink) - MsvcSink - OutputDebugString on Windows (replaces spdlog msvc_sink) - AsyncSink - async logging via BlockingQueue worker thread (replaces spdlog async_logger) - NullSink, MessageOnlyFormatter - utility types - Thread-safe timestamp caching in formatters using RwLock **Moved to zenutil/logging/:** - FullFormatter - full log formatting with timestamp, logger name, level, source location, multiline alignment - JsonFormatter - structured JSON log output - RotatingFileSink - rotating file sink with atomic size tracking **API changes:** - Log levels are now an enum (LogLevel) instead of int, eliminating the zen::logging::level namespace - LoggerRef no longer wraps shared_ptr - it holds a raw pointer with the registry owning lifetime - Logger error handler is wired through Registry and propagated to all loggers on registration - Logger::Log() now populates ThreadId on every message **Cleanup:** - Deleted thirdparty/spdlog/ entirely (110+ files) - Deleted full_test_formatter (was ~80% duplicate of FullFormatter) - Renamed snake_case classes to PascalCase (full_formatter -> FullFormatter, json_formatter -> JsonFormatter, sentry_sink -> SentrySink) - Removed spdlog from xmake dependency graph ### Build / test impact - zencore no longer depends on spdlog - zenutil and zenvfs xmake.lua updated to drop spdlog dep - zentelemetry xmake.lua updated to drop spdlog dep - All existing tests pass, no test changes required beyond formatter class renames
* zenstore bug-fixes from static analysis pass (#815)Stefan Boberg14 days5-18/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Bug fixes across zenstore, zenremotestore, and related subsystems, primarily surfaced by static analysis.** ## Cache subsystem (cachedisklayer.cpp) - Fixed tombstone scoping bug: tombstone flag and missing entry were recorded outside the block where data was removed, causing non-missing entries to be incorrectly tombstoned - Fixed use-after-overwrite: `RemoveMemCachedData`/`RemoveMetaData` were called after `Payload` was overwritten on cache put, leaking stale data - Fixed incorrect retry sleep formula (`100 - (3 - RetriesLeft) * 100` always produced the same or negative value; corrected to `(3 - RetriesLeft) * 100`) - Fixed broken `break` missing from sidecar file read loop, causing reads past valid data - Fixed missing format argument in three `ZEN_WARN`/`ZEN_ERROR` log calls (format string had `{}` placeholders with no corresponding argument, or vice versa) - Fixed elapsed timer being accumulated inside the wrong scope in `HandleRpcGetCacheRecords` - Fixed test asserting against unserialized `RecordPolicy` instead of the deserialized `Loaded` copy - Initialized `AbortFlag`/`PauseFlag` atomics at declaration (UB if read before first write) ## Build store (buildstore.cpp / buildstore.h) - Fixed wrong variable used in warning log: used loop index `ResultIndex` instead of `Index`/`MetaLocationResultIndexes[Index]`, logging wrong hash values - Fixed `sizeof(AccessTimesHeader)` used instead of `sizeof(AccessTimeRecord)` when advancing write offset, corrupting the access times file if the sizes differ - Initialized `m_LastAccessTimeUpdateCount` atomic member (was uninitialized) - Changed map iteration loops to use `const auto&` to avoid unnecessary copies ## Project store (projectstore.cpp / projectstore.h) - Fixed wrong iterator dereferenced in `IterateChunks`: used `ChunkIt->second` (from a different map lookup) instead of `MetaIt->second` - Fixed wrong assert variable: `Sizes[Index]` should be `RawSizes[Index]` - Fixed `MakeTombstone`/`IsTombstone` inconsistency: `MakeTombstone` was zeroing `OpLsn` but `IsTombstone` checks `OpLsn.Number != 0`; tombstone creation now preserves `OpLsn` - Fixed uninitialized `InvalidEntries` counter - Fixed format string mismatch in warning log - Initialized `AbortFlag`/`PauseFlag` atomics; changed map iteration to `const auto&` ## Workspaces (workspaces.cpp) - Fixed missing alias registration when a workspace share is updated: alias was deleted but never re-inserted - Fixed integer overflow in range clamping: `(RequestedOffset + RequestedSize) > Size` could wrap; corrected to `RequestedSize > Size - RequestedOffset` - Changed map iteration loops to `const auto&` ## CAS subsystem (cas.cpp, caslog.cpp, compactcas.cpp, filecas.cpp) - Fixed `IterateChunks` passing original `Payload` buffer instead of the modified `Chunk` buffer (content type was set on the copy but the original was sent to the callback) - Fixed invalid `std::future::get()` call on default-constructed futures - Fixed sign-comparison in `CasLogFile::Replay` loop (`int i` vs `size_t`) - Changed `CasLogFile::IsValid` and `Open` to take `const std::filesystem::path&` instead of by value - Fixed format string in `~CasContainerStrategy` error log ## Remote store (zenremotestore) - Fixed `FolderContent::operator==` always returning true: loop variable `PathCount` was initialized to 0 instead of `Paths.size()` - Fixed `GetChunkIndexForRawHash` looking up from wrong map (`RawHashToSequenceIndex` instead of `ChunkHashToChunkIndex`) - Fixed double-counted `UniqueSequencesFound` stat (incremented in both branches of an if/else) - Fixed `RawSize` sentinel value truncation: `(uint32_t)-1` assigned to a `uint64_t` field; corrected to `(uint64_t)-1` - Initialized uninitialized atomic and struct members across `buildstorageoperations.h`, `chunkblock.h`, and `remoteprojectstore.h`
* added TEST_SUITE_BEGIN/END around some TEST_CASEs which didn't have them (#809)Stefan Boberg2026-03-051-0/+4
| | | | | | | * added TEST_SUITE_BEGIN/END around some TEST_CASEs which didn't have them * fixed some stats issues * ScopedSpan should Initialize * annotated classes in stats.h with some documentation comments
* compute orchestration (#763)Stefan Boberg2026-03-042-5/+5
| | | | | | | | | | - Added local process runners for Linux/Wine, Mac with some sandboxing support - Horde & Nomad provisioning for development and testing - Client session queues with lifecycle management (active/draining/cancelled), automatic retry with configurable limits, and manual reschedule API - Improved web UI for orchestrator, compute, and hub dashboards with WebSocket push updates - Some security hardening - Improved scalability and `zen exec` command Still experimental - compute support is disabled by default
* unity build fixes (#802)Stefan Boberg2026-03-042-7/+7
| | | | | Various fixes to make cpp files build in unity build mode as an aside using Unity build doesn't really seem to work on Linux, unsure why but it leads to link-time issues
* use multi range requests (#800)Dan Engelbrecht2026-03-0319-686/+1535
| | | | | | | - Improvement: `zen builds download` now uses multi-range requests for blocks to reduce download size - Improvement: `zen oplog-import` now uses partial block with multi-range requests for blocks to reduce download size - Improvement: Improved feedback in log/console during `zen oplog-import` - Improvement: `--allow-partial-block-requests` now defaults to `true` for `zen builds download` and `zen oplog-import` (was `mixed`) - Improvement: Improved range merging analysis when downloading partial blocks
* Add test suites (#799)Stefan Boberg2026-03-028-0/+32
| | | | | | | | | | | | | Makes all test cases part of a test suite. Test suites are named after the module and the name of the file containing the implementation of the test. * This allows for better and more predictable filtering of which test cases to run which should also be able to reduce the time CI spends in tests since it can filter on the tests for that particular module. Also improves `xmake test` behaviour: * instead of an explicit list of projects just enumerate the test projects which are available based on build system state * also introduces logic to avoid running `xmake config` unnecessarily which would invalidate the existing build and do lots of unnecessary work since dependencies were invalidated by the updated config * also invokes build only for the chosen test targets As a bonus, also adds `xmake sln --open` which allows opening IDE after generation of solution/xmake project is done.
* add support in http client to accept multi-range responses (#788)Dan Engelbrecht2026-02-271-74/+78
| | | * add support in http client to accept multi-range responses
* use partial blocks for oplog import (#780)Dan Engelbrecht2026-02-2421-345/+1122
| | | | | Feature: Add --allow-partial-block-requests to zen oplog-import Improvement: zen oplog-import now uses partial block requests to reduce download size Improvement: Use latency to Cloud Storage host and Zen Cache host when calculating partial block requests
* move partial chunk block anailsys to chunkblock.h/cpp (#767)Dan Engelbrecht2026-02-204-679/+887
|
* fix builds download indexing timer (#769)Dan Engelbrecht2026-02-201-7/+0
| | | * fix build download indexing timer log
* convert ZEN_ASSERTs to exception to handle corrupt data gracefully (#760)Dan Engelbrecht2026-02-182-13/+58
| | | * convert ZEN_ASSERTs to exception to handle corrupt data gracefully
* fix target folder cleanup in builds download (#746)Dan Engelbrecht2026-02-061-12/+9
|
* only disable backlog scheduling when downloaded payload is not on disk (#741)Dan Engelbrecht2026-02-022-7/+14
|
* Avoid conversion from JSon to compact binary when querying for builds to ↵Dan Engelbrecht2026-01-285-13/+11
| | | | avoid integer vs float conversion issues (#735)
* allow download specification for zen builds download (#734)Dan Engelbrecht2026-01-275-108/+272
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds download` now supports `--download-spec-path` to determine what content to download from a build - The unstructured format expects one line per file relative to the root with '/' as a path delimiter - The structured format uses JSon format and the `--download-spec-path` must have extension `.json` to enable structured input { "parts": { "default" : { "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* avoid big ioworker backlog (#733)Dan Engelbrecht2026-01-262-37/+42
| | | | | | * add ability to override scheduling mode in ParallelWork * Don't increase buffering size when copying from local cache with --boost-worker-memory enabled * Rework scheduling writes of downloaded data to reduce memory usage
* builds scanning cache (#727)v5.7.19-pre0Dan Engelbrecht2026-01-238-84/+892
| | | | - Feature: Added `--chunking-cache-path` option to `zen builds upload` and `zen builds diff` - Path to cache for chunking information of scanned files. Default is empty resulting in no caching
* make sure new blocks generated by a part is accessible to following parts (#732)Dan Engelbrecht2026-01-232-81/+92
|
* builds multipart upload (#722)Dan Engelbrecht2026-01-206-800/+1960
| | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds upload` now support structure manifest input for `--manifest-path` when the path has a `.json` extension - The structured manifest supports splitting a build into multiple parts { "parts": { "default" : { "partId": "f939f3939939fff3f3202", # optional - used to control the id of each part "files": [ "foo/bar", "baz.exe" ] }, "symbols": { "files": [ "baz.pdb" ] } } }
* fix init of buildpart id in zen oplog-download (#711)Dan Engelbrecht2026-01-151-1/+1
|
* structured output for builds ls (#709)Dan Engelbrecht2026-01-141-3/+12
| | | | * make ResolveBuildStore respect Verbose flag * add structured output to zen builds ls command
* added options to configure exclude folders and extensions to zen build ↵Dan Engelbrecht2026-01-134-150/+222
| | | | | | | | commands (#706) * added `--exclude-folders` to `zen upload`, `zen download` and `zen diff` added `--exclude-extensions` to `zen upload` and `zen diff` excluded folder names are now matched by folder name in subfolders in addition to root level folders * allow multiple token separators
* fixed block generation after bug introduced in PR #704 (#707)Dan Engelbrecht2026-01-131-12/+15
|
* various optimizations (#704)Dan Engelbrecht2026-01-092-22/+126
| | | | | | | | | - Improvement: Validate chunk hashes when dechunking files in oplog import - Improvement: Use stream decompression when dechunking files - Improvement: When assembling blocks for oplog export, make sure we keep under/at block size limit - Improvement: Make cancelling of oplog import more responsive - Improvement: Use decompress to composite to avoid allocating a new memory buffer for uncompressed chunks during oplog import - Improvement: Reduce memory buffer size and allocate it on demand when writing multiple chunks to block store - Improvement: Reduce lock contention when fetching/checking existence of chunks in block store
* read build state files into memory instead of relying on memory mapping (#702)Dan Engelbrecht2026-01-081-2/+7
|
* optimize scavenge part 2 (#699)Dan Engelbrecht2026-01-082-148/+109
| | | | * optimize scavenge discovery by iterate over chunks only instead of iterating through all chunks in the scavenged files * refactor scavenge lookup
* Merge branch 'main' into zs/limit-overwrite-defaultZousar Shaker2025-12-194-169/+241
|\
| * optimize scavenge (#697)Dan Engelbrecht2025-12-194-169/+241
| | | | | | | | * optimize FindScavengeContent * optimize GetValidFolderContent
* | Ensure upstream put propagation includes overwritezousar2025-12-193-6/+20
|/ | | | When changing the default limit-overwrite behavior, a unit test surfaced a bug where an put of data with overwrite cache policy would not get propagated via zen's built-in upstream mechanism with a matching overwrite cache policy to the upstream. This change ensures that it does and leaves the unit test configured to exercise this scenario.
* fix scavenge source path match (again) (#694)Dan Engelbrecht2025-12-171-11/+12
| | | | * fix check for scavenged path matching target path
* add boost-worker oplog import export options (#693)Dan Engelbrecht2025-12-165-16/+154
| | | | | | | | | | - Feature: `zen oplog-export`, `zen oplog-import` and `zen oplog-download` now has options to boost workers - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive - Improvement: Refactored boost options for `zen builds` operations `upload`, `download`, `diff`, `prime-cache`, `fetch-blob` and `validate-part` - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive
* remove found chunks as they are found in blocks (#691)Dan Engelbrecht2025-12-151-7/+18
|
* oplog download size (#690)Dan Engelbrecht2025-12-159-295/+627
| | | | - Bugfix: Upload of oplogs could reference multiple blocks for the same chunk causing redundant downloads of blocks - Improvement: Use the improved block reuse selection function from zen builds upload in zen oplog-export to reduce oplog download size
* show download source data (#689)Dan Engelbrecht2025-12-1211-2/+163
| | | * show source stats for jupiter/cache
* remove direct console output from code that is used from service mode (#688)Dan Engelbrecht2025-12-116-28/+145
|
* fix buffer memory in builds cache (#682)Dan Engelbrecht2025-12-101-18/+68
| | | | * add --zen-cache-upload option to zen oplog-import command * fix buildstoragecache to not hold on to possibly materialized buffers
* safer lambda captures (#633)Dan Engelbrecht2025-12-081-8/+10
|
* exists must be called before equivalent (#678)Dan Engelbrecht2025-12-051-1/+1
|
* add checks to protect against access violation due to failed disk read (#675)Dan Engelbrecht2025-12-041-0/+5
| | | * add checkes to protect against access violation due to failed disk read
* use scope guards to secure that BasicFile::Detach is called even on ↵Dan Engelbrecht2025-12-011-1/+1
| | | | exception (#672)
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-20 08:13:46 +0000