aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'main' into sb/reduce-allocssb/reduce-allocsStefan Boberg42 hours15-107/+528
|\
| * hub deprovision all (#938)Dan Engelbrecht42 hours4-5/+110
| | | | | | * implement "deprovision all" for hub
| * dashboard search (#936)Dan Engelbrecht44 hours7-13/+187
| | | | | | | | | | - Improvement: Dashboard paginated lists now include a search input that jumps to the page containing the first match and highlights the row - Improvement: Dashboard paginated lists show a loading indicator while fetching data - Improvement: Hub dashboard navigates to and highlights newly provisioned instances
| * improve messaging when zen builds download target disk does not have enought ↵Dan Engelbrecht44 hours2-1/+5
| | | | | | | | space (#935)
| * update rpmalloc and tweak for commit/decommit churn (#934)Dan Engelbrecht44 hours5-88/+226
| | | | | | | | - Improvement: Updated rpmalloc to develop branch commit feb43aee0d4d (2025-10-26), which fixes `VirtualAlloc(MEM_COMMIT)` failures being silently ignored under memory pressure - Improvement: Increased rpmalloc page decommit thresholds to reduce commit/decommit churn under high allocation turnover
* | Reduce short-lived heap allocations in zenhttpStefan Boberg2 days4-6/+10
| | | | | | | | | | - Replace std::vector<std::string_view> with eastl::fixed_vector<std::string_view, 4> for route capture segments, eliminating a heap allocation on every routed request - Add transparent comparator to status provider map, eliminating std::string temporary on every /status/ lookup
* | Reduce short-lived heap allocations in zenserverStefan Boberg2 days4-14/+19
| | | | | | | | | | | | | | - Replace fmt::format float temporaries in CbXmlWriter with fmt::format_to via StringBuilderAppender - Replace std::vector<std::string> URL tokenizer with eastl::fixed_vector<std::string_view, 5> in cache details handler - Replace std::strtoull/strtoul(std::string(...).c_str()) with zen::ParseInt in session log handler - Replace StrCaseCompare(std::string(sv).c_str(), ...) with string_view overload in workspace handlers
* | Merge branch 'main' into sb/reduce-allocsStefan Boberg2 days11-137/+515
|\|
| * HTTP range responses (RFC 7233) - httpobjectstore (#928)Dan Engelbrecht2 days11-137/+515
| | | | | | | | | | | | | | | | | | - Improvement: HTTP range responses (RFC 7233) are now fully compliant across the object store and build store - 206 Partial Content responses now include a `Content-Range` header; previously absent for single-range requests, which broke `HttpClient::GetRanges()` - 416 Range Not Satisfiable responses now include `Content-Range: bytes */N` as required by RFC 7233 - Out-of-bounds range requests return 416 Range Not Satisfiable (was 400 Bad Request) - Single-byte ranges (`bytes=N-N`) are now correctly accepted (were previously rejected) - Range byte positions widened from 32-bit to 64-bit; RFC 7233 imposes no size limit on byte range values - Build store binary GET requests with a Range header now return 206 Partial Content with `Content-Range` (previously returned 200 OK without it)
* | Reduce short-lived heap allocations in zen CLI progress barStefan Boberg2 days3-37/+100
| | | | | | | | | | | | | | - Add AppendFill(char, size_t) to StringBuilderBase for efficient repeated-character fills - Replace all fmt::format temporaries in ProgressBar::UpdateState with ExtendableStringBuilder - Replace std::string Task copy+pad with string_view and deferred AppendFill padding - Eliminate 4-7 heap allocations per progress tick across Plain, Pretty, and Log modes
* | Reduce short-lived heap allocations in zenutilStefan Boberg3 days4-56/+155
| | | | | | | | | | | | | | | | | | | | | | - Replace Sha256ToHex fmt::format_to per-byte loop with hex lookup table and ExtendableStringBuilder - Replace AwsUriEncode per-char fmt::format for percent-encoding with hex lookup table - Replace transient fmt::format calls in SignRequestV4 and GeneratePresignedUrl with ExtendableStringBuilder appends - Return string_view from GetDateStamp instead of allocating an 8-byte std::string - Replace fmt::format for XML tag construction in ExtractXmlValue with ExtendableStringBuilder - Replace fmt::format in KeyToPath/BucketRootPath with ExtendableStringBuilder - Replace ToLower heap allocations in case-insensitive MatchWildcard with inline tolower loop - Replace fmt::format per log line in logstreamlistener with ExtendableStringBuilder
* | Reduce short-lived heap allocations in zencoreStefan Boberg3 days7-30/+97
| | | | | | | | | | | | | | | | | | | | - Add AppendPaddedInt() to StringBuilderBase for zero-padded integer formatting without fmt::format - Add StringBuilderAppender output iterator adapter for direct use with fmt::format_to - Replace 18 fmt::format temporaries in DateTime::ToString and TimeSpan::ToString with AppendPaddedInt - Replace fmt::format float formatting in JSON/YAML writers with fmt::format_to via StringBuilderAppender - Replace fmt::format for Uuid/ObjectId in YAML writer with direct ToString(StringBuilderBase&) - Replace std::vector with eastl::fixed_vector in CB validation (field names, attachments) - Avoid Path.string() allocation in CreateDirectories by checking native() directly
* | Reduce short-lived heap allocations in zenstoreStefan Boberg3 days3-23/+35
| | | | | | | | | | | | | | - Replace std::vector with eastl::fixed_vector in hot paths: URI parser (per-request), CommitPutBatch, WriteChunks, IterateBlock, IterateChunks - Hoist InvalidEntryReason string outside ReadLog replay lambda to reuse buffer across iterations - Replace std::string concatenation with ExtendableStringBuilder in path helpers - Eliminate 257 heap allocations in Drop() by replacing vector<unique_ptr<ExclusiveLockScope>> with fixed_vector<ExclusiveLockScope, 256>
* | Reduce short-lived heap allocations in zencomputeStefan Boberg3 days8-60/+88
| | | | | | | | | | | | | | | | | | - Replace fmt::format with ExtendableStringBuilder for transient URL and field-name construction in remotehttprunner, httpcomputeservice - Use TransparentStringHash on orchestrator worker/client maps to eliminate std::string temporaries on lookup - Replace std::atoi(std::string(...).c_str()) with zen::ParseInt in httporchestrator - Switch macrunner BuildSandboxProfile from std::string concatenation to ExtendableStringBuilder<1024> - Eliminate triple std::string allocation in workertimeline TimelinePath - Change SubmitActionBatch signature from const std::string& to std::string_view
* | Add TransparentStringHash for heterogeneous unordered_map lookupStefan Boberg3 days1-0/+19
| | | | | | | | | | - Enables std::string_view lookups on std::string-keyed unordered maps without allocating a temporary std::string - Uses is_transparent tag for C++20 heterogeneous lookup support
* | Add reduce-allocs skill for Claude CodeStefan Boberg3 days1-0/+35
|/ | | | | - Skill reviews code for unnecessary short-lived heap allocations - Suggests concrete fixes using stack-friendly patterns (ExtendableStringBuilder, eastl::fixed_vector, TRefCounted, etc.)
* reduce test runtime (#933)Dan Engelbrecht3 days15-1386/+1242
| | | | | | | | * reduce zenserver spawns in tests * fix filesystemutils wrong test suite name * tweak tests for faster runtime * reduce more test runtime * more wall time improvements * fast http and processmanager tests
* Update CHANGELOG.mdStefan Boberg4 days1-0/+4
|
* Fix ZenServerState stale entry detection on PID reuse (k8s) (#932)Stefan Boberg4 days1-0/+31
| | | | | | - Detect stale shared-memory entries whose PID matches the current process but predate our registration (m_OurEntry == nullptr) - Sweep() now reclaims such entries instead of skipping them - Lookup() and LookupByEffectivePort() skip stale same-PID entries - Fixes startup failure on k8s where PID 1 is always reused after an unclean shutdown
* Add async HTTP client (curl_multi + ASIO) (#918)Stefan Boberg4 days6-269/+1776
| | | | | | | | | | | | | | | | | | | | | | | - Adds `AsyncHttpClient` — an asynchronous HTTP client using `curl_multi_socket_action` integrated with ASIO for event-driven I/O. Supports GET, POST, PUT, DELETE, HEAD with both callback-based and `std::future`-based APIs. - Extracts shared curl helpers (callbacks, URL encoding, header construction, error mapping) into `httpclientcurlhelpers.h`, eliminating duplication between the sync and async implementations. ## Design - All curl_multi state is serialized on an `asio::strand`, safe with multi-threaded io_contexts. - Two construction modes: owned io_context (creates internal thread) or external io_context (caller runs the loop). - Socket readiness is detected via `asio::ip::tcp::socket::async_wait` driven by curl's `CURLMOPT_SOCKETFUNCTION`/`CURLMOPT_TIMERFUNCTION` — no polling, sub-millisecond latency. - Completion callbacks are dispatched off the strand onto the io_context so slow callbacks don't starve the curl event loop. Exceptions in callbacks are caught and logged. ## Files | File | Change | |------|--------| | `zenhttp/include/zenhttp/asynchttpclient.h` | New public header | | `zenhttp/clients/asynchttpclient.cpp` | Implementation (~1000 lines) | | `zenhttp/clients/httpclientcurlhelpers.h` | Shared curl helpers extracted from sync client | | `zenhttp/clients/httpclientcurl.cpp` | Removed duplicated helpers, uses shared header | | `zenhttp/asynchttpclient_test.cpp` | 8 test cases: verbs, payloads, callbacks, concurrency, external io_context, connection errors | | `zenhttp/zenhttp.cpp` | Forcelink registration for new tests |
* migrate from http_parser to llhttp (#929)Dan Engelbrecht4 days12-143/+423
|
* 5.8.3v5.8.3Dan Engelbrecht4 days1-1/+1
|
* 5.8.3-pre2v5.8.3-pre2Dan Engelbrecht5 days1-1/+1
|
* fully provisioned hub instances now sets initial check status to "passing" ↵Dan Engelbrecht5 days4-7/+15
| | | | in consul (#930)
* use correct return code for unsupported multirange requests in objectstore ↵Dan Engelbrecht5 days3-2/+65
| | | | (#927)
* don't hard fail if .pending folder is not empty on oplog export (#926)Dan Engelbrecht5 days3-2/+11
|
* fix missing chunk in oplog export (#925)Dan Engelbrecht5 days2-0/+161
| | | * add reused block to oplog during export
* hydration data obliteration (#923)Dan Engelbrecht5 days16-153/+738
| | | | - Feature: Hub obliterate operation deletes all local and backend hydration data for a module - Improvement: Hub dashboard adds obliterate button for individual, bulk, and by-name module deletion
* sort items on dashboard (#924)Dan Engelbrecht5 days3-79/+114
| | | * add pagination and consistent sorting on cache and projects ui pages
* add pagination of cooked projects and caches on dashboard front page (#922)Dan Engelbrecht6 days3-65/+173
|
* incremental dehydrate (#921)Dan Engelbrecht6 days22-1148/+1910
| | | | | | | | | | | | | | | - Feature: Incremental CAS-based hydration/dehydration replacing the previous full-copy approach - Feature: S3 hydration backend with multipart upload/download support - Feature: Configurable thread pools for hub instance provisioning and hydration `--hub-instance-provision-threads` defaults to `max(cpu_count / 4, 2)`. Set to 0 for synchronous operation. `--hub-hydration-threads` defaults to `max(cpu_count / 4, 2)`. Set to 0 for synchronous operation. - Improvement: Hub triggers GC on instance before deprovisioning to compact storage before dehydration - Improvement: GC status now reports pending triggers as running - Improvement: S3 client debug logging gated behind verbose mode to reduce log noise at default verbosity - Improvement: Hub dashboard Resources tile now shows total memory - Improvement: `filesystemutils` moved from `zenremotestore` to `zenutil` for broader reuse - Improvement: Hub uses separate provision and hydration worker pools to avoid deadlocks - Improvement: Hibernate/wake/deprovision on non-existent or already-in-target-state modules are idempotent - Improvement: `ScopedTemporaryDirectory` with empty path now creates a temporary directory instead of asserting
* disable zencompute in bundle stepStefan Boberg10 days1-0/+3
|
* 5.8.3-pre0v5.8.3-pre0Dan Engelbrecht10 days1-1/+1
|
* fix hub consule health endpoint registration (#917)Dan Engelbrecht10 days3-1/+6
| | | | * use correct health endpoint for zenhubserver consul registration * add total disk space on hub resource pane
* 5.8.2v5.8.2Dan Engelbrecht11 days1-1/+1
|
* 5.8.2-pre1v5.8.2-pre1Dan Engelbrecht11 days1-1/+1
|
* s3 and consul fixes (#916)Dan Engelbrecht11 days6-7/+284
| | | | | | | | | | | * fix endpoint for stats/hub in compute/hub.html page * fix api token call failure for imds (using wrong overload for Put) * add "localhost" to healt check url in consul when no address is given * add consul fallback deregister if normal deregister fails * add consul registration unit test
* add provision button to hub ui (#915)Dan Engelbrecht11 days2-0/+134
|
* hub instance dashboard proxy (#914)Dan Engelbrecht11 days24-34/+714
| | | - Feature: Hub dashboard proxy - instance dashboards are accessible through the hub server at `/hub/proxy/{port}/` without requiring direct port access
* 5.8.2-pre0v5.8.2-pre0Dan Engelbrecht12 days1-1/+1
|
* fix fork() issues on linux and MacOS (#910)Dan Engelbrecht12 days7-22/+177
| | | | | - Improvement: Hub child process spawning on macOS now uses `posix_spawn` in line with Apple recommendations - Bugfix: Hub child process spawning on Linux now uses `vfork` instead of `fork`, preventing ENOMEM failures on systems with strict memory overcommit (`vm.overcommit_memory=2`) - Bugfix: Fixed process group management on POSIX; child processes were not placed into the correct process group, breaking group-wide signal delivery
* consul env token refresh (#912)Dan Engelbrecht12 days6-13/+34
| | | - Improvement: Consul token is now re-read from the environment variable on every request, allowing token rotation without restarting the service
* kill stale test processes (zenserver, minio, nomad, consul) before and after ↵Stefan Boberg12 days2-4/+92
| | | | | CI test runs (#909) Adds steps to the validate workflow on all platforms that kill any zenserver, minio, nomad, or consul processes launched from the build output directory. Runs before tests to clear stale processes from previous runs, and after tests (always, even on failure) to clean up.
* Zs/oplog export zero size attachment fix (#911)Zousar Shaker12 days4-2/+157
| | | | | | * Unit test coverage for zero byte file handling in oplogs * Unit test fixes for the zero length file case * Fixes for zero length file attachments * Additional fix for zero length file attachments
* 5.8.1v5.8.1Dan Engelbrecht13 days1-1/+1
|
* 5.8.1-pre1v5.8.1-pre1Dan Engelbrecht13 days1-1/+1
|
* fix potential race with stats counters missing when to Stop filtered values ↵Dan Engelbrecht13 days4-79/+92
| | | | | | | | (#907) * fix potential race with stats counters missing when to Stop filtered values * fix off by one in PutMultipartBuildBlob retry path * use move operation instead of copy operation PutMultipartBlob * fix filter Stop() for upload operations and fix bug with generateblock count filter
* fix jupiterbuildstorage concurrency (#906)Dan Engelbrecht13 days2-15/+35
| | | - Bugfix: Fixed concurrency issue in JupiterBuildStorage when updating stats
* 5.8.1-pre0v5.8.1-pre0Dan Engelbrecht13 days1-1/+1
|
* add lua config options for all zenhubserver command line options (#904)Dan Engelbrecht13 days4-3/+505
| | | | | | | | | | | - Improvement: Hub server now supports Lua config file for all hub-specific options - `hub.upstreamnotification.*` - upstream notification endpoint and instance ID - `hub.consul.*` - service registration endpoint, token, health interval, deregister timeout - `hub.instance.*` - base port, HTTP class, thread count, core limit, config path - `hub.instance.limits.*` - instance count cap, disk and memory usage limits - `hub.hydration.*` - hydration target spec and config path - `hub.watchdog.*` - cycle timing, inactivity timeouts, and activity check timeouts - Improvement: Added `--hub-instance-base-port-number` as an alias for `--hub-base-port-number`, and `--upstream-notification-instance-id` as an alias for `--instance-id` - Improvement: Added hub mode documentation at docs/hub.md
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-13 05:03:17 +0000