| Commit message (Collapse) | Author | Age | Files | Lines |
| |\ |
|
| | |
| |
| | |
* implement "deprovision all" for hub
|
| | |
| |
| |
| |
| | |
- Improvement: Dashboard paginated lists now include a search input that jumps to the page containing the first match and highlights the row
- Improvement: Dashboard paginated lists show a loading indicator while fetching data
- Improvement: Hub dashboard navigates to and highlights newly provisioned instances
|
| | |
| |
| |
| | |
space (#935)
|
| | |
| |
| |
| |
| | |
- Replace std::vector<std::string_view> with eastl::fixed_vector<std::string_view, 4> for route capture segments, eliminating a heap allocation on every routed request
- Add transparent comparator to status provider map, eliminating std::string temporary on every /status/ lookup
|
| | |
| |
| |
| |
| |
| |
| | |
- Replace fmt::format float temporaries in CbXmlWriter with fmt::format_to via StringBuilderAppender
- Replace std::vector<std::string> URL tokenizer with eastl::fixed_vector<std::string_view, 5> in cache details handler
- Replace std::strtoull/strtoul(std::string(...).c_str()) with zen::ParseInt in session log handler
- Replace StrCaseCompare(std::string(sv).c_str(), ...) with string_view overload in workspace handlers
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
- Improvement: HTTP range responses (RFC 7233) are now fully compliant across the object store and build store
- 206 Partial Content responses now include a `Content-Range` header; previously absent for single-range requests, which broke `HttpClient::GetRanges()`
- 416 Range Not Satisfiable responses now include `Content-Range: bytes */N` as required by RFC 7233
- Out-of-bounds range requests return 416 Range Not Satisfiable (was 400 Bad Request)
- Single-byte ranges (`bytes=N-N`) are now correctly accepted (were previously rejected)
- Range byte positions widened from 32-bit to 64-bit; RFC 7233 imposes no size limit on byte range values
- Build store binary GET requests with a Range header now return 206 Partial Content with `Content-Range` (previously returned 200 OK without it)
|
| | |
| |
| |
| |
| |
| |
| | |
- Add AppendFill(char, size_t) to StringBuilderBase for efficient repeated-character fills
- Replace all fmt::format temporaries in ProgressBar::UpdateState with ExtendableStringBuilder
- Replace std::string Task copy+pad with string_view and deferred AppendFill padding
- Eliminate 4-7 heap allocations per progress tick across Plain, Pretty, and Log modes
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Replace Sha256ToHex fmt::format_to per-byte loop with hex lookup table and ExtendableStringBuilder
- Replace AwsUriEncode per-char fmt::format for percent-encoding with hex lookup table
- Replace transient fmt::format calls in SignRequestV4 and GeneratePresignedUrl with ExtendableStringBuilder appends
- Return string_view from GetDateStamp instead of allocating an 8-byte std::string
- Replace fmt::format for XML tag construction in ExtractXmlValue with ExtendableStringBuilder
- Replace fmt::format in KeyToPath/BucketRootPath with ExtendableStringBuilder
- Replace ToLower heap allocations in case-insensitive MatchWildcard with inline tolower loop
- Replace fmt::format per log line in logstreamlistener with ExtendableStringBuilder
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Add AppendPaddedInt() to StringBuilderBase for zero-padded integer formatting without fmt::format
- Add StringBuilderAppender output iterator adapter for direct use with fmt::format_to
- Replace 18 fmt::format temporaries in DateTime::ToString and TimeSpan::ToString with AppendPaddedInt
- Replace fmt::format float formatting in JSON/YAML writers with fmt::format_to via StringBuilderAppender
- Replace fmt::format for Uuid/ObjectId in YAML writer with direct ToString(StringBuilderBase&)
- Replace std::vector with eastl::fixed_vector in CB validation (field names, attachments)
- Avoid Path.string() allocation in CreateDirectories by checking native() directly
|
| | |
| |
| |
| |
| |
| |
| | |
- Replace std::vector with eastl::fixed_vector in hot paths: URI parser (per-request), CommitPutBatch, WriteChunks, IterateBlock, IterateChunks
- Hoist InvalidEntryReason string outside ReadLog replay lambda to reuse buffer across iterations
- Replace std::string concatenation with ExtendableStringBuilder in path helpers
- Eliminate 257 heap allocations in Drop() by replacing vector<unique_ptr<ExclusiveLockScope>> with fixed_vector<ExclusiveLockScope, 256>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
- Replace fmt::format with ExtendableStringBuilder for transient URL and field-name construction in remotehttprunner, httpcomputeservice
- Use TransparentStringHash on orchestrator worker/client maps to eliminate std::string temporaries on lookup
- Replace std::atoi(std::string(...).c_str()) with zen::ParseInt in httporchestrator
- Switch macrunner BuildSandboxProfile from std::string concatenation to ExtendableStringBuilder<1024>
- Eliminate triple std::string allocation in workertimeline TimelinePath
- Change SubmitActionBatch signature from const std::string& to std::string_view
|
| |/
|
|
|
| |
- Enables std::string_view lookups on std::string-keyed unordered maps without allocating a temporary std::string
- Uses is_transparent tag for C++20 heterogeneous lookup support
|
| |
|
|
|
|
|
|
| |
* reduce zenserver spawns in tests
* fix filesystemutils wrong test suite name
* tweak tests for faster runtime
* reduce more test runtime
* more wall time improvements
* fast http and processmanager tests
|
| |
|
|
|
|
| |
- Detect stale shared-memory entries whose PID matches the current process but predate our registration (m_OurEntry == nullptr)
- Sweep() now reclaims such entries instead of skipping them
- Lookup() and LookupByEffectivePort() skip stale same-PID entries
- Fixes startup failure on k8s where PID 1 is always reused after an unclean shutdown
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Adds `AsyncHttpClient` — an asynchronous HTTP client using `curl_multi_socket_action` integrated with ASIO for event-driven I/O. Supports GET, POST, PUT, DELETE, HEAD with both callback-based and `std::future`-based APIs.
- Extracts shared curl helpers (callbacks, URL encoding, header construction, error mapping) into `httpclientcurlhelpers.h`, eliminating duplication between the sync and async implementations.
## Design
- All curl_multi state is serialized on an `asio::strand`, safe with multi-threaded io_contexts.
- Two construction modes: owned io_context (creates internal thread) or external io_context (caller runs the loop).
- Socket readiness is detected via `asio::ip::tcp::socket::async_wait` driven by curl's `CURLMOPT_SOCKETFUNCTION`/`CURLMOPT_TIMERFUNCTION` — no polling, sub-millisecond latency.
- Completion callbacks are dispatched off the strand onto the io_context so slow callbacks don't starve the curl event loop. Exceptions in callbacks are caught and logged.
## Files
| File | Change |
|------|--------|
| `zenhttp/include/zenhttp/asynchttpclient.h` | New public header |
| `zenhttp/clients/asynchttpclient.cpp` | Implementation (~1000 lines) |
| `zenhttp/clients/httpclientcurlhelpers.h` | Shared curl helpers extracted from sync client |
| `zenhttp/clients/httpclientcurl.cpp` | Removed duplicated helpers, uses shared header |
| `zenhttp/asynchttpclient_test.cpp` | 8 test cases: verbs, payloads, callbacks, concurrency, external io_context, connection errors |
| `zenhttp/zenhttp.cpp` | Forcelink registration for new tests |
|
| | |
|
| |
|
|
| |
in consul (#930)
|
| |
|
|
| |
(#927)
|
| | |
|
| |
|
| |
* add reused block to oplog during export
|
| |
|
|
| |
- Feature: Hub obliterate operation deletes all local and backend hydration data for a module
- Improvement: Hub dashboard adds obliterate button for individual, bulk, and by-name module deletion
|
| |
|
| |
* add pagination and consistent sorting on cache and projects ui pages
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Incremental CAS-based hydration/dehydration replacing the previous full-copy approach
- Feature: S3 hydration backend with multipart upload/download support
- Feature: Configurable thread pools for hub instance provisioning and hydration
`--hub-instance-provision-threads` defaults to `max(cpu_count / 4, 2)`. Set to 0 for synchronous operation.
`--hub-hydration-threads` defaults to `max(cpu_count / 4, 2)`. Set to 0 for synchronous operation.
- Improvement: Hub triggers GC on instance before deprovisioning to compact storage before dehydration
- Improvement: GC status now reports pending triggers as running
- Improvement: S3 client debug logging gated behind verbose mode to reduce log noise at default verbosity
- Improvement: Hub dashboard Resources tile now shows total memory
- Improvement: `filesystemutils` moved from `zenremotestore` to `zenutil` for broader reuse
- Improvement: Hub uses separate provision and hydration worker pools to avoid deadlocks
- Improvement: Hibernate/wake/deprovision on non-existent or already-in-target-state modules are idempotent
- Improvement: `ScopedTemporaryDirectory` with empty path now creates a temporary directory instead of asserting
|
| |
|
|
| |
* use correct health endpoint for zenhubserver consul registration
* add total disk space on hub resource pane
|
| |
|
|
|
|
|
|
|
|
|
| |
* fix endpoint for stats/hub in compute/hub.html page
* fix api token call failure for imds (using wrong overload for Put)
* add "localhost" to healt check url in consul when no address is given
* add consul fallback deregister if normal deregister fails
* add consul registration unit test
|
| | |
|
| |
|
| |
- Feature: Hub dashboard proxy - instance dashboards are accessible through the hub server at `/hub/proxy/{port}/` without requiring direct port access
|
| |
|
|
|
| |
- Improvement: Hub child process spawning on macOS now uses `posix_spawn` in line with Apple recommendations
- Bugfix: Hub child process spawning on Linux now uses `vfork` instead of `fork`, preventing ENOMEM failures on systems with strict memory overcommit (`vm.overcommit_memory=2`)
- Bugfix: Fixed process group management on POSIX; child processes were not placed into the correct process group, breaking group-wide signal delivery
|
| |
|
| |
- Improvement: Consul token is now re-read from the environment variable on every request, allowing token rotation without restarting the service
|
| |
|
|
|
| |
CI test runs (#909)
Adds steps to the validate workflow on all platforms that kill any zenserver, minio, nomad, or consul processes launched from the build output directory. Runs before tests to clear stale processes from previous runs, and after tests (always, even on failure) to clean up.
|
| |
|
|
|
|
| |
* Unit test coverage for zero byte file handling in oplogs
* Unit test fixes for the zero length file case
* Fixes for zero length file attachments
* Additional fix for zero length file attachments
|
| |
|
|
|
|
|
|
| |
(#907)
* fix potential race with stats counters missing when to Stop filtered values
* fix off by one in PutMultipartBuildBlob retry path
* use move operation instead of copy operation PutMultipartBlob
* fix filter Stop() for upload operations and fix bug with generateblock count filter
|
| |
|
| |
- Bugfix: Fixed concurrency issue in JupiterBuildStorage when updating stats
|
| |
|
|
|
|
|
|
|
|
|
| |
- Improvement: Hub server now supports Lua config file for all hub-specific options
- `hub.upstreamnotification.*` - upstream notification endpoint and instance ID
- `hub.consul.*` - service registration endpoint, token, health interval, deregister timeout
- `hub.instance.*` - base port, HTTP class, thread count, core limit, config path
- `hub.instance.limits.*` - instance count cap, disk and memory usage limits
- `hub.hydration.*` - hydration target spec and config path
- `hub.watchdog.*` - cycle timing, inactivity timeouts, and activity check timeouts
- Improvement: Added `--hub-instance-base-port-number` as an alias for `--hub-base-port-number`, and `--upstream-notification-instance-id` as an alias for `--instance-id`
- Improvement: Added hub mode documentation at docs/hub.md
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Security: Input validation & path safety
- **Reject local file references by default** in package parsing — only allow when explicitly opted in by the service (`ParseFlags::kAllowLocalReferences`) and validated by an `ILocalRefPolicy` (fail-closed: no policy = rejected)
- **`DataRootLocalRefPolicy`** restricts local ref paths to the server's data root via canonical path prefix matching
- **Validate attachment hashes** in compute HTTP handlers — decompresses and re-hashes each attachment at ingestion time to reject tampered payloads
- **Path traversal validation** for worker descriptions (`pathvalidation.h`) — rejects absolute paths, `..` components, Windows reserved device names, and invalid filename characters
- **Harden CbPackage parsing** against corrupt inputs — overflow-safe attachment count, bounds checks on local ref offset/size, graceful failure instead of `ZEN_ASSERT` for untrusted data
- **Harden legacy package parser** — reject zero-size binary fields, missing mappers, and optionally validate resolved attachment hashes
- **Bounds check in `CbPackageReader::MarshalLocalChunkReference`** — detect when `MakeFromFile` silently clamps offset+size to file size
### Reliability: Lock consolidation & bug fixes
- **Consolidate three action map locks into one** (`m_ActionMapLock`) — eliminates deadlock risk from multi-lock ordering, simplifies state transitions, and fixes a race where newly enqueued actions were briefly invisible to `GetActionResult`/`FindActionResult`
- **Fix infinite loop in `BaseRunnerGroup::SubmitActions`** when actions exceed total runner capacity — cap round-robin at `TotalCapacity` and default unassigned results to "No capacity"
- **Fix `MakeSafeAbsolutePathInPlace` for UNC paths** — `\server\share` now correctly becomes `\?\UNC\server\share` instead of `\?\server\share`
- **Fix `max_retries=0`** — previously fell through to the default of 3; now correctly means "no retries"
### New: ManagedProcessRunner
- Cross-platform process runner backed by `SubprocessManager` — uses async exit callbacks instead of polling, delegates CPU/memory metrics to the manager's built-in sampler
- `ProcessGroup` (JobObject on Windows, process group on POSIX) for bulk cancellation on shutdown
- `--managed` flag on `zen exec inproc` to select this runner
- Refactored monitor thread lifecycle — `StartMonitorThread()` now called from derived constructors to avoid calling virtual functions from base constructor
### Process management
- **Suppress crash dialogs** via `JOB_OBJECT_UILIMIT_ERRORMODE` + `SEM_NOGPFAULTERRORBOX` in both `WindowsProcessRunner` and `JobObject::Initialize` — prevents WER/Dr. Watson modal dialogs from blocking the monitor thread
- **CREATE_SUSPENDED → AssignProcessToJobObject → ResumeThread** pattern in `WindowsProcessRunner` — ensures job object assignment before process execution
- **Move stdout/stderr callbacks to `Spawn()` parameters** in `SubprocessManager` — prevents race where early output could be missed before callback installation
- Consistent PID logging across all runner types
### Test infrastructure
- **`zentest-appstub`**: Added `Fail` (configurable exit code) and `Crash` (abort / nullptr deref) test functions
- **Compute integration tests**: exit code handling, auto-retry exhaustion, manual reschedule after failure, mixed success/failure queues, crash handling (abort + nullptr), crash auto-retry, immediate query visibility after enqueue
- **Package format tests**: truncated header, bad magic, attachment count overflow, truncated data, local ref rejection/acceptance, policy enforcement (inside/outside root, traversal, no-policy fail-closed)
- **Legacy package parser tests**: empty input, zero-size binary, hash resolution with/without mapper, hash mismatch detection
- **UNC path tests** for `MakeSafeAbsolutePath`
### Misc
- ANSI color helper macros (`ZEN_RED`, `ZEN_BRIGHT_WHITE`, etc.) and `ZEN_BOLD`/`ZEN_DIM`/etc.
- Generic `fmt::formatter` for types with free `ToString` functions
- Compute dashboard: truncated hash display with monospace font and hover for full value
- Renamed `usonpackage_forcelink` → `cbpackage_forcelink`
- Compute enabled by default in xmake config (releases still explicitly disable)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Added `--hub-hydration-target-config` option to specify the hydration target via a JSON config file (mutually exclusive with `--hub-hydration-target-spec`); supports `file` and `s3` types with structured settings
```json
{
"type": "file",
"settings": {
"path": "/path/to/hydration/storage"
}
}
```
```json
{
"type": "s3",
"settings": {
"uri": "s3://bucket[/prefix]",
"region": "us-east-1",
"endpoint": "http://localhost:9000",
"path-style": true
}
}
```
- Improvement: Hub hydration dehydration skips the `.sentry-native` directory
- Bugfix: Fixed `MakeSafeAbsolutePathInPlace` when a UNC prefix is present but path uses mixed delimiters
|
| |
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Hub dashboard now shows a Resources tile with disk and memory usage against configured limits
- Feature: Hub module listing now shows state-change timestamps and duration for each instance
- Improvement: Hub provisioning rejects new instances when disk or memory usage exceeds configurable thresholds; limits are disabled by default (0 = no limit)
- `--hub-provision-disk-limit-bytes` - Reject provisioning when used disk exceeds this many bytes
- `--hub-provision-disk-limit-percent` - Reject provisioning when used disk exceeds this percentage of total disk
- `--hub-provision-memory-limit-bytes` - Reject provisioning when used memory exceeds this many bytes
- `--hub-provision-memory-limit-percent` - Reject provisioning when used memory exceeds this percentage of total RAM
- Improvement: Hub process metrics are now tracked atomically per active instance slot, eliminating per-query process handle lookups
- Improvement: Hub, Build Store, and Workspaces service stats sections in the dashboard are now collapsible
- Bugfix: Hub watchdog loop did not check `m_ShutdownFlag`, causing it to spin indefinitely on shutdown
|
| |
|
| |
Replace doctest SUBCASEs with sequential scoped blocks so the MinIO server is spawned once and torn down via RAII at scope exit, instead of being restarted for every subcase re-entry. Fixes flaky CI on macOS caused by repeated MinIO process start/stop.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
- **Eliminate `<regex>` usage** — Replaced `std::regex`-based URL parsing in `jupiterbuildstorage.cpp` with manual `string_view` parsing. Added `CXXOPTS_NO_REGEX` to disable regex in cxxopts. Includes comprehensive tests for the new URL parser.
- **Add missing HTTP response codes** — Added `102`, `103`, `203`, `207`, `208`, `226`, `306`, `421`, `425`, `451` to the enum and reason string lookup.
- **Add `ForceColor` support to zen CLI** — Plumbed the `ForceColor` logging option through to the zen client.
- **Add `.clangd` config** — Strips MSVC-specific flags clangd can't handle and suppresses noisy clang-tidy checks.
- **Generic `fmt::formatter` for `ToString`** — Concept-based formatter that auto-formats any type with a free `ToString()` function, removing the need for per-type specializations.
- **Fix OpenSSL dependency** — Changed `zenhorde` to use `openssl3` package on Linux/macOS.
- **Add `<cmath>` include** — Missing include in `hyperloglog.h`.
- **GCC compile fix** — Moved `static constinit` variable inside lambda in `logging.cpp`.
|
| |
|
|
|
|
| |
- Feature: Added Workspaces dashboard page with HTTP request stats and per-workspace metrics
- Feature: Added Build Storage dashboard page with service-specific HTTP request stats
- Improvement: Front page now shows Hub and Object Store activity tiles; HTTP panel is fixed above the tiles grid
- Improvement: HTTP stats tiles now include 5m/15m rates and p999/max latency across all service pages
|
| |
|
| |
CPR is no longer needed now that HttpClient has fully transitioned to raw libcurl. This removes the CPR library, its build integration, implementation files, and all conditional compilation guards, leaving curl as the sole HTTP client backend.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Hub watchdog automatically deprovisions inactive provisioned and hibernated instances
- Feature: Added `stats/activity_counters` endpoint to measure server activity
- Feature: Added configuration options for hub watchdog
- `--hub-watchdog-provisioned-inactivity-timeout-seconds` Inactivity timeout before a provisioned instance is deprovisioned
- `--hub-watchdog-hibernated-inactivity-timeout-seconds` Inactivity timeout before a hibernated instance is deprovisioned
- `--hub-watchdog-inactivity-check-margin-seconds` Margin before timeout at which an activity check is issued
- `--hub-watchdog-cycle-interval-ms` Watchdog poll interval in milliseconds
- `--hub-watchdog-cycle-processing-budget-ms` Maximum time budget per watchdog cycle in milliseconds
- `--hub-watchdog-instance-check-throttle-ms` Minimum delay between checks on a single instance
- `--hub-watchdog-activity-check-connect-timeout-ms` Connect timeout for activity check requests
- `--hub-watchdog-activity-check-request-timeout-ms` Request timeout for activity check requests
|
| |
|
|
|
|
| |
- Improvement: Provisioning a hibernated instance now automatically wakes it instead of requiring an explicit wake call first
- Improvement: Deprovisioning now accepts instances in Crashed or Hibernated states, not just Provisioned
- Improvement: Added `--consul-health-interval-seconds` and `--consul-deregister-after-seconds` options to control Consul health check behavior (defaults: 10s and 30s)
- Improvement: Consul registration now occurs when provisioning starts; health check intervals are applied once provisioning completes
|
| |
|
|
|
| |
- Improvement: Hub provision, deprovision, hibernate, and wake operations are now async. HTTP requests returns 202 Accepted while the operation completes in the background
- Improvement: Hub returns 202 Accepted (instead of 409 Conflict) when the same async operation is already in progress for a module
- Improvement: Hub returns 200 OK when a requested state transition is already satisfied
|