| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
This change introduces job object support on Windows to be able to more accurately track and limit resource usage on storage instances created by the hub service. It also ensures that all child instances can be torn down reliably on exit.
Also made it so hub tests no longer pop up console windows while running.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
**CI/CD improvements (validate.yml):**
- Add test reporter (`ue-foundation/test-reporter@v2`) for all three platforms, rendering JUnit test results directly in PR check runs
- Add "Trust workspace" step on Windows to fix git safe.directory ownership issue with self-hosted runners
- Clean stale report files before each test run to prevent false failures from leftover XML
- Broaden `paths-ignore` to skip builds for non-code changes (`*.md`, `LICENSE`, `.gitignore`, `docs/**`)
**Test improvements:**
- Convert `CHECK` to `REQUIRE` in several test suites (projectstore, integration, http) for fail-fast behavior
- Mark some tests with `doctest::skip()` for selective execution
- Skip httpclient transport tests pending investigation
- Add `--noskip` option to `xmake test` task
- Add `--repeat=<N>` option to `xmake test` task, to run tests repeatedly N times or until there is a failure
**xmake test output improvements:**
- Add totals row to test summary table
- Right-justify numeric columns in summary table
|
| |
|
| |
* add multirange requests to blob store
|
| |
|
|
|
|
|
|
|
|
| |
- Add a summary table printed after all test suites complete, showing per-suite test case counts, assertion counts, timings and pass/fail status.
- Add failure reporting: individual failing test cases are listed at the end with their file path and line number for easy navigation.
- Made zenserver instances spawned by a hub not create new console windows for a better background testing experience
- The TestListener in testing.cpp now writes a machine-readable summary file (via `ZEN_TEST_SUMMARY_FILE` env var) containing aggregate counts and per-test-case failure details. This runs as a doctest listener alongside any active reporter, so it works with both console and JUnit modes.
- Tests now run in a deterministic order defined by a single ordered list that also serves as the test name/target mapping, replacing the previous unordered table + separate order list.
- The `--run` option now accepts comma-separated values (e.g. `--run=core,http,util`) and validates each name, reporting unknown test names early.
- Fix platform detection in `xmake test`: the config command now passes `-p` explicitly, fixing "mingw" misdetection when running from Git Bash on Windows.
- Add missing "util" entry to the help text for `--run`.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* This branch adds full WebSocket (RFC 6455) support to the HTTP server layer, covering both transport backends, a client, and tests.
- **`websocket.h`** -- Core interfaces: `WebSocketOpcode`, `WebSocketMessage`, `WebSocketConnection` (ref-counted), and `IWebSocketHandler`. Services opt in to WebSocket support by implementing `IWebSocketHandler` alongside their existing `HttpService`.
- **`httpwsclient.h`** -- `HttpWsClient`: an ASIO-backed `ws://` client with both standalone (own thread) and shared `io_context` modes. Supports connect timeout and optional auth token injection via `IWsClientHandler` callbacks.
- **`wsasio.cpp/h`** -- `WsAsioConnection`: WebSocket over ASIO TCP. Takes over the socket after the HTTP 101 handshake and runs an async read/write loop with a queued write path (guarded by `RwLock`).
- **`wshttpsys.cpp/h`** -- `WsHttpSysConnection`: WebSocket over http.sys opaque-mode connections (Windows only). Uses `HttpReceiveRequestEntityBody` / `HttpSendResponseEntityBody` via IOCP, sharing the same threadpool as normal http.sys traffic. Self-ref lifetime management ensures graceful drain of outstanding async ops.
- **`httpsys_iocontext.h`** -- Tagged `OVERLAPPED` wrapper (`HttpSysIoContext`) used to distinguish normal HTTP transactions from WebSocket read/write completions in the single IOCP callback.
- **`wsframecodec.cpp/h`** -- `WsFrameCodec`: static helpers for parsing (unmasked and masked) and building (unmasked server frames and masked client frames) RFC 6455 frames across all three payload length encodings (7-bit, 16-bit, 64-bit). Also computes `Sec-WebSocket-Accept` keys.
- **`clients/httpwsclient.cpp`** -- `HttpWsClient::Impl`: ASIO-based client that performs the HTTP upgrade handshake, then hands off to the frame codec for the read loop. Manages its own `io_context` thread or plugs into an external one.
- **`httpasio.cpp`** -- ASIO server now detects `Upgrade: websocket` requests, checks the matching `HttpService` for `IWebSocketHandler` via `dynamic_cast`, performs the RFC 6455 handshake (101 response), and spins up a `WsAsioConnection`.
- **`httpsys.cpp`** -- Same upgrade detection and handshake logic for the http.sys backend, using `WsHttpSysConnection` and `HTTP_SEND_RESPONSE_FLAG_OPAQUE`.
- **`httpparser.cpp/h`** -- Extended to surface the `Upgrade` / `Connection` / `Sec-WebSocket-Key` headers needed by the handshake.
- **`httpcommon.h`** -- Minor additions (probably new header constants or response codes for the WS upgrade).
- **`httpserver.h`** -- Small interface changes to support WebSocket registration.
- **`zenhttp.cpp` / `xmake.lua`** -- New source files wired in; build config updated.
- **Unit tests** (`websocket.framecodec`): round-trip encode/decode for text, binary, close frames; all three payload sizes; masked and unmasked variants; RFC 6455 `Sec-WebSocket-Accept` test vector.
- **Integration tests** (`websocket.integration`): full ASIO server tests covering handshake (101), normal HTTP coexistence, echo, server-push broadcast, client close handshake, ping/pong auto-response, sequential messages, and rejection of upgrades on non-WS services.
- **Client tests** (`websocket.client`): `HttpWsClient` connect+echo+close, connection failure (bad port -> close code 1006), and server-initiated close.
* changed HttpRequestParser::ParseCurrentHeader to use switch instead of if/else chain
* remove spurious printf
---------
Co-authored-by: Stefan Boberg <[email protected]>
|
| |
|
|
|
| |
eliminates spurious sentry log output during startup as the new channel defaults to WARN
The level can be overridden via `--log-debug=sentry-sdk` or `--log-info=sentry-sdk`
|
| | |
|
| |
|
| |
previously it would stall some 40s in this case
|
| |
|
| |
* add support in http client to accept multi-range responses
|
| |
|
| |
* Ported "lane trace" feature from UE (by way of IAX)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add comprehensive `HttpClient` test suite. Covers:
- **HTTP verbs** -- GET, POST, PUT, DELETE, HEAD dispatch correctly
- **GET/POST/PUT/Upload/Download** -- payload round-trips (IoBuffer, CbObject, CompositeBuffer), content types, large payloads, file-spill downloads
- **Status codes** -- 2xx/4xx/5xx classification, exact code matching
- **Response API** -- IsSuccess, AsText, AsObject, ToText, ErrorMessage, ThrowError
- **Error handling** -- connection refused, request timeout, nonexistent endpoints
- **Session management** -- default ID, SetSessionId, reset to zero
- **Authentication** -- token provider, expired tokens, bearer verification
- **Content type detection** -- text, JSON, binary, CbObject
- **Request metadata** -- elapsed time, upload/download byte counts
- **Retry logic** -- retry after transient 503s, no-retry baseline
- **Latency measurement** -- MeasureLatency against live and unreachable servers
- **KeyValueMap** -- construction from pairs, string_views, initializer lists
- **Transport-level faults (GET)** -- connection reset/close before response, partial headers, truncated body, mid-body reset, stalled response timeout, retry after RST
- **Transport-level faults (POST)** -- server reset/close before consuming body, mid-body reset, early 503 without consuming upload, stalled upload timeout, retry with large body after transient failures
Also adds zenhttp-test to the xmake test runner (xmake test --run=http).
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| | |
* tweaked doctest.h to avoid shutdown issues due to thread_local variables running destructors after the main thread has torn down everything including the heap
* disabled zenserver exit thread waiting since doctest should hopefully not be causing issues during shutdown anymore after my workaround
This should help reduce the duration of tests spawning lots of server instances
|
| | |
| |
| |
| |
| |
| |
| | |
- zenhttp: added `GetServiceUri()`/`GetExternalHost()`
- enables code to quickly generate an externally reachable URI for a given service
- frontend: improved Uri handling (better defaults)
- added support for 404 page (to make it easier to find a good URL)
|
| | |
| |
| |
| |
| |
| |
| | |
- Add missing includes in hashutils.h (`<cstddef>`, `<type_traits>`)
- Add `ZenContentType` parameter to all `IoBufferBuilder` factory methods so content type is set at buffer creation time
- Fix null dereference in `SharedBuffer::GetFileReference()` when buffer is null
- Fix out-of-bounds read in trace command-line argument parsing when arg length exactly matches option length
- Add unit tests for 32-bit `CountLeadingZeros`
|
| | |
| |
| |
| |
| | |
Feature: Add --allow-partial-block-requests to zen oplog-import
Improvement: zen oplog-import now uses partial block requests to reduce download size
Improvement: Use latency to Cloud Storage host and Zen Cache host when calculating partial block requests
|
| | |
| |
| |
| |
| | |
Allows user to automate launching of zenserver dashboard, including when multiple instances are running. If multiple instances are running you can open all dashboards with `--all`, and also using the in-terminal chooser which also allows you to open a specific instance.
Also includes a fix to `zen exec` when using offset/stride/limit
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
zencore fixes:
- filesystem.cpp: ReadFile error reporting logic
- compactbinaryvalue.h: CbValue::As*String error reporting logic
zenhttp fixes:
- httpasio BindAcceptor would `return 0;` in a function returning `std::string` (UB)
- httpsys async workpool initialization race
zenstore fixes:
- cas.cpp: GetFileCasResults Results param passed by value instead of reference (large chunk results were silently lost)
- structuredcachestore.cpp: MissCount unconditionally incremented (counted hits as misses)
- cacherpc.cpp: Wrong boolean in Incomplete response array (all entries marked incomplete)
- cachedisklayer.cpp: sizeof(sizeof(...)) in two validation checks computed sizeof(size_t) instead of struct size
- buildstore.cpp: Wrong hash tracked in GC key list (BlobHash pushed twice instead of MetadataHash)
- buildstore.cpp: Removed duplicate m_LastAccessTimeUpdateCount increment in PutBlob
zenserver fixes:
- httpbuildstore.cpp: Reversed subtraction in HTTP range calculation (unsigned underflow)
- hubservice.cpp: Deadlock in Provision() calling Wake() while holding m_Lock (extracted WakeLocked helper)
- zipfs.cpp: Data race in GetFile() lazy initialization (added RwLock with shared/exclusive paths)
|
| | |
| |
| |
| | |
This reverts commit 3c89c486338890ce39ddebe5be4722a09e85701a.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
zenstore fixes:
- cas.cpp: GetFileCasResults Results param passed by value instead of reference (large chunk results were silently lost)
- structuredcachestore.cpp: MissCount unconditionally incremented (counted hits as misses)
- cacherpc.cpp: Wrong boolean in Incomplete response array (all entries marked incomplete)
- cachedisklayer.cpp: sizeof(sizeof(...)) in two validation checks computed sizeof(size_t) instead of struct size
- buildstore.cpp: Wrong hash tracked in GC key list (BlobHash pushed twice instead of MetadataHash)
- buildstore.cpp: Removed duplicate m_LastAccessTimeUpdateCount increment in PutBlob
zenserver fixes:
- httpbuildstore.cpp: Reversed subtraction in HTTP range calculation (unsigned underflow)
- hubservice.cpp: Deadlock in Provision() calling Wake() while holding m_Lock (extracted WakeLocked helper)
- zipfs.cpp: Data race in GetFile() lazy initialization (added RwLock with shared/exclusive paths)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
|
| | |
| |
| |
| |
| | |
this implements a yaml generation strategy similar to the JSON generation where we just build a string instead of building a ryml tree.
This also removes the dependency on ryml for reduced binary/build times.
|
| | |
| |
| |
| | |
string literals in zen.cpp (#776)
|
| | |
| |
| | |
Co-authored-by: Stefan Boberg <[email protected]>
|
| | | |
|
| | |
| |
| | |
* fix build download indexing timer log
|
| | |
| |
| | |
* fix plain progress not updating current state
|
| | |
| |
| |
| |
| | |
(was MakeSafeAbsolutePathÍnPlace - note accent)
Also fixed misleading comments on multiple functions in filesystem.h
|
| | |
| |
| |
| |
| |
| |
| |
| | |
file retry logic (#766)
* GC - fix handling of attachment ranges
* fix trace/log strings
* fix HTTP access token expiration time logic
* added missing lock retry in zenserver startup
|
| | | |
|
| |/ |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver
this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes
to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system
all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
|
| | |
| |
| |
| |
| | |
* implemented selective request logging for http.sys for consistency with asio
* fixed traversal of GetLogicalProcessorInformationEx to account for variable-sized records
* also adds CPU usage metrics
|
| | |
| |
| | |
* convert ZEN_ASSERTs to exception to handle corrupt data gracefully
|
| | | |
|
| | | |
|
| | | |
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Feature: Added `--security-config-path` option to zenserver to configure security settings
- Expects a path to a .json file
- Default is an empty path resulting in no extra security settings and legacy behavior
- Current support is a top level filter of incoming http requests restricted to the `password` type
- `password` type will check the `Authorization` header and match it to the selected authorization strategy
- Currently the security settings is very basic and configured to a fixed username+password at startup
{
"http" {
"root": {
"filter": {
"type": "password",
"config": {
"password": {
"username": "<username>",
"password": "<password>"
},
"protect-machine-local-requests": false,
"unprotected-uris": [
"/health/",
"/health/info",
"/health/version"
]
}
}
}
}
}
|
| | |
| |
| |
| |
| |
| |
| |
| | |
* `RwLock::WithSharedLock` and `RwLock::WithExclusiveLock` can now return a value (which is returned by the passed function)
* Comma-separated logger specification now correctly deals with commas
* `GetSystemMetrics` properly accounts for cores
* cpr response formatter passes arguments in the right order
* `HttpServerRequest::SetLogRequest` can be used to selectively log HTTP requests
|
| | |
| |
| | |
also made sure log initialization calls it to ensure the console output format is retained even if the console logger was set up before logging is initialized
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|