| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
- Add `--no-network` CLI option which disables all TCP/HTTPS listeners, restricting zenserver to Unix domain socket communication only.
- Also fixes asio upgrade breakage on main
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Migrate removed deprecated APIs:
- io_service -> io_context
- io_service::work -> executor_work_guard
- resolver::query/iterator -> resolver::resolve() with results_type
- address::from_string() -> make_address()
---
Breaking Changes (1.33.0)
- deferred as default completion token — can omit token in coroutines: co_await socket.async_read_some(buf)
- cancel_after / cancel_at — timeout any async operation: co_await sock.async_read_some(buf, cancel_after(5s))
- Partial completion token adapters — as_tuple, redirect_error, bind_executor etc. composable via pipe: co_await (async_write(sock, buf) | as_tuple | cancel_after(10s))
- composed — simpler alternative to async_compose for stateful operation implementations
- co_composed moved out of experimental
1.35.0 — Allocator & Resolver
- Allocator constructors for io_context and thread_pool — control memory allocation for services, I/O objects, strands
- Configurable resolver thread pool ("resolver"/"threads")
- Timer heap pre-allocation ("timer"/"heap_reserve")
1.37.0 — Inline Executors & Reactor Tuning
- inline_executor — always executes inline (useful as completion executor)
- inline_or_executor<> — tries inline first, falls back to wrapped executor
- New dispatch/post/defer overloads that run a function on one executor and deliver result to a handler on another
- redirect_disposition — captures disposition into a variable (like redirect_error but generic)
- Reactor config: reset_edge_on_partial_read, use_eventfd, use_timerfd
Notable Fixes
- Resource leak in awaitable move assignment (1.37.0)
- Memory leak in SSL stream move assignment (1.37.0)
- Thread sanitizer issue in kqueue reactor (1.37.0)
- co_spawn non-reentrant completion handler fix (1.36.0)
- Windows file append mode fix (1.32.0)
- SSL engine move assignment leak (1.33.0)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main goal of this change is to eliminate the cpr back-end altogether and replace it with the curl implementation. I would expect to drop cpr as soon as we feel happy with the libcurl back-end. That would leave us with a direct dependency on libcurl only, and cpr can be eliminated as a dependency.
### HttpClient Backend Overhaul
- Implemented a new **libcurl-based HttpClient** backend (`httpclientcurl.cpp`, ~2000 lines)
as an alternative to the cpr-based one
- Made HttpClient backend **configurable at runtime** via constructor arguments
and `-httpclient=...` CLI option (for zen, zenserver, and tests)
- Extended HttpClient test suite to cover multipart/content-range scenarios
### Unix Domain Socket Support
- Added Unix domain socket support to **httpasio** (server side)
- Added Unix domain socket support to **HttpClient**
- Added Unix domain socket support to **HttpWsClient** (WebSocket client)
- Templatized `HttpServerConnectionT<SocketType>` and `WsAsioConnectionT<SocketType>`
to handle TCP, Unix, and SSL sockets uniformly via `if constexpr` dispatch
### HTTPS Support
- Added **preliminary HTTPS support to httpasio** (for Mac/Linux via OpenSSL)
- Added **basic HTTPS support for http.sys** (Windows)
- Implemented HTTPS test for httpasio
- Split `InitializeServer` into smaller sub-functions for http.sys
### Other Notable Changes
- Improved **zenhttp-test stability** with dynamic port allocation
- Enhanced port retry logic in http.sys (handles ERROR_ACCESS_DENIED)
- Fatal signal/exception handlers for backtrace generation in tests
- Added `zen bench http` subcommand to exercise network + HTTP client/server communication stack
|
| |
|
|
|
|
|
|
|
|
| |
- **Frontend dashboard overhaul**: Unified compute/main dashboards into a single shared UI. Added new pages for cache, projects, metrics, sessions, info (build/runtime config, system stats). Added live-update via WebSockets with pause control, sortable detail tables, themed styling. Refactored compute/hub/orchestrator pages into modular JS.
- **HTTP server fixes and stats**: Fixed http.sys local-only fallback when default port is in use, implemented root endpoint redirect for http.sys, fixed Linux/Mac port reuse. Added /stats endpoint exposing HTTP server metrics (bytes transferred, request rates). Added WebSocket stats tracking.
- **OTEL/diagnostics hardening**: Improved OTLP HTTP exporter with better error handling and resilience. Extended diagnostics services configuration.
- **Session management**: Added new sessions service with HTTP endpoints for registering, updating, querying, and removing sessions. Includes session log file support. This is still WIP.
- **CLI subcommand support**: Added support for commands with subcommands in the zen CLI tool, with improved command dispatch.
- **Misc**: Exposed CPU usage/hostname to frontend, fixed JS compact binary float32/float64 decoding, limited projects displayed on front page to 25 sorted by last access, added vscode:// link support.
Also contains some fixes from TSAN analysis.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Removes the vendored spdlog library (~12,000 lines) and replaces it with a purpose-built logging system in zencore (~1,800 lines). The new implementation provides the same functionality with fewer abstractions, no shared_ptr overhead, and full control over the logging pipeline.
### What changed
**New logging core in zencore/logging/:**
- LogMessage, Formatter, Sink, Logger, Registry - core abstractions matching spdlog's model but simplified
- AnsiColorStdoutSink - ANSI color console output (replaces spdlog stdout_color_sink)
- MsvcSink - OutputDebugString on Windows (replaces spdlog msvc_sink)
- AsyncSink - async logging via BlockingQueue worker thread (replaces spdlog async_logger)
- NullSink, MessageOnlyFormatter - utility types
- Thread-safe timestamp caching in formatters using RwLock
**Moved to zenutil/logging/:**
- FullFormatter - full log formatting with timestamp, logger name, level, source location, multiline alignment
- JsonFormatter - structured JSON log output
- RotatingFileSink - rotating file sink with atomic size tracking
**API changes:**
- Log levels are now an enum (LogLevel) instead of int, eliminating the zen::logging::level namespace
- LoggerRef no longer wraps shared_ptr - it holds a raw pointer with the registry owning lifetime
- Logger error handler is wired through Registry and propagated to all loggers on registration
- Logger::Log() now populates ThreadId on every message
**Cleanup:**
- Deleted thirdparty/spdlog/ entirely (110+ files)
- Deleted full_test_formatter (was ~80% duplicate of FullFormatter)
- Renamed snake_case classes to PascalCase (full_formatter -> FullFormatter, json_formatter -> JsonFormatter, sentry_sink -> SentrySink)
- Removed spdlog from xmake dependency graph
### Build / test impact
- zencore no longer depends on spdlog
- zenutil and zenvfs xmake.lua updated to drop spdlog dep
- zentelemetry xmake.lua updated to drop spdlog dep
- All existing tests pass, no test changes required beyond formatter class renames
|
| |
|
|
|
|
|
|
|
|
| |
- Added local process runners for Linux/Wine, Mac with some sandboxing support
- Horde & Nomad provisioning for development and testing
- Client session queues with lifecycle management (active/draining/cancelled), automatic retry with configurable limits, and manual reschedule API
- Improved web UI for orchestrator, compute, and hub dashboards with WebSocket push updates
- Some security hardening
- Improved scalability and `zen exec` command
Still experimental - compute support is disabled by default
|
| |
|
|
|
| |
- Add GetTotalBytesReceived/GetTotalBytesSent to HttpServer with implementations in ASIO and http.sys backends
- Add ExpectedErrorCodes to HttpClientSettings to suppress warn/info logs for anticipated HTTP error codes
- Also fixes minor issues in `CprHttpClient::Download`
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* This branch adds full WebSocket (RFC 6455) support to the HTTP server layer, covering both transport backends, a client, and tests.
- **`websocket.h`** -- Core interfaces: `WebSocketOpcode`, `WebSocketMessage`, `WebSocketConnection` (ref-counted), and `IWebSocketHandler`. Services opt in to WebSocket support by implementing `IWebSocketHandler` alongside their existing `HttpService`.
- **`httpwsclient.h`** -- `HttpWsClient`: an ASIO-backed `ws://` client with both standalone (own thread) and shared `io_context` modes. Supports connect timeout and optional auth token injection via `IWsClientHandler` callbacks.
- **`wsasio.cpp/h`** -- `WsAsioConnection`: WebSocket over ASIO TCP. Takes over the socket after the HTTP 101 handshake and runs an async read/write loop with a queued write path (guarded by `RwLock`).
- **`wshttpsys.cpp/h`** -- `WsHttpSysConnection`: WebSocket over http.sys opaque-mode connections (Windows only). Uses `HttpReceiveRequestEntityBody` / `HttpSendResponseEntityBody` via IOCP, sharing the same threadpool as normal http.sys traffic. Self-ref lifetime management ensures graceful drain of outstanding async ops.
- **`httpsys_iocontext.h`** -- Tagged `OVERLAPPED` wrapper (`HttpSysIoContext`) used to distinguish normal HTTP transactions from WebSocket read/write completions in the single IOCP callback.
- **`wsframecodec.cpp/h`** -- `WsFrameCodec`: static helpers for parsing (unmasked and masked) and building (unmasked server frames and masked client frames) RFC 6455 frames across all three payload length encodings (7-bit, 16-bit, 64-bit). Also computes `Sec-WebSocket-Accept` keys.
- **`clients/httpwsclient.cpp`** -- `HttpWsClient::Impl`: ASIO-based client that performs the HTTP upgrade handshake, then hands off to the frame codec for the read loop. Manages its own `io_context` thread or plugs into an external one.
- **`httpasio.cpp`** -- ASIO server now detects `Upgrade: websocket` requests, checks the matching `HttpService` for `IWebSocketHandler` via `dynamic_cast`, performs the RFC 6455 handshake (101 response), and spins up a `WsAsioConnection`.
- **`httpsys.cpp`** -- Same upgrade detection and handshake logic for the http.sys backend, using `WsHttpSysConnection` and `HTTP_SEND_RESPONSE_FLAG_OPAQUE`.
- **`httpparser.cpp/h`** -- Extended to surface the `Upgrade` / `Connection` / `Sec-WebSocket-Key` headers needed by the handshake.
- **`httpcommon.h`** -- Minor additions (probably new header constants or response codes for the WS upgrade).
- **`httpserver.h`** -- Small interface changes to support WebSocket registration.
- **`zenhttp.cpp` / `xmake.lua`** -- New source files wired in; build config updated.
- **Unit tests** (`websocket.framecodec`): round-trip encode/decode for text, binary, close frames; all three payload sizes; masked and unmasked variants; RFC 6455 `Sec-WebSocket-Accept` test vector.
- **Integration tests** (`websocket.integration`): full ASIO server tests covering handshake (101), normal HTTP coexistence, echo, server-push broadcast, client close handshake, ping/pong auto-response, sequential messages, and rejection of upgrades on non-WS services.
- **Client tests** (`websocket.client`): `HttpWsClient` connect+echo+close, connection failure (bad port -> close code 1006), and server-initiated close.
* changed HttpRequestParser::ParseCurrentHeader to use switch instead of if/else chain
* remove spurious printf
---------
Co-authored-by: Stefan Boberg <[email protected]>
|
| |
|
|
|
|
|
| |
- zenhttp: added `GetServiceUri()`/`GetExternalHost()`
- enables code to quickly generate an externally reachable URI for a given service
- frontend: improved Uri handling (better defaults)
- added support for 404 page (to make it easier to find a good URL)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
zencore fixes:
- filesystem.cpp: ReadFile error reporting logic
- compactbinaryvalue.h: CbValue::As*String error reporting logic
zenhttp fixes:
- httpasio BindAcceptor would `return 0;` in a function returning `std::string` (UB)
- httpsys async workpool initialization race
zenstore fixes:
- cas.cpp: GetFileCasResults Results param passed by value instead of reference (large chunk results were silently lost)
- structuredcachestore.cpp: MissCount unconditionally incremented (counted hits as misses)
- cacherpc.cpp: Wrong boolean in Incomplete response array (all entries marked incomplete)
- cachedisklayer.cpp: sizeof(sizeof(...)) in two validation checks computed sizeof(size_t) instead of struct size
- buildstore.cpp: Wrong hash tracked in GC key list (BlobHash pushed twice instead of MetadataHash)
- buildstore.cpp: Removed duplicate m_LastAccessTimeUpdateCount increment in PutBlob
zenserver fixes:
- httpbuildstore.cpp: Reversed subtraction in HTTP range calculation (unsigned underflow)
- hubservice.cpp: Deadlock in Provision() calling Wake() while holding m_Lock (extracted WakeLocked helper)
- zipfs.cpp: Data race in GetFile() lazy initialization (added RwLock with shared/exclusive paths)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Added `--security-config-path` option to zenserver to configure security settings
- Expects a path to a .json file
- Default is an empty path resulting in no extra security settings and legacy behavior
- Current support is a top level filter of incoming http requests restricted to the `password` type
- `password` type will check the `Authorization` header and match it to the selected authorization strategy
- Currently the security settings is very basic and configured to a fixed username+password at startup
{
"http" {
"root": {
"filter": {
"type": "password",
"config": {
"password": {
"username": "<username>",
"password": "<password>"
},
"protect-machine-local-requests": false,
"unprotected-uris": [
"/health/",
"/health/info",
"/health/version"
]
}
}
}
}
}
|
| |
|
|
|
|
|
|
| |
* `RwLock::WithSharedLock` and `RwLock::WithExclusiveLock` can now return a value (which is returned by the passed function)
* Comma-separated logger specification now correctly deals with commas
* `GetSystemMetrics` properly accounts for cores
* cpr response formatter passes arguments in the right order
* `HttpServerRequest::SetLogRequest` can be used to selectively log HTTP requests
|
| |
|
|
|
| |
requests (#753)
* add IHttpRequestFilter to allow server implementation to filter/reject requests
|
| |
|
| |
* add IsLocalMachineRequest to HttpServerRequest
|
| |
|
| |
* add simple http client tests and fix run loop of http server to not rely on application quit
|
| |
|
|
|
|
| |
mode (#745)
* only check file size for requested range in asio TransmitFileAsync in debug mode
* only do Transmit file for whole files
|
| |\
| |
| | |
Run Windows service as an interactive session, so we correctly respond to RequestApplicationExit when requested to stop by the service manager
|
| | |
| |
| |
| | |
non-interactive sessions
|
| | |
| |
| |
| |
| | |
* implemented chunking for TransmitFile path, to ensure payloads exceeding the TransmitFile API limit of 4GB can be handled
* also fixes similar issue with memory path
|
| |/
|
|
| |
* make sure we properly convert command line args for zenserver as well
* make sure we *add* wildcards/excludes in addition to defaults
|
| |
|
|
|
|
|
| |
This change primarily introduces improved logic for dealing with sending data from file references.
This is intended to reduce the amount of memory-mapping we end up doing when sending data from files. Windows now uses `TransmitFile` to send file data more efficiently using kernel-side I/O, but Linux/Mac basically behaves as before since they don't offer any true async file I/O support via asio. This should be implemented separately using a background I/O thread pool.
This PR also includes improved memory management for http/asio with reduced allocation counts, and a fix for a potential use-after-free in very high load scenarios.
|
| |
|
| |
* changed some logging string so they don't get caught in CI logging
|
| |
|
|
|
|
|
|
| |
this change adds OTEL tracing to a few places
* Top-level application lifecycle (config/init/cleanup, main loop)
* http.sys requests
it also brings some otlptrace optimizations and dynamic configuration of tracing. OTLP tracing is currently always disabled
|
| |
|
|
| |
* refactored `HttpServer` so all subclass member functions are proctected, to make it easier to extend base functionality
* added API service, can be used to enumerate registered endpoints (at `/api`). Currently only very basic information is provided
|
| |
|
| |
* implement --dedicated option on asio http server
|
| |
|
| |
this change makes it possible to use zenserver on hosts where IPv6 has been disabled
|
| |
|
|
|
|
|
|
|
|
|
|
| |
effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower.
* added --corelimit option to zenserver
* made sure thread pools are configured lazily and not during global init
* added log output indicating effective and HW concurrency
* added change log entry
* removed debug logging from ZenEntryPoint::Run()
also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
|
| |
|
|
|
| |
- Change BadAlloc exceptions in GC to warnings
- Add explict ASSERT exception catch in http plugin request processing
- Make exceptions handled in http request processing to warnings
|
| |
|
|
|
| |
* clean up trace command line options
explicitly shut down worker pools
* some additional startup trace scopes
|
| |
|
|
|
| |
* extend log on failed httpsys response
* fix formatting for "Desired port is in use, retrying"
* add warning log if port is remapped
|
| |
|
|
| |
* Suppress progress report callback if oplog import detects oplog with zero ops
* output error code when catching system errors
|
| |
|
| |
This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
|
| |
|
|
| |
- Improvement: Add file and line to ASSERT exceptions
- Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
|
| |
|
| |
this change adds support for tracing http payloads when using the asio path. This was already supported when using the `--http=plugin` path and this change moves some code into a shared class for reuse.
|
| |
|
|
|
|
| |
- Shared memory for zenserver state may hang around after all zenserver processes exit - make sure we find a valid entry in `zen up` before bailing
- Httpasio add retry for desired port
- Httpasio only call listen() once
- Httpasio explicitly close acceptor sockets
|
| | |
|
| |
|
| |
- Bugfix: Implement two listening sockets in ASIO (ipv4+ipv6) when either we start with `--http-forceloopback` or we resort to that mode because of a failure to bind to the "any" address
|
| |
|
|
|
|
| |
* added log level control/query to LoggerRef
* added debug logging to http plugin implementation
* added GetDebugName() to transport plugin interfaces
* added debug name to log output
|
| |
|
|
| |
* New option for zenserver - `--http-forceloopback` which forces opening of the server http server using loopback (local) connection (UE-199776)
* add fallback to local connection for asio if we get access denied on public port
|
| | |
|
| |
|
|
|
|
|
|
|
| |
this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future.
As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link.
Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future.
The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
|
|
|
separating the http server implementations into a directory and moved diagsvcs into zenserver since it's somewhat hard-coded for it
|