aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/main.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Transparent proxy mode (#823)Stefan Boberg5 days1-0/+9
| | | | | | | | | | | | | | | | | Adds a **transparent TCP proxy mode** to zenserver (activated via `zenserver proxy`), allowing it to sit between clients and upstream Zen servers to inspect and monitor HTTP/1.x traffic in real time. Primarily useful during development, to be able to observe multi-server/client interactions in one place. - **Dedicated proxy port** -- Proxy mode defaults to port 8118 with its own data directory to avoid collisions with a normal zenserver instance. - **TCP proxy core** (`src/zenserver/proxy/`) -- A new transparent TCP proxy that forwards connections to upstream targets, with support for both TCP/IP and Unix socket listeners. Multi-threaded I/O for connection handling. Supports Unix domain sockets for both upstream/downstream. - **HTTP traffic inspection** -- Parses HTTP/1.x request/response streams inline to extract method, path, status, content length, and WebSocket upgrades without breaking the proxied data. - **Proxy dashboard** -- A web UI showing live connection stats, per-target request counts, active connections, bytes transferred, and client IP/session ID rollups. - **Server mode display** -- Dashboard banner now shows the running server mode (Zen Proxy, Zen Compute, etc.). Supporting changes included in this branch: - **Wildcard log level matching** -- Log levels can now be set per-category using wildcard patterns (e.g. `proxy.*=debug`). - **`zen down --all`** -- New flag to shut down all running zenserver instances; also used by the new `xmake kill` task. - Minor test stability fixes (flaky hash collisions, per-thread RNG seeds). - Support ZEN_MALLOC environment variable for default allocator selection and switch default to rpmalloc - Fixed sentry-native build to allow LTO on Windows
* Eliminate spdlog dependency (#773)Stefan Boberg8 days1-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes the vendored spdlog library (~12,000 lines) and replaces it with a purpose-built logging system in zencore (~1,800 lines). The new implementation provides the same functionality with fewer abstractions, no shared_ptr overhead, and full control over the logging pipeline. ### What changed **New logging core in zencore/logging/:** - LogMessage, Formatter, Sink, Logger, Registry - core abstractions matching spdlog's model but simplified - AnsiColorStdoutSink - ANSI color console output (replaces spdlog stdout_color_sink) - MsvcSink - OutputDebugString on Windows (replaces spdlog msvc_sink) - AsyncSink - async logging via BlockingQueue worker thread (replaces spdlog async_logger) - NullSink, MessageOnlyFormatter - utility types - Thread-safe timestamp caching in formatters using RwLock **Moved to zenutil/logging/:** - FullFormatter - full log formatting with timestamp, logger name, level, source location, multiline alignment - JsonFormatter - structured JSON log output - RotatingFileSink - rotating file sink with atomic size tracking **API changes:** - Log levels are now an enum (LogLevel) instead of int, eliminating the zen::logging::level namespace - LoggerRef no longer wraps shared_ptr - it holds a raw pointer with the registry owning lifetime - Logger error handler is wired through Registry and propagated to all loggers on registration - Logger::Log() now populates ThreadId on every message **Cleanup:** - Deleted thirdparty/spdlog/ entirely (110+ files) - Deleted full_test_formatter (was ~80% duplicate of FullFormatter) - Renamed snake_case classes to PascalCase (full_formatter -> FullFormatter, json_formatter -> JsonFormatter, sentry_sink -> SentrySink) - Removed spdlog from xmake dependency graph ### Build / test impact - zencore no longer depends on spdlog - zenutil and zenvfs xmake.lua updated to drop spdlog dep - zentelemetry xmake.lua updated to drop spdlog dep - All existing tests pass, no test changes required beyond formatter class renames
* added `--verbose` option to zenserver-test and `xmake test` (#798)Stefan Boberg2026-03-011-1/+0
| | | | | | * when `--verbose` is specified to zenserver-test, all child process output (typically, zenserver instances) is piped through to stdout. you can also pass `--verbose` to `xmake test` to accomplish the same thing. * this PR also consolidates all test runner `main` function logic (such as from zencore-test, zenhttp-test etc) into central implementation in zencore for consistency and ease of maintenance * also added extended utf8-tests including a fix to `Utf8ToWide()`
* work around doctest shutdown issues with static CRT (#784)Stefan Boberg2026-02-251-0/+9
| | | | | | * tweaked doctest.h to avoid shutdown issues due to thread_local variables running destructors after the main thread has torn down everything including the heap * disabled zenserver exit thread waiting since doctest should hopefully not be causing issues during shutdown anymore after my workaround This should help reduce the duration of tests spawning lots of server instances
* structured compute basics (#714)Stefan Boberg2026-02-181-5/+50
| | | | | | | | | this change adds the `zencompute` component, which can be used to distribute work dispatched from UE using the DDB (Derived Data Build) APIs via zenserver this change also adds a distinct zenserver compute mode (`zenserver compute`) which is intended to be used for leaf compute nodes to exercise the compute functionality without directly involving UE, a `zen exec` subcommand is also added, which can be used to feed replays through the system all new functionality is considered *experimental* and disabled by default at this time, behind the `zencompute` option in xmake config
* logging config move to zenutil (#754)Stefan Boberg2026-02-131-1/+1
| | | made logging config options from zenserver available in zen CLI
* hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-221-2/+5
| | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
* zen hub (#574)Stefan Boberg2026-01-211-0/+4
| | | | | Initial implementation of zenserver "hub" mode. This is an experimental feature. zenserver can be started in hub mode by specifying `hub` as the first argument to zenserver
* zenserver API changes, some other minor changes (#720)Stefan Boberg2026-01-191-13/+41
| | | | | | | * add system metrics output to top command * removed unnecessary xmake directives * file system API/comment tweaks * fixed out-of-range access in httpserver test * updated ZenServer base API to allow customization by mode
* use current locale when converting wide char command line arguments to ↵Dan Engelbrecht2026-01-151-0/+8
| | | | | | string (#712) * set utf8 locale, only set LC_CTYPE as we don't want to affect how decimal point or sorting orders are handled * set language/region explicitly so we can use LC_ALL
* add otel instrumentation (#581)Stefan Boberg2025-12-111-0/+3
| | | | | | | | this change adds OTEL tracing to a few places * Top-level application lifecycle (config/init/cleanup, main loop) * http.sys requests it also brings some otlptrace optimizations and dynamic configuration of tracing. OTLP tracing is currently always disabled
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-011-2/+2
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* restructured zenserver configuration (#575)Stefan Boberg2025-10-151-47/+62
| | | this breaks out the configuration logic to allow multiple applications to share common configuration and initialization logic whilst customizing chosen aspects of the process
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-141-13/+7
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* split storage config from base config (#570)Stefan Boberg2025-10-131-1/+1
| | | | | this change splits the configuration logic into a base part and a storage server part manually tested with shared server config file
* extract storage server into separate source files (#569)Stefan Boberg2025-10-131-342/+5
| | | | this change moves common base service code into `zenserver.(cpp|h)` and storage server code into `zenstorageserver.(cpp|h)` to more clearly separate concerns but also to make way for another service
* move service common code into base class (#567)Stefan Boberg2025-10-131-102/+139
| | | | | Separates storage server code and generic server code into two classes. This is a change to prepare for different services to be implemented using the same framework, into the same executable
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-1/+8
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* zenutil cleanup (#550)Dan Engelbrecht2025-10-031-1/+1
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-261-24/+33
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* improve console output (#476)Dan Engelbrecht2025-08-261-2/+2
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
* Merge pull request #139 from ue-foundation/de/zen-service-commandLiam Mitchell2025-08-221-1/+21
|\ | | | | zen service command
| * Move windows service utilities to zenutil and fix clang-format errorsLiam Mitchell2025-08-221-2/+1
| |
| * Move ReportServiceStatus to zenutil and remove extraneous loggingLiam Mitchell2025-08-221-50/+0
| |
| * Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-211-7/+10
| |\
| * | Ignore unused variable warnings on platforms where ReportServiceStatus is ↵Liam Mitchell2025-08-211-0/+1
| | | | | | | | | | | | unimplemented
| * | Add pre-built systemd library and remove vcpkg dependencyLiam Mitchell2025-08-201-0/+2
| | |
| * | Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-291-6/+12
| |\ \
| * | | Update Linux service type and add libsystemd dependencyLiam Mitchell2025-03-051-0/+4
| | | |
| * | | Initial implementation of service status reportingLiam Mitchell2025-03-041-0/+45
| | | |
| * | | Implementation of service commands for Linux.Liam Mitchell2025-02-271-0/+4
| | | |
| * | | clang formatDan Engelbrecht2025-01-151-1/+4
| | | |
| * | | systemd unit file, incompleteDan Engelbrecht2025-01-151-0/+13
| | | |
* | | | clean up trace options parsing (#473)Dan Engelbrecht2025-08-221-24/+12
| |_|/ |/| | | | | | | | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* | | Set ready event when reusing an existing server instance. (#448)Martin Ridgers2025-08-041-6/+10
| |/ |/| | | | | | | * Moved firing of child-id event into a lambda * Fire notify event to parent after attaching to an existing instance * Updated changelog
* | sentry config (#430)Dan Engelbrecht2025-06-121-5/+7
| | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
* | revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-7/+5
| |
* | add sentry configurations options for debug/environmentDan Engelbrecht2025-06-101-5/+7
| | | | | | | | | | add env-variable parsing for sentry option
* | `--sentry-dsn` option for zen command line and zenserver to control Sentry ↵Dan Engelbrecht2025-06-091-1/+5
| | | | | | | | | | reporting endpoint (#427) moved sentry database path to temporary directory for zen commandline
* | add sentry for zen command (#373)Dan Engelbrecht2025-05-061-1/+1
| | | | | | | | * refactor sentry integration and add to zen command line tool * move add_ldflags("-framework Security")
* | long filename support (#330)Dan Engelbrecht2025-03-311-3/+3
|/ | | - Bugfix: Long file paths now works correctly on Windows
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+11
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* Insights-compatible memory tracking (#214)Stefan Boberg2024-11-251-10/+14
| | | | | | | | | | | | | This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future. To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks. This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`: * `mimalloc` - mimalloc (default, same as before) * `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows. * `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows. * `ansi` - fallback to default system allocator
* sponsor process attach hardening (#208)Dan Engelbrecht2024-11-041-0/+1
| | | * make sure to clear sponsor slot if pickup does not happen
* safer calls to IsProcessRunning (#131)Dan Engelbrecht2024-08-221-5/+20
| | | * safer calls to IsProcessRunning to handle cases where we can't check status of processes
* if a zenserver is already using our named mutex - exit with error code ↵Dan Engelbrecht2024-08-221-0/+5
| | | | instead of reporting error to Sentry (#132)
* added `--detach` option to zenserver (#115)Stefan Boberg2024-08-141-5/+8
| | | | | added `--detach` option to zenserver. When this is passed in with a false value, we do not create a new process group in order to behave more as expected when running with `xmake run zenserver`. Without this change the zenserver process does not receive any signals and won't exit when xmake does, causing processes to linger in the background. The default behaviour (when run from UE) is unchanged.
* improve logging on main failure (#111)Dan Engelbrecht2024-08-141-2/+10
| | | | | | * add support for indenting callstack output * Explicitly catch option-parse error and reduce log spam on bad parameters * add command line to sentry error reports * log command line at startup
* don't use "error:" in log messages unless there is an error (#87)Dan Engelbrecht2024-05-271-2/+2
| | | Improvement: Don't use "error:" in log messages unless there is an error as Horde CI will pick up that log line and interpret it as an error
* import oplog improvements (#54)Dan Engelbrecht2024-04-201-2/+3
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-17 12:06:45 +0000