aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp
Commit message (Collapse)AuthorAgeFilesLines
...
* OTLP/trace improvements (#717)Stefan Boberg2026-01-191-4/+6
| | | | | | | This PR brings over some changes made to avoid performing setup for otel instrumentation if we are not sending otel information anywhere anyway. It also adds the ability to configure an OTLP endpoint on the command line using `--otlp-endpoint=<URI>`. Bear in mind that OTLP support is still not officially supported so this should not be used in production at this stage.
* asio/http optimizations (#449)Stefan Boberg2026-01-141-104/+638
| | | | | | | This change primarily introduces improved logic for dealing with sending data from file references. This is intended to reduce the amount of memory-mapping we end up doing when sending data from files. Windows now uses `TransmitFile` to send file data more efficiently using kernel-side I/O, but Linux/Mac basically behaves as before since they don't offer any true async file I/O support via asio. This should be implemented separately using a background I/O thread pool. This PR also includes improved memory management for http/asio with reduced allocation counts, and a fix for a potential use-after-free in very high load scenarios.
* CprHttpClient cleanup (#703)Dan Engelbrecht2026-01-092-7/+3
|
* Add base port getter and set the dll directory for plugin dependencies (#692)Tomasz Obrębski2025-12-192-7/+20
| | | | | | | * Add base port getter and set the dll directory for plugin dependencies * Use UTF8 for dll paths * Rename BasePort to m_BasePort for consistency
* remove error warning in output (#695)Dan Engelbrecht2025-12-172-9/+9
| | | * changed some logging string so they don't get caught in CI logging
* add otel instrumentation (#581)Stefan Boberg2025-12-115-21/+141
| | | | | | | | this change adds OTEL tracing to a few places * Top-level application lifecycle (config/init/cleanup, main loop) * http.sys requests it also brings some otlptrace optimizations and dynamic configuration of tracing. OTLP tracing is currently always disabled
* HTTP server API changes for improved extensibility (#684)Stefan Boberg2025-12-1111-55/+177
| | | | * refactored `HttpServer` so all subclass member functions are proctected, to make it easier to extend base functionality * added API service, can be used to enumerate registered endpoints (at `/api`). Currently only very basic information is provided
* fix reading of stale iterator when doing download-resume (#681)Dan Engelbrecht2025-12-091-9/+9
|
* implement --dedicated option on asio http server (#679)Dan Engelbrecht2025-12-054-45/+101
| | | * implement --dedicated option on asio http server
* fix crash when parsing empty key in httpstats service (#671)Dan Engelbrecht2025-12-011-2/+1
|
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-271-1/+1
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* Add regex-free route matching support (#662)Stefan Boberg2025-11-242-50/+474
| | | | | This change adds support for non-regex matching of routes. Instead of using regex patterns you can associate matcher functions with pattern names and string literal components are identified and matched directly. Also implemented tests for `HttpRequestRouter` class.
* add `--boost-worker-memory` option to zen builds (#639)Dan Engelbrecht2025-11-102-6/+7
|
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-20/+2
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* add `--verbose-http` option to builds commands (#630)Dan Engelbrecht2025-11-042-0/+55
|
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-012-4/+14
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* move cpr in-tree (#605)Stefan Boberg2025-10-241-3/+1
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-223-2/+7
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* silence warnings in CprHttpClient if request was aborted (#591)Dan Engelbrecht2025-10-202-197/+219
| | | | * silence warnings in CprHttpClient if request was aborted * refactor CprHttpClient
* add ability to abort http requests (#586)Dan Engelbrecht2025-10-175-27/+76
| | | * add abort-check function to httpclient
* clean up http response formatters (#584)Dan Engelbrecht2025-10-163-50/+61
| | | * remove use of 'error:' in zen output which triggers Horde CI
* make asiohttp work without IPv6 (#562)Stefan Boberg2025-10-131-49/+225
| | | this change makes it possible to use zenserver on hosts where IPv6 has been disabled
* hide http.sys options when unavailable (#568)Stefan Boberg2025-10-131-4/+6
|
* add ability to limit concurrency (#565)Stefan Boberg2025-10-103-3/+3
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-062-5/+13
|
* fix for RPC replay issue (wrong content-type) (#536)Stefan Boberg2025-10-022-1/+23
| | | | | likely fall-out from HttpClient refactor. The content type used to be explicit via headers but is now taken from the `IoBuffer` also fixed issue which meant the original request session ID would also not be propagated as intended
* HttpClient support for pluggable back-ends (#532)Stefan Boberg2025-09-306-1634/+1942
| | | refactored HttpClient to separate out cpr implementation into separate classes, with an abstract base class to allow plugging in multiple implementations in the future
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-296-103/+210
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* fixed ParsePackageMessage error message when mismatched header magic is ↵Stefan Boberg2025-09-291-1/+1
| | | | detected (#521)
* Added `--oidctoken-exe-unattended` to`zen builds` and `zen oplog-download` ↵Dan Engelbrecht2025-09-222-3/+5
| | | | command to use unattended mode when launching oidc-token.exe (#506)
* revise exception vs error (#495)Dan Engelbrecht2025-09-153-10/+18
| | | | | - Change BadAlloc exceptions in GC to warnings - Add explict ASSERT exception catch in http plugin request processing - Make exceptions handled in http request processing to warnings
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-102-13/+15
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* asio large header support (#484)Dan Engelbrecht2025-09-042-123/+128
| | | - Improvement: Allow large headers support in asio server implementation
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-044-46/+103
| | | * add validation of compact binary payloads before reading them
* clean up trace options parsing (#473)Dan Engelbrecht2025-08-225-0/+24
| | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-122-3/+27
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* list build part content (#462)Dan Engelbrecht2025-08-112-6/+20
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* de/stringbuilder safety (#456)Dan Engelbrecht2025-08-051-1/+1
| | | | | | - Improvement: Safeguard FormatCallstack to not throw exceptions when building the callstack string - Improvement: Limit thread name length when setting it for debugger use - Improvemnet: Don't allow assert callbacks to throw exception - Improvement: When formatting log output for malformed attachments in a package message, allow the string buffer to grow instead of throwing exception
* crash in composite buffer stream (#431)Dan Engelbrecht2025-06-101-1/+1
| | | * fix BufferedReadFileStream calculating buffer end wrong
* use unique tmp name for auth token file (#426)Dan Engelbrecht2025-06-091-3/+3
| | | * use Oid to generate unique name since std::tmpnam is not good practice
* http client streaming upload (#413)Dan Engelbrecht2025-06-021-27/+283
| | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* handle exception with batch work (#401)Dan Engelbrecht2025-05-191-2/+4
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* extend log on failed httpsys response (#394)Dan Engelbrecht2025-05-132-8/+22
| | | | | * extend log on failed httpsys response * fix formatting for "Desired port is in use, retrying" * add warning log if port is remapped
* close servers in multi server (#371)Dmytro Ivanov2025-04-241-0/+4
|
* add retry on internal error / bad gateway (#370)Dan Engelbrecht2025-04-241-1/+3
| | | * do http client retry on internal error and bad gateway
* Make plugin loading errors non fatal (#364)Dmytro Ivanov2025-04-233-29/+46
| | | | make plugin loading errors non fatal
* review fixesDmytro Ivanov2025-04-221-1/+1
|
* Added config, versioning and logging for pluginsDmytro Ivanov2025-04-223-106/+167
|
* filesystem retry fixes (#354)Dan Engelbrecht2025-04-101-5/+5
| | | | | | | | | * add more forgiving retries with filesystem * fall back to FindFirstFile if access prevents us from using GetFileAttributes * only validate hash if we have a complete payload in http client * changelog
* use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-023-1/+113
| | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-20 09:55:19 +0000