aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp
Commit message (Collapse)AuthorAgeFilesLines
* add `--boost-worker-memory` option to zen builds (#639)Dan Engelbrecht2025-11-102-6/+7
|
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-20/+2
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* add `--verbose-http` option to builds commands (#630)Dan Engelbrecht2025-11-042-0/+55
|
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-012-4/+14
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* move cpr in-tree (#605)Stefan Boberg2025-10-241-3/+1
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-223-2/+7
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* silence warnings in CprHttpClient if request was aborted (#591)Dan Engelbrecht2025-10-202-197/+219
| | | | * silence warnings in CprHttpClient if request was aborted * refactor CprHttpClient
* add ability to abort http requests (#586)Dan Engelbrecht2025-10-175-27/+76
| | | * add abort-check function to httpclient
* clean up http response formatters (#584)Dan Engelbrecht2025-10-163-50/+61
| | | * remove use of 'error:' in zen output which triggers Horde CI
* make asiohttp work without IPv6 (#562)Stefan Boberg2025-10-131-49/+225
| | | this change makes it possible to use zenserver on hosts where IPv6 has been disabled
* hide http.sys options when unavailable (#568)Stefan Boberg2025-10-131-4/+6
|
* add ability to limit concurrency (#565)Stefan Boberg2025-10-103-3/+3
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-062-5/+13
|
* fix for RPC replay issue (wrong content-type) (#536)Stefan Boberg2025-10-022-1/+23
| | | | | likely fall-out from HttpClient refactor. The content type used to be explicit via headers but is now taken from the `IoBuffer` also fixed issue which meant the original request session ID would also not be propagated as intended
* HttpClient support for pluggable back-ends (#532)Stefan Boberg2025-09-306-1634/+1942
| | | refactored HttpClient to separate out cpr implementation into separate classes, with an abstract base class to allow plugging in multiple implementations in the future
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-296-103/+210
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* fixed ParsePackageMessage error message when mismatched header magic is ↵Stefan Boberg2025-09-291-1/+1
| | | | detected (#521)
* Added `--oidctoken-exe-unattended` to`zen builds` and `zen oplog-download` ↵Dan Engelbrecht2025-09-222-3/+5
| | | | command to use unattended mode when launching oidc-token.exe (#506)
* revise exception vs error (#495)Dan Engelbrecht2025-09-153-10/+18
| | | | | - Change BadAlloc exceptions in GC to warnings - Add explict ASSERT exception catch in http plugin request processing - Make exceptions handled in http request processing to warnings
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-102-13/+15
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* asio large header support (#484)Dan Engelbrecht2025-09-042-123/+128
| | | - Improvement: Allow large headers support in asio server implementation
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-044-46/+103
| | | * add validation of compact binary payloads before reading them
* clean up trace options parsing (#473)Dan Engelbrecht2025-08-225-0/+24
| | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-122-3/+27
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* list build part content (#462)Dan Engelbrecht2025-08-112-6/+20
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* de/stringbuilder safety (#456)Dan Engelbrecht2025-08-051-1/+1
| | | | | | - Improvement: Safeguard FormatCallstack to not throw exceptions when building the callstack string - Improvement: Limit thread name length when setting it for debugger use - Improvemnet: Don't allow assert callbacks to throw exception - Improvement: When formatting log output for malformed attachments in a package message, allow the string buffer to grow instead of throwing exception
* crash in composite buffer stream (#431)Dan Engelbrecht2025-06-101-1/+1
| | | * fix BufferedReadFileStream calculating buffer end wrong
* use unique tmp name for auth token file (#426)Dan Engelbrecht2025-06-091-3/+3
| | | * use Oid to generate unique name since std::tmpnam is not good practice
* http client streaming upload (#413)Dan Engelbrecht2025-06-021-27/+283
| | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* handle exception with batch work (#401)Dan Engelbrecht2025-05-191-2/+4
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* extend log on failed httpsys response (#394)Dan Engelbrecht2025-05-132-8/+22
| | | | | * extend log on failed httpsys response * fix formatting for "Desired port is in use, retrying" * add warning log if port is remapped
* close servers in multi server (#371)Dmytro Ivanov2025-04-241-0/+4
|
* add retry on internal error / bad gateway (#370)Dan Engelbrecht2025-04-241-1/+3
| | | * do http client retry on internal error and bad gateway
* Make plugin loading errors non fatal (#364)Dmytro Ivanov2025-04-233-29/+46
| | | | make plugin loading errors non fatal
* review fixesDmytro Ivanov2025-04-221-1/+1
|
* Added config, versioning and logging for pluginsDmytro Ivanov2025-04-223-106/+167
|
* filesystem retry fixes (#354)Dan Engelbrecht2025-04-101-5/+5
| | | | | | | | | * add more forgiving retries with filesystem * fall back to FindFirstFile if access prevents us from using GetFileAttributes * only validate hash if we have a complete payload in http client * changelog
* use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-023-1/+113
| | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
* long filename support (#330)Dan Engelbrecht2025-03-311-1/+1
| | | - Bugfix: Long file paths now works correctly on Windows
* optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-1/+1
| | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* zen build cache service (#318)Dan Engelbrecht2025-03-263-47/+70
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* build list filters (#313)Dan Engelbrecht2025-03-192-3/+3
| | | | | - Feature: `zen builds list` command has new options - `--query-path` - path to a .json (json format) or .cbo (compact binary object format) with the search query to use - `--result-path` - path to a .json (json format) or .cbo (compact binary object format) to write output result to, if omitted json format will be output to console
* Build command tweaks (#301)Stefan Boberg2025-03-111-7/+10
| | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* partial block fetch (#298)Dan Engelbrecht2025-03-071-66/+95
| | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-064-18/+38
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* limit and validate responses before logging the text (#292)Stefan Boberg2025-03-042-1/+93
| | | Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
* refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-031-0/+5
| | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-4/+23
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* improved builds api interface in jupiter (#281)Dan Engelbrecht2025-02-123-140/+178
| | | | | * multipart upload/download iterface in jupiter * review fixes
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-222-0/+105
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-19 10:46:18 +0000