aboutsummaryrefslogtreecommitdiff
path: root/src/zencore
Commit message (Collapse)AuthorAgeFilesLines
* implemented dynamic scaling support for WorkerThreadPoolStefan Boberg2026-02-233-22/+136
|
* add friends to fix build issueStefan Boberg2026-02-201-0/+3
|
* made http.sys async worker thread pool configurable via the same option as ↵Stefan Boberg2026-02-202-28/+45
| | | | | | the main I/O pool this is also to gain control over threads to improve shutdown behaviour
* fix MakeSafeAbsolutePathInPlace mis-spelling (#765)Stefan Boberg2026-02-202-21/+21
| | | | | (was MakeSafeAbsolutePathÍnPlace - note accent) Also fixed misleading comments on multiple functions in filesystem.h
* add selective request logging support to http.sys (#762)Stefan Boberg2026-02-182-36/+134
| | | | | * implemented selective request logging for http.sys for consistency with asio * fixed traversal of GetLogicalProcessorInformationEx to account for variable-sized records * also adds CPU usage metrics
* add http server root password protection (#757)Dan Engelbrecht2026-02-171-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `--security-config-path` option to zenserver to configure security settings - Expects a path to a .json file - Default is an empty path resulting in no extra security settings and legacy behavior - Current support is a top level filter of incoming http requests restricted to the `password` type - `password` type will check the `Authorization` header and match it to the selected authorization strategy - Currently the security settings is very basic and configured to a fixed username+password at startup { "http" { "root": { "filter": { "type": "password", "config": { "password": { "username": "<username>", "password": "<password>" }, "protect-machine-local-requests": false, "unprotected-uris": [ "/health/", "/health/info", "/health/version" ] } } } } }
* misc fixes brought over from sb/proto (#759)Stefan Boberg2026-02-173-11/+11
| | | | | | | | * `RwLock::WithSharedLock` and `RwLock::WithExclusiveLock` can now return a value (which is returned by the passed function) * Comma-separated logger specification now correctly deals with commas * `GetSystemMetrics` properly accounts for cores * cpr response formatter passes arguments in the right order * `HttpServerRequest::SetLogRequest` can be used to selectively log HTTP requests
* added ResetConsoleLog (#758)Stefan Boberg2026-02-162-0/+9
| | | also made sure log initialization calls it to ensure the console output format is retained even if the console logger was set up before logging is initialized
* ported optimizations of MeasureVarUInt (#747)Dan Engelbrecht2026-02-061-2/+12
|
* reduce batch size for reads (#740)Dan Engelbrecht2026-01-291-1/+1
| | | | | * reduce maximum size per chunk to read to reduce disk contention * increase timeout before warning on slow shut down of zenserver * reduce default window size for blockstore chunk iteration
* avoid big ioworker backlog (#733)Dan Engelbrecht2026-01-261-2/+14
| | | | | | * add ability to override scheduling mode in ParallelWork * Don't increase buffering size when copying from local cache with --boost-worker-memory enabled * Rework scheduling writes of downloaded data to reduce memory usage
* zenserver API changes, some other minor changes (#720)Stefan Boberg2026-01-191-1/+14
| | | | | | | * add system metrics output to top command * removed unnecessary xmake directives * file system API/comment tweaks * fixed out-of-range access in httpserver test * updated ZenServer base API to allow customization by mode
* remove ZENCORE_API completely (#718)Stefan Boberg2026-01-1919-418/+399
| | | initially we had ZENCORE_API macros to potentially allow for DLL linkage. It turns out that this is not useful and the macros just contribute noise, so this change removes them completely.
* consul package and basic client added (#716)Stefan Boberg2026-01-196-20/+239
| | | | | | | | * this adds a consul package which can be used to fetch a consul binary * it also adds a `ConsulProcess` helper which can be used to spawn and manage a consul service instance * zencore dependencies brought across: - `except_fmt.h` for easer generation of formatted exception messages - `process.h/cpp` changes (adds `Kill` operation and process group support on Windows) - `string.h` changes to allow generic use of `WideToUtf8()`
* use current locale when converting wide char command line arguments to ↵Dan Engelbrecht2026-01-151-0/+4
| | | | | | string (#712) * set utf8 locale, only set LC_CTYPE as we don't want to affect how decimal point or sorting orders are handled * set language/region explicitly so we can use LC_ALL
* added options to configure exclude folders and extensions to zen build ↵Dan Engelbrecht2026-01-134-32/+125
| | | | | | | | commands (#706) * added `--exclude-folders` to `zen upload`, `zen download` and `zen diff` added `--exclude-extensions` to `zen upload` and `zen diff` excluded folder names are now matched by folder name in subfolders in addition to root level folders * allow multiple token separators
* fix zero length stream decompress (#685)Dan Engelbrecht2025-12-101-1/+1
| | | * fix validation during stream decompress of zero-size rawsize
* catch exception leaks in windows thread pool (#677)Dan Engelbrecht2025-12-051-4/+18
| | | | * catch exception leaks in windows thread pool * make non-assert exception an error, assert is kept as warning as it is already reported
* use scope guards to secure that BasicFile::Detach is called even on ↵Dan Engelbrecht2025-12-011-10/+5
| | | | exception (#672)
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-273-21/+61
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* fix stream decompress (#668)Dan Engelbrecht2025-11-271-10/+14
| | | * fix source file offset in DecompressToStream implementations
* RawOffset can be anything and we expect an empty buffer to be returned along ↵Dan Engelbrecht2025-11-261-4/+19
| | | | with RawSize = 0 if the offset was out of bounds for the value. (#666)
* GetSystemMetrics for Mac,Linux (#660)Stefan Boberg2025-11-242-5/+151
| | | Implemented GetSystemMetrics function for Mac and Linux
* loose chunk filtering bug when using wildcards (#654)Dan Engelbrecht2025-11-182-12/+46
| | | | | | | | | | * fix filtering of loose chunks when downloading with a filter add tests * changelog * move InlineRemoveUnusedHashes * remove extra braces
* Change curl defaults on MacOS (#645)Stefan Boberg2025-11-111-3/+1
| | | | | | | * changed curl config to match the default from vcpkg (i.e `CURL_CA_FALLBACK=ON`) * disables use of Secure Transport for Mac since it's deprecated * Also worked around an issue (with `CURL_CA_BUNDLE`) where cross compiling curl on Mac would not configure curl in the same way as when compiling natively. This meant builds would not download on ARM macs when the CI build machine architecture was x86. The workaround should be redundant if we upgrade to 8.17 and use Apple SecTrust for cert validation. This should happen soon. * Also added various verbose logging to facilitate trouble shooting
* switch to xmake for package management (#611)Stefan Boberg2025-11-077-42/+257
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* get oplog attachments (#622)Dan Engelbrecht2025-11-072-0/+26
| | | * add support for downloading individual attachments from an oplog
* added MacOS-native IterateCommandlineArgs (#624)Stefan Boberg2025-11-031-1/+15
| | | fixes issue where command line arguments to control tracing would not work on Mac
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-019-39/+39
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* fix minor memory leak in command line parsing (#619)Stefan Boberg2025-10-291-0/+5
| | | when calling `getdelim` in a loop, it will return an allocated block of memory even if it is at EOF
* fix for Latch race (#617)Stefan Boberg2025-10-292-29/+19
| | | | | | Because the counter is decreased in `CountDown()` and subsequently checked against zero to determine if the completion event should be set: `Latch::Wait` checks the counter against zero on entry and would exit early, before waiting on the completion event. This could then lead to the `Latch` instance being torn down before the `CountDown()` does a `Set()` which could then lead to unexpected and unwanted things happening.
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-2810-18/+27
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* move cpr in-tree (#605)Stefan Boberg2025-10-241-0/+6
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* refactor CasContainerStrategy::IterateOneBlock to make it more readable (#607)Dan Engelbrecht2025-10-241-1/+2
|
* in-tree spdlog (#602)Stefan Boberg2025-10-241-2/+3
| | | | move spdlog into the tree to remove dependency on vcpkg::spdlog, to allow diverging from the official version and evolve it to fit better with OTLP logging requirements
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-227-1189/+234
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* fix error log when using TryCloneFile on Mac/Linux (#597)Dan Engelbrecht2025-10-221-3/+3
|
* updated chunking strategy (#589)Dan Engelbrecht2025-10-201-0/+132
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
* added separate xmake.lua for thirdparty (#578)Stefan Boberg2025-10-151-23/+7
| | | | | Moves out third-party stuff from zencore Establishes new pattern for incorporating thirdparty code. The integration is cleaner, clearer and also surfaces the code in the generated .sln
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-142-0/+30
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-132-12/+12
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* block reference couting copy support windows (#564)Dan Engelbrecht2025-10-112-32/+267
| | | | - Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during `zen builds download` operations - Enabled by default, disable with `--allow-file-clone=false`
* add ability to limit concurrency (#565)Stefan Boberg2025-10-102-2/+25
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* speed up tests (#555)Dan Engelbrecht2025-10-063-10/+17
| | | | | | | | | | | | * faster FileSystemTraversal test * faster jobqueue test * faster NamedEvent test * faster cache tests * faster basic http tests * faster blockstore test * faster cache store tests * faster compactcas tests * more responsive zenserver launch * tweak worker pool sizes in tests
* fix link error with operator new (#553)Stefan Boberg2025-10-042-23/+36
|
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-033-0/+346
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-292-0/+58
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* fixed race condition in zen::logging::Get (#519)Stefan Boberg2025-09-291-3/+12
| | | if two requests for the same logger came in from different threads they could end up creating the same logger twice which causes an exception on registration
* Make sure we call the previous terminate handle if present when we intercept ↵Dan Engelbrecht2025-09-265-17/+84
| | | | | | terminate calls (#514) Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
* new append op rpc method (#511)Dan Engelbrecht2025-09-263-1/+29
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 02:12:33 +0000