aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
* Include version string on the dashboard's start page. (#651)Martin Ridgers2025-11-174-2/+26
| | | | | | | | | | | * Method to get plain text from an async request * Include server's version on the dashboard start page * Same paragraph style as the rest of the method * Updated changelog * Update frontend archive
* control cache upload (#646)Dan Engelbrecht2025-11-121-0/+2
| | | * add option to enable/disable upload to builds cache
* switch to xmake for package management (#611)Stefan Boberg2025-11-074-10/+55
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-018-34/+35
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* make sure OpenProcessCache is initialized before use (#618)Stefan Boberg2025-10-291-4/+4
| | | previously, a null reference would be passed into ProjectStore constructor
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-282-2/+4
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* move cpr in-tree (#605)Stefan Boberg2025-10-241-2/+1
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* add host discovery and zen cache support for oplog import (#601)Dan Engelbrecht2025-10-231-13/+29
| | | * add host discovery and zen cache support for oplog import
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-2210-6/+168
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* Mr/dashboard stats summary tweak (#592)Martin Ridgers2025-10-202-9/+14
| | | | | | | * More robust dashboard stats summary * Updated changelog * Updated frontend zip archive
* fix builds storage stats (#590)Dan Engelbrecht2025-10-201-18/+22
| | | * restructure builds storage stats to match web-ui expectations
* restructured zenserver configuration (#575)Stefan Boberg2025-10-1513-210/+317
| | | this breaks out the configuration logic to allow multiple applications to share common configuration and initialization logic whilst customizing chosen aspects of the process
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-1432-44/+31
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* split storage config from base config (#570)Stefan Boberg2025-10-136-1208/+1355
| | | | | this change splits the configuration logic into a base part and a storage server part manually tested with shared server config file
* extract storage server into separate source files (#569)Stefan Boberg2025-10-135-1185/+1236
| | | | this change moves common base service code into `zenserver.(cpp|h)` and storage server code into `zenstorageserver.(cpp|h)` to more clearly separate concerns but also to make way for another service
* move service common code into base class (#567)Stefan Boberg2025-10-135-499/+626
| | | | | Separates storage server code and generic server code into two classes. This is a change to prepare for different services to be implemented using the same framework, into the same executable
* hide http.sys options when unavailable (#568)Stefan Boberg2025-10-132-3/+10
|
* add ability to limit concurrency (#565)Stefan Boberg2025-10-105-2/+13
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* shorten thread pool names for Linux which has a limit of 15 characters (#563)Stefan Boberg2025-10-081-1/+1
| | | without this change we just get a whole bunch of threads named "main" or "zenserver" depending on timing
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-061-2/+2
|
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-031-1/+1
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-033-15/+5
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-031-1/+1
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-036-617/+16
| | | * move zen vfs implementation to zenstore
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-034-8/+26
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* cache RPC replay fixes (minor) (#544)Stefan Boberg2025-10-031-1/+1
| | | | | * fixes misleading server side log message when using `--onhost` * fixed `--onhost` behaviour in zen (no longer replays twice)
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-0313-5436/+6
| | | * move remoteproject code to remotestorelib
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-0210-8831/+5
|
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-028-1681/+1524
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* Zs/OIDC exe path handling (#538)Zousar Shaker2025-10-021-4/+10
| | | | | * Fix handling of oidc-exe-path The path was only being set if the file DIDN'T exist on the server (along with the warning about it not existing). Now set if it DOES exist, and the warning emitted if it DOESN'T.
* Fix bug in ReplayLogEntries for large op blobs (#537)Matt Peters2025-10-011-0/+1
| | | | | - Bugfix: Fixed issue where large ops in oplog would fail to read properly ReplayLogEntries: Fix bug in the case where an op is too big for OpBl…obsBuffer. The OpBufferView needs to be pointed to the temporary buffer.
* skip dropped oplogs in ScanForOplogs() (#534)Dan Engelbrecht2025-09-301-0/+5
|
* projectstore refactor (#531)Dan Engelbrecht2025-09-304-721/+667
| | | | | | | | | | | | | | | | | * convert ProjectStore::GetProjectFiles to not use http return codes * convert ProjectStore::GetProjectChunkInfos to not use http return codes * convert ProjectStore::GetChunkInfo to not use http return codes * convert ProjectStore::GetChunkRange to not use http return codes * convert ProjectStore::GetChunk to not use http return codes * convert ProjectStore::PutChunk to not use http return codes * convert ProjectStore::WriteOplog to not use http return codes * convert ProjectStore::ReadOplog to not use http return codes
* gracefully handle missing chunks when exporting an oplog (#526)Dan Engelbrecht2025-09-292-28/+66
| | | | - Improvement: Gracefully handle missing chunks when exporting an oplog - Bugfix: Add object validation when replaying oplog to avoid crash on malformed data
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-295-185/+92
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* some bug fixes (#522)Stefan Boberg2025-09-291-2/+2
| | | | | | * fix for invalid regex in HttpBuildStoreService - triggers with most recent MSVC version * in GcScheduler don't wait for exit signal if exit has already been requested. this caused extended waits for shutdown in some automated tests on very fast machines, possibly also due to some behaviour change in condition_variable * speculative fix/workaround for issue with TLS teardown on secondary thread while main was tearing down trace
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-227/+375
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge pull request #509 from ue-foundation/zs/put-overwrite-policy-responseZousar Shaker2025-09-251-5/+10
|\ | | | | Zs/put overwrite policy response
| * Report Incomplete Records To Clientzousar2025-09-241-3/+5
| | | | | | | | When requesting partial records, report back when a record is incomplete via an "Incomplete" array of bools that is a sibling to the "Result" array for batch/rpc operations, or via the HttpResponseCode::PartialContent status code for individual record requests.
| * Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-226-10/+40
| |\
| * | Change batch put responses for client reportingzousar2025-09-191-2/+5
| | | | | | | | | | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* | | limit number of chunks per block (#508)Dan Engelbrecht2025-09-233-49/+77
| |/ |/| | | | | | | - Improvement: Chunk block generation for `zen oplog-export` and `zen builds upload` command are now limited to max 4000 chunks per block zen builds list-block testing function to check results for recently used blocks in a context
* | dont set Size or RawSize when fetching chunkinfos if the payload can't be ↵Dan Engelbrecht2025-09-221-2/+7
| | | | | | | | found/validated (#507)
* | more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-221-0/+18
| | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
* | Added `--oidctoken-exe-unattended` to`zen builds` and `zen oplog-download` ↵Dan Engelbrecht2025-09-225-8/+15
|/ | | | command to use unattended mode when launching oidc-token.exe (#506)
* Sorting oplog tree view by size would raise an error. (#497)Martin Ridgers2025-09-162-1/+1
| | | | | | | * Sorting oplog tree view by size would raise an error * Updated frontend .zip archive * Updated changelog
* new in-memory storage strategy for oplogs (#490)Dan Engelbrecht2025-09-154-518/+748
| | | - Improvement: Revised project oplog in-memory representation which reduces load times and memory usage
* initialize all services before healthy (#493)Dan Engelbrecht2025-09-121-35/+58
| | | * make sure all services are initialized and registered before we respond OK to health check
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-104-771/+805
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* faster oplog entries with referenceset (#488)Dan Engelbrecht2025-09-085-56/+86
| | | | - Improvement: Faster project store `/entries` endpoint, 10-15% faster when using a reference set to limit entries fix missing space after task name in pretty progressbar
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-22 06:31:34 +0000