| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
This change introduces job object support on Windows to be able to more accurately track and limit resource usage on storage instances created by the hub service. It also ensures that all child instances can be torn down reliably on exit.
Also made it so hub tests no longer pop up console windows while running.
|
| |
|
| |
* Ported "lane trace" feature from UE (by way of IAX)
|
| |
|
|
|
|
|
| |
- Add missing includes in hashutils.h (`<cstddef>`, `<type_traits>`)
- Add `ZenContentType` parameter to all `IoBufferBuilder` factory methods so content type is set at buffer creation time
- Fix null dereference in `SharedBuffer::GetFileReference()` when buffer is null
- Fix out-of-bounds read in trace command-line argument parsing when arg length exactly matches option length
- Add unit tests for 32-bit `CountLeadingZeros`
|
| |
|
|
|
| |
Allows user to automate launching of zenserver dashboard, including when multiple instances are running. If multiple instances are running you can open all dashboards with `--all`, and also using the in-terminal chooser which also allows you to open a specific instance.
Also includes a fix to `zen exec` when using offset/stride/limit
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
zencore fixes:
- filesystem.cpp: ReadFile error reporting logic
- compactbinaryvalue.h: CbValue::As*String error reporting logic
zenhttp fixes:
- httpasio BindAcceptor would `return 0;` in a function returning `std::string` (UB)
- httpsys async workpool initialization race
zenstore fixes:
- cas.cpp: GetFileCasResults Results param passed by value instead of reference (large chunk results were silently lost)
- structuredcachestore.cpp: MissCount unconditionally incremented (counted hits as misses)
- cacherpc.cpp: Wrong boolean in Incomplete response array (all entries marked incomplete)
- cachedisklayer.cpp: sizeof(sizeof(...)) in two validation checks computed sizeof(size_t) instead of struct size
- buildstore.cpp: Wrong hash tracked in GC key list (BlobHash pushed twice instead of MetadataHash)
- buildstore.cpp: Removed duplicate m_LastAccessTimeUpdateCount increment in PutBlob
zenserver fixes:
- httpbuildstore.cpp: Reversed subtraction in HTTP range calculation (unsigned underflow)
- hubservice.cpp: Deadlock in Provision() calling Wake() while holding m_Lock (extracted WakeLocked helper)
- zipfs.cpp: Data race in GetFile() lazy initialization (added RwLock with shared/exclusive paths)
|
| |
|
| |
Co-authored-by: Stefan Boberg <[email protected]>
|
| |
|
|
|
| |
(was MakeSafeAbsolutePathÍnPlace - note accent)
Also fixed misleading comments on multiple functions in filesystem.h
|
| |
|
|
|
| |
* implemented selective request logging for http.sys for consistency with asio
* fixed traversal of GetLogicalProcessorInformationEx to account for variable-sized records
* also adds CPU usage metrics
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Added `--security-config-path` option to zenserver to configure security settings
- Expects a path to a .json file
- Default is an empty path resulting in no extra security settings and legacy behavior
- Current support is a top level filter of incoming http requests restricted to the `password` type
- `password` type will check the `Authorization` header and match it to the selected authorization strategy
- Currently the security settings is very basic and configured to a fixed username+password at startup
{
"http" {
"root": {
"filter": {
"type": "password",
"config": {
"password": {
"username": "<username>",
"password": "<password>"
},
"protect-machine-local-requests": false,
"unprotected-uris": [
"/health/",
"/health/info",
"/health/version"
]
}
}
}
}
}
|
| |
|
|
|
|
|
|
| |
* `RwLock::WithSharedLock` and `RwLock::WithExclusiveLock` can now return a value (which is returned by the passed function)
* Comma-separated logger specification now correctly deals with commas
* `GetSystemMetrics` properly accounts for cores
* cpr response formatter passes arguments in the right order
* `HttpServerRequest::SetLogRequest` can be used to selectively log HTTP requests
|
| |
|
| |
also made sure log initialization calls it to ensure the console output format is retained even if the console logger was set up before logging is initialized
|
| | |
|
| |
|
|
|
|
| |
* add ability to override scheduling mode in ParallelWork
* Don't increase buffering size when copying from local cache with --boost-worker-memory enabled
* Rework scheduling writes of downloaded data to reduce memory usage
|
| |
|
|
|
|
|
| |
* add system metrics output to top command
* removed unnecessary xmake directives
* file system API/comment tweaks
* fixed out-of-range access in httpserver test
* updated ZenServer base API to allow customization by mode
|
| |
|
| |
initially we had ZENCORE_API macros to potentially allow for DLL linkage. It turns out that this is not useful and the macros just contribute noise, so this change removes them completely.
|
| |
|
|
|
|
|
|
| |
* this adds a consul package which can be used to fetch a consul binary
* it also adds a `ConsulProcess` helper which can be used to spawn and manage a consul service instance
* zencore dependencies brought across:
- `except_fmt.h` for easer generation of formatted exception messages
- `process.h/cpp` changes (adds `Kill` operation and process group support on Windows)
- `string.h` changes to allow generic use of `WideToUtf8()`
|
| |
|
|
|
|
|
|
| |
commands (#706)
* added `--exclude-folders` to `zen upload`, `zen download` and `zen diff`
added `--exclude-extensions` to `zen upload` and `zen diff`
excluded folder names are now matched by folder name in subfolders in addition to root level folders
* allow multiple token separators
|
| |
|
|
|
| |
- Improvement: Deeper validation of data when scrub is activated (cas/cache/project)
- Improvement: Enabled more multi threading when running scrub operations
- Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
|
| |
|
| |
Implemented GetSystemMetrics function for Mac and Linux
|
| |
|
|
|
|
|
|
|
|
| |
* fix filtering of loose chunks when downloading with a filter
add tests
* changelog
* move InlineRemoveUnusedHashes
* remove extra braces
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder.
It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build.
* Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist
* BLAKE3 is now in-tree at `thirdparty/blake3`
* cpr is now in-tree at `thirdparty/cpr`
* cxxopts is now in-tree at `thirdparty/cxxopts`
* fmt is now in-tree at `thirdparty/fmt`
* robin-map is now in-tree at `thirdparty/robin-map`
* ryml is now in-tree at `thirdparty/ryml`
* sol2 is now in-tree at `thirdparty/sol2`
* spdlog is now in-tree at `thirdparty/spdlog`
* utfcpp is now in-tree at `thirdparty/utfcpp`
* xmake package repo definitions is in `repo`
* implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS
* the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux
* crashpad
* bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
|
| |
|
| |
* add support for downloading individual attachments from an oplog
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* gcc: avoid using memset on nontrivial struct
* redundant `return std::move`
* fixed various compilation issues flagged by gcc
* fix issue in xmake.lua detecting whether we are building with the UE toolchain or not
* add GCC ignore -Wundef (comment is inaccurate)
* remove redundant std::move
* don't catch exceptions by value
* unreferenced variables
* initialize "by the book" instead of memset
* remove unused exception reference
* add #include <cstring> to fix gcc build
* explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation
* remove unreferenced variable
* eliminate redundant `std::move` which gcc complains about
* fix gcc compilation by including <cstring>
* tag unreferenced variable to fix gcc compilation
* fixes for various cases of naming members the same as their type
|
| |
|
|
|
|
| |
Because the counter is decreased in `CountDown()` and subsequently checked against zero to determine if the completion event should be set:
`Latch::Wait` checks the counter against zero on entry and would exit early, before waiting on the completion event. This could then lead to the `Latch` instance being torn down before the `CountDown()` does a `Set()` which could then lead to unexpected and unwanted things happening.
|
| |
|
|
|
|
| |
* make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate
* fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header
with this fix, it is now possible to build with `--zentrace=no` again
|
| | |
|
| |
|
|
|
|
|
|
| |
- adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP)
- moved existing stats functionality from `zencore` to `zentelemetry`
- adds `TRefCounted<T>` for vtable-less refcounting
- adds `MemoryArena` class which allows for linear allocation of memory from chunks
- adds `protozero` which is used to encode OTLP protobuf messages
|
| |
|
|
|
|
|
|
|
|
| |
- Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage
- Improvement: `zen builds` now treats `.msixvc` files as non-compressable
Moved and cleaned up compactbinary_helpers functions
Tweaked fixed chunking implementation for better performance
Refactored so we have one list of "non-compressable" extensions
Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace
Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
|
| |
|
|
|
| |
* fix metadata info in filebuildstorage GetBuild
* move MakeSafeAbsolutePathÃnPlace to filesystem.h/cpp
* add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
|
| |
|
| |
Move builds download code from builds_cmd.cpp to remotestorelib
|
| |
|
|
| |
- Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during `zen builds download` operations
- Enabled by default, disable with `--allow-file-clone=false`
|
| |
|
|
|
|
|
|
|
|
|
|
| |
effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower.
* added --corelimit option to zenserver
* made sure thread pools are configured lazily and not during global init
* added log output indicating effective and HW concurrency
* added change log entry
* removed debug logging from ZenEntryPoint::Run()
also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp
* remove dependency to zenutil/workerpools.h from buildstoragecache.cpp
* remove unneded include
* move jupiter helpers to zenremotestore
* move parallelwork to zencore
* remove zenutil dependency from zenremotestore
* clean up test project dependencies - use indirect dependencies
|
| |
|
|
|
| |
these changes remove cpr from anything which is not `HttpClient` internals.
The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
|
| |
|
|
|
|
| |
terminate calls (#514)
Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls
Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
|
| |
|
|
|
|
| |
- Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present
- Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files
- Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client
- Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
|
| |
|
| |
- Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Improvement: Use consistent language for command line argument parsing errors
- Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error
- Improvement: Refactor zen command return codes to conform to valid Linux range (0-255)
kSuccess = 0,
kOtherError = 1,
kBadInput = 2,
kOutOfMemory = 16,
kOutOfDisk = 17,
kAssertError = 70,
kHttpOtherClientError = 80,
kHttpCantConnectError = 81,
kHttpNotFound = 66, // NotFound(404)
kHttpUnauthorized = 77, // Unauthorized(401),
kHttpSLLError = 82,
kHttpForbidden = 83, // Forbidden(403)
kHttpTimeout = 84, // RequestTimeout(408)
kHttpConflict = 85, // Conflict(409)
kHttpNoHost = 86,
kHttpOtherServerError = 90,
kHttpInternalServerError = 91, // InternalServerError(500)
kHttpServiceUnavailable = 69, // ServiceUnavailable(503)
kHttpBadGateway = 92, // BadGateway(502)
kHttpGatewayTimeout = 93, // GatewayTimeout(504)
|
| |
|
| |
* add validation of compact binary payloads before reading them
|
| |
|
|
| |
* add color coded logging level to console output (for warn/err/crit levels)
* clean up console output
|
| |\
| |
| | |
zen service command
|
| | |\ |
|
| | |\ \ |
|
| | |\ \ \ |
|
| | |\ \ \ \
| | | | | |
| | | | | |
| | | | | | |
de/zen-service-command
|
| | | | | | | |
|
| | | | | | | |
|
| | |\ \ \ \ \
| | |/ / / /
| |/| | | | |
|
| | |\ \ \ \ \ |
|