aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
...
* Actually thow exception if we can't parse the jobid when starting async job ↵Dan Engelbrecht2024-01-313-4/+15
| | | | | | (#645) Return system error code on exception throw in zen command Clean up HttpClient::Response::ErrorMessage to remove redundant ": "
* improve oplog export logging (#644)Dan Engelbrecht2024-01-319-126/+232
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-293-6/+7
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* exception on basicfile read error (#642)Dan Engelbrecht2024-01-292-10/+30
| | | | | * Change BasicFile::Read to throw exception like IoBuffer - Don't ASSERT on dwNumberOfBytesRead == NumberOfBytesToRead - throw exception with details instead - Use proper return type for pread()
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-257-39/+68
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Use proper format for range request header (#640)Dan Engelbrecht2024-01-241-4/+10
| | | | Clear header callback after use Use separate temp-vector for headers
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-249-1258/+688
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* oplog import/export improvements (#634)Dan Engelbrecht2024-01-234-226/+371
| | | | * improve feedback from oplog import/export * improve oplog save performance
* add --ignore-missing-attachments to oplog-import command (#637)Dan Engelbrecht2024-01-225-53/+98
|
* improved errors from jupiter upstream (#636)Dan Engelbrecht2024-01-223-6/+82
| | | * get more detailed error messages from jupiter upstream
* jobqueue - allow multiple threads to report progress/messages (#635)Dan Engelbrecht2024-01-224-19/+37
| | | jobqueue - add AbortReason and properly propagate error when running async command
* make sure to advance read buffer pointer in BasicFileWriter::Write (#633)Dan Engelbrecht2024-01-221-1/+3
|
* Use correct HTTP range values. (#632)Per Larsson2024-01-171-1/+1
|
* reinstate formatter settings for file sink (#631)Stefan Boberg2024-01-171-0/+9
| | | fixes issue introduced in #615
* remove use of <random> in stats (#628)Stefan Boberg2024-01-102-6/+45
| | | | the previous implementation was quite slow due to use of mt and uniform_distribution.
* release RPC recording memory early (#627)Stefan Boberg2023-12-201-2/+7
| | | previously the segment would call `clear()` on the `m_Entries` vector but this does not release the backing memory so we have to do the `swap` dance
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-2031-1789/+2002
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-1912-28/+27
|
* Don't capture local variables in loop by reference (#623)Dan Engelbrecht2023-12-191-27/+27
| | | * Don't capture local variables in loop by reference
* ensure we can build without trace (#619)Stefan Boberg2023-12-195-2/+13
| | | | `xmake config -zentrace=n` would previously not build cleanly
* various TSAN/ASAN/LeakAnalyzer fixes (#622)Stefan Boberg2023-12-196-10/+32
| | | | | | | | | * fix JobQueue test threading issue. The inner job queued with `QueueJob` would reference `I` from inside the captured closure which would subsequently disappear * made sure application exit is thread safe * don't try to access string data out of bounds * keep-alive flag is accessed from multiple threads * fix memory leaks in Zen upstream client code * TSAN fixes for Event
* fix ChunkIndexToChunkHash indexing (#621)Stefan Boberg2023-12-192-2/+2
| | | would previously index into a reserved-but-not-sized vector which is bad but not crash-inducing bad
* fix leak in IoBuffer for manifested small chunk (#618)Stefan Boberg2023-12-191-2/+2
| | | | * fix leak in IoBuffer for manifested small chunk. previously it would null out the `m_DataPtr` member on every path from `IoBufferExtendedCore::~IoBufferExtendedCore()` but it only really makes sense to null it out when the buffer has been memory mapped
* cache RPC recorder threading fixes (#617)Stefan Boberg2023-12-193-54/+110
| | | | | * ensure all access to m_Entries is done while holding lock * RPC recorder concurrency fixes - setup/teardown of recorder needs to be done while holding an exclusive lock. Calls into recorder should be done while holding a shared lock.
* Fix crash bug when trying to inspect non-open block file in GC (#614)Dan Engelbrecht2023-12-182-7/+20
|
* Make sure we initialize the pattern of FileSink before it is added as a ↵Dan Engelbrecht2023-12-181-12/+8
| | | | usable logger (#615)
* log each block that is compacted (#613)Dan Engelbrecht2023-12-151-0/+5
|
* fixed v2 rpc recording issue with >4GB data per segment (#612)Stefan Boberg2023-12-154-146/+323
| | | | | | | | | * fixed v2 rpc recording issue with >4GB data per segment * implemented recovery logic to deal with partial RPC recordings * added check for invalid/null requests in RPC replay * also made sure at least one worker thread is configured * fix problem where "null" requests would cause infinite loop! * added basic RPC recorder tests
* implement cache recording segment split by age (#611)Stefan Boberg2023-12-143-17/+110
| | | - also fixes weird DateTime/TimeSpan comparison operator
* Make sure IoBuffer is a valid null-buffer after move operation (#610)Dan Engelbrecht2023-12-142-4/+15
|
* Don't use copy of Payloads array when fetching memcached payload in GC (#609)Dan Engelbrecht2023-12-131-1/+1
| | | * Don't use copy of Payloads array when fetching memcached payload in GC
* fix peak disk load in gc status (#608)Dan Engelbrecht2023-12-131-12/+11
| | | * MaxLoad is max load per monitor slot, not the MaxLoad for the entire graph
* skip invalid chunks when reclaiming space in block store (#607)Dan Engelbrecht2023-12-131-0/+13
| | | * skip invalid chunks when reclaiming space in block store
* improve trace (#606)Dan Engelbrecht2023-12-135-37/+65
| | | | | * Adding some more trace scopes for better visiblity * Removed spammy trace scope when replaying oplogs * Remove "::Disk" from trace scopes - redundant now that we have merge disk and memory layers
* fix crash at log exit (#605)Dan Engelbrecht2023-12-131-7/+5
| | | * keep g_FileSink alive until spdlog has shut down
* use mimalloc where available (#601)Stefan Boberg2023-12-121-15/+11
| | | enabling mimalloc path for `Memory::Alloc` and `Memory::Free`
* Adding an info command to display a top-level summary of disk space etc (#602)Stefan Boberg2023-12-128-31/+336
| | | | | this also adds a central, shared folder for storing information which may be found by any instance on the host. The directory is currently located alongside the default install and state directory. Initially this is used to store a collection of known `root_manifest` locations and a copy of the latest manifest version which allow us to find all known locations where zen state is present.
* premature logging shutdown fix (#603)v0.2.36Stefan Boberg2023-12-121-0/+9
|
* header only compressed buffers are valid (#604)Dan Engelbrecht2023-12-121-2/+2
| | | - Bugfix: Allow attachments that contains a raw size of zero
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-12-11117-5211/+9027
|\
| * Make sure that PathFromHandle don't hide true error when throwing exceptions ↵Dan Engelbrecht2023-12-113-16/+31
| | | | | | | | | | | | | | | | | | (#600) * Make sure that PathFromHandle don't hide true error when throwing exceptions * changelog * return error info in PathFromHandle if path fails to resolve
| * mem cache perf improvements (#592)Dan Engelbrecht2023-12-112-120/+168
| | | | | | | | | | | | | | | | - Improvement: Refactor memory cache for faster trimming and correct trim reporting - Improvement: Added trace scopes for memory cache trimming Adding a link back to the cache item payload on the memory cache item allows us to iterate over only the items cached in memory instead of over the entire index. This also allows us to do efficient compact of the memory cache array when trimming. It adds 4 bytes of overhead to each item cached in memory.
| * added details to trace initialization (#588)Stefan Boberg2023-12-114-11/+17
| | | | | | this adds information on program name and command line to trace initialization
| * improved scrubbing of oplogs and filecas (#596)Stefan Boberg2023-12-1117-137/+439
| | | | | | | | | | | | - Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks) - Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command - Improvement: Implemented oplog scrubbing (previously was a no-op) - Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
| * multi-line logging improvements (#597)Stefan Boberg2023-12-114-78/+127
| | | | | | | | | | | | * added ZEN_SCOPED_WARN and implemented multi-line logging * changed so file log also uses `fullformatter` for consistency and to get the multi-line support across the board
| * SuppressConsoleLog now removes any existing console logger (#599)Stefan Boberg2023-12-111-0/+5
| | | | | | SuppressConsoleLog now removes any existing console logger to avoid exceptions in spdlog
| * fix deadlock at bucket creation (#598)Dan Engelbrecht2023-12-112-177/+205
| | | | | | | | | | | | - Make sure we don't hold the namespace bucket lock when we create buckets to avoid deadlock - Pass lock scope to helper functions to clarify locking rules - Block flush and gc operations for a bucket that is not yet initialized - Add ZenCacheDiskLayer::GetOrCreateBucket to avoid code duplication
| * Merge branch 'main' into zs/get-all-chunk-infoszs/get-all-chunk-infoszousar2023-12-0731-209/+512
| |\
| | * fixed bug in CasContainerStrategy::ReadIndexFile (#595)Stefan Boberg2023-12-071-1/+3
| | | | | | | | | this was introduced in a recent optimization and would cause CAS items to not be found after a shutdown/restart cycle
| | * logging configuration via command line options (#589)Stefan Boberg2023-12-066-7/+114
| | | | | | | | | | | | | | | | | | | | | with these changes it is possible to configure loggers on the command line. For instance: `xmake run zenserver --log-trace=http_requests,http` will configure the system so that the `http_request` and `http` loggers are set to TRACE level
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-16 05:58:16 +0000