aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
...
* improved process monitoring behaviour with invalid pids (#16)Stefan Boberg2024-03-211-2/+3
|
* non memory copy compressed range (#13)Dan Engelbrecht2024-03-204-44/+53
| | | | | * Add CompressedBuffer::GetRange that references source data rather than make a memory copy * Use Compressed.CopyRange in project store GetChunkRange * docs for CompressedBuffer::CopyRange and CompressedBuffer::GetRange
* read jupiter token from file (#10)Dan Engelbrecht2024-03-181-5/+12
| | | | | * Add ability to specify a json file for cloud access token OidcToken.exe in UE can be asked to produce such a file with the -OutFile option * avoid division by zero when reporting progress
* add a retry when trying to create the lock file (#9)Dan Engelbrecht2024-03-181-2/+8
| | | * add a retry when trying to create the lock file if UE is trying to read it at the same time
* make sure zenserver reacts and exist on SIGTERM signal (#8)Dan Engelbrecht2024-03-152-0/+7
| | | | | * make sure zenserver reacts and exist on SIGTERM signal * add zen tag to all runners * temp disable mac codesigning
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-144-624/+1325
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* clean up test linking (#4)Dan Engelbrecht2024-03-145-288/+16
| | | | | | | - Improvement: Add zenhttp-test and zenutil-test - Improvement: Moved cachepolicy test to cachepolicy.cpp - Improvement: Renamed cachestore tests from z$ to cachestore - Improvement: Moved test linking so test for a lib is linked by <lib>-test - Improvement: Removed HttpRequestParseRelativeUri in httpstructuredcache.cpp and use the one in cacherequests.h instead
* fix potential partially written files (#2)Dan Engelbrecht2024-03-131-8/+2
| | | | * Make sure WriteFile() does not leave incomplete files * use TemporaryFile and MoveTemporaryIntoPlace to avoid leaving partial files on error
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-281-1/+3
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* remove reference caching (#658)Dan Engelbrecht2024-02-274-17/+4
| | | * remove reference caching
* hashing fixes (#657)Dan Engelbrecht2024-02-261-1/+0
| | | | | * move structuredcachestore tests to zenstore-test * Don't materialize entire files when hashing if it is a large files * rewrite CompositeBuffer::Mid to never materialize buffers
* Keep track of added ops during GCV2 instead of rescanning full oplog when ↵Dan Engelbrecht2024-02-132-8/+25
| | | | added ops are detected (#652)
* Save compressed large attachments to temporary files on disk (#650)Dan Engelbrecht2024-02-122-183/+217
| | | | | | | | | * Save large compressed large attachments to temporary files on disk * bump oplog block max size up to 64Mb again * Make sure CompositeBuffer::AppendBuffers actually moves inputs when it should * removed parallell execution of fetching payload for block assembly it was not actually helping and added complexity * make sure we move/release payload buffers as soon as possible * make sure we don't read in full large attachments to memory when computing hash
* compress large attachments on demand (#647)Dan Engelbrecht2024-02-054-362/+550
| | | | | | | - Improvement: Speed up oplog export by fetching/compressing big attachments on demand - Improvement: Speed up oplog export by batch-fetcing small attachments - Improvement: Speed up oplog import by batching writes of oplog ops - Improvement: Tweak oplog export default block size and embed size limit - Improvement: Add more messaging and progress during oplog import/export
* respond with BadRequest result instead of throwing exception on bad request ↵Dan Engelbrecht2024-02-051-2/+12
| | | | input (#648)
* changed RPC recording to MPSC setup (#638)Stefan Boberg2024-01-311-21/+1
| | | fixes rare race condition when using RPC recording for long periods of time
* only try to traverse an objectstore bucket if it really exists (#646)Dan Engelbrecht2024-01-311-0/+1
| | | * Only try to traverse a objectstore bucket if it really exists
* improve oplog export logging (#644)Dan Engelbrecht2024-01-318-117/+223
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-292-5/+6
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-254-36/+51
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-245-1037/+178
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* oplog import/export improvements (#634)Dan Engelbrecht2024-01-233-224/+368
| | | | * improve feedback from oplog import/export * improve oplog save performance
* add --ignore-missing-attachments to oplog-import command (#637)Dan Engelbrecht2024-01-223-49/+83
|
* improved errors from jupiter upstream (#636)Dan Engelbrecht2024-01-222-5/+72
| | | * get more detailed error messages from jupiter upstream
* jobqueue - allow multiple threads to report progress/messages (#635)Dan Engelbrecht2024-01-221-0/+5
| | | jobqueue - add AbortReason and properly propagate error when running async command
* Use correct HTTP range values. (#632)Per Larsson2024-01-171-1/+1
|
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-2011-1773/+71
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-1911-7697/+6
|
* ensure we can build without trace (#619)Stefan Boberg2023-12-192-2/+4
| | | | `xmake config -zentrace=n` would previously not build cleanly
* various TSAN/ASAN/LeakAnalyzer fixes (#622)Stefan Boberg2023-12-191-0/+5
| | | | | | | | | * fix JobQueue test threading issue. The inner job queued with `QueueJob` would reference `I` from inside the captured closure which would subsequently disappear * made sure application exit is thread safe * don't try to access string data out of bounds * keep-alive flag is accessed from multiple threads * fix memory leaks in Zen upstream client code * TSAN fixes for Event
* fix ChunkIndexToChunkHash indexing (#621)Stefan Boberg2023-12-191-1/+1
| | | would previously index into a reserved-but-not-sized vector which is bad but not crash-inducing bad
* cache RPC recorder threading fixes (#617)Stefan Boberg2023-12-192-15/+62
| | | | | * ensure all access to m_Entries is done while holding lock * RPC recorder concurrency fixes - setup/teardown of recorder needs to be done while holding an exclusive lock. Calls into recorder should be done while holding a shared lock.
* Don't use copy of Payloads array when fetching memcached payload in GC (#609)Dan Engelbrecht2023-12-131-1/+1
| | | * Don't use copy of Payloads array when fetching memcached payload in GC
* improve trace (#606)Dan Engelbrecht2023-12-132-36/+53
| | | | | * Adding some more trace scopes for better visiblity * Removed spammy trace scope when replaying oplogs * Remove "::Disk" from trace scopes - redundant now that we have merge disk and memory layers
* Adding an info command to display a top-level summary of disk space etc (#602)Stefan Boberg2023-12-125-31/+258
| | | | | this also adds a central, shared folder for storing information which may be found by any instance on the host. The directory is currently located alongside the default install and state directory. Initially this is used to store a collection of known `root_manifest` locations and a copy of the latest manifest version which allow us to find all known locations where zen state is present.
* mem cache perf improvements (#592)Dan Engelbrecht2023-12-112-120/+168
| | | | | | | | - Improvement: Refactor memory cache for faster trimming and correct trim reporting - Improvement: Added trace scopes for memory cache trimming Adding a link back to the cache item payload on the memory cache item allows us to iterate over only the items cached in memory instead of over the entire index. This also allows us to do efficient compact of the memory cache array when trimming. It adds 4 bytes of overhead to each item cached in memory.
* added details to trace initialization (#588)Stefan Boberg2023-12-112-4/+4
| | | this adds information on program name and command line to trace initialization
* improved scrubbing of oplogs and filecas (#596)Stefan Boberg2023-12-118-87/+301
| | | | | | - Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks) - Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command - Improvement: Implemented oplog scrubbing (previously was a no-op) - Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
* multi-line logging improvements (#597)Stefan Boberg2023-12-111-6/+7
| | | | | | * added ZEN_SCOPED_WARN and implemented multi-line logging * changed so file log also uses `fullformatter` for consistency and to get the multi-line support across the board
* fix deadlock at bucket creation (#598)Dan Engelbrecht2023-12-112-177/+205
| | | | | | - Make sure we don't hold the namespace bucket lock when we create buckets to avoid deadlock - Pass lock scope to helper functions to clarify locking rules - Block flush and gc operations for a bucket that is not yet initialized - Add ZenCacheDiskLayer::GetOrCreateBucket to avoid code duplication
* Merge branch 'main' into zs/get-all-chunk-infoszs/get-all-chunk-infoszousar2023-12-075-21/+80
|\
| * logging configuration via command line options (#589)Stefan Boberg2023-12-063-3/+28
| | | | | | | | | | | | | | with these changes it is possible to configure loggers on the command line. For instance: `xmake run zenserver --log-trace=http_requests,http` will configure the system so that the `http_request` and `http` loggers are set to TRACE level
| * Use correct iterator index when looking up memcached payload in ↵Dan Engelbrecht2023-12-051-5/+4
| | | | | | | | | | GatherReferences (#591) * Use correct iterator index when looking up memcached payload in gatherreferences
| * HTTP plugin request debug logging (#587)Stefan Boberg2023-12-051-1/+1
| | | | | | | | | | | | * added log level control/query to LoggerRef * added debug logging to http plugin implementation * added GetDebugName() to transport plugin interfaces * added debug name to log output
| * reserve vectors in gcv2 upfront / load factor for robin_map (#582)Dan Engelbrecht2023-12-041-5/+20
| | | | | | | | | | * reserve vectors in gcv2 upfront * set max load factor for robin_map indexes to reduce memory usage * set min load factor for robin_map indexes to allow them to shrink
| * memory usage estimation for memcached entries (#586)Dan Engelbrecht2023-12-041-5/+24
| | | | | | | | * do a more accurate memory usage estimation for memcached entries * early exit when checking memcache usage
| * Add context to Zen upstream resolve error message (#585)Stefan Boberg2023-12-041-1/+1
| |
| * safe threadpool shutdown (#584)Dan Engelbrecht2023-12-041-1/+2
| | | | | | * shut down thread pools earlier to worker threads has a chance to terminate before main thread atexit
* | Change naming to ChunkInfos instead of Chunkszousar2023-12-064-12/+12
| |
* | Ran precommitzousar2023-12-052-7/+3
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-24 00:24:52 +0000