aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
...
| * adjust resource usage for dedicated servers (#466)Stefan Boberg2023-10-127-26/+84
| | | | | | | | | | | | | | | | | | when dedicated mode is enabled via `--dedicated` or `server.dedicated` then we tune http.sys server settings to be more suitable for a shared server initially we tune two things * the thread pool used to service I/O requests allows a larger number of threads to be created when needed. The minimum thread count is unchanged but in dedicated server mode we double the maximum number of threads allowed * the http.sys request queue length (`HttpServerQueueLengthProperty`) is increased to 50,000 in dedicated mode. The regular default is 1,000
* | Change default port to 8558zousar2023-10-119-13/+13
|/ | | | Changes the default port without altering config for shared instances.
* added explicit implementation of IoHash equals operator (#464)Stefan Boberg2023-10-111-0/+2
| | | | | | | | | * added explicit implementation of IoHash equals and less than operator using memcpy This improves codegen on MSVC in particular. Without this the operator is not inlined and consists of 20 individual (not in a loop) byte-by-byte comparisons each with an associated branch. With this change we get three comparisons (two uint64, one uint32) and three branches. So less pressure on branch predictors and as the code gets inlined the predictor should hopefully also function better.
* pluggable asio transport (#460)Stefan Boberg2023-10-1116-117/+607
| | | | | | | | | added pluggable transport based on asio. This is in an experimental state and is not yet a replacement for httpasio even though that is the ultimate goal also moved plugin API header into dedicated part of the tree to clarify that it is meant to be usable in isolation, without any dependency on zencore et al moved transport implementations into dedicated source directory in zenhttp note that this adds code to the build but nothing should change at runtime since the instantiation of the new code is conditional and is inactive by default
* hide HttpAsioServer interface behind factory function (#463)Stefan Boberg2023-10-113-24/+29
|
* updated plugin API class names (#462)Stefan Boberg2023-10-119-98/+97
| | | | | | | | | * renamed some interfaces to improve pluggable transport API TransportConnectionInterface -> TransportConnection TransportPluginInterface -> TransportPlugin TransportServerConnectionHandler -> TransportServerConnection TransportServerInterface -> TransportServer
* remove legacy compute interfaces (#461)Stefan Boberg2023-10-1112-3212/+41
| | | | | * removed legacy compute code, which will be replaced with a new implementation in the future * also updated references to Jupiter storage
* fix clang-format whoopsieStefan Boberg2023-10-111-1/+1
|
* added back main thread nameStefan Boberg2023-10-111-0/+2
| | | | this was there before but lost in a merge recently
* suppress no-op auth debug loggingStefan Boberg2023-10-101-1/+6
|
* experimental pluggable transport support (#436)Stefan Boberg2023-10-1015-42/+2039
| | | this change adds a `--http=plugin` mode where we support pluggable transports. Currently this defaults to a barebones blocking winsock implementation but there is also support for dynamic loading of transport plugins, which will be further developed in the near future.
* cache reference tracking (#455)Dan Engelbrecht2023-10-109-188/+598
| | | | | - Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC - disabled by default, enable with `--cache-reference-cache-enabled` - Improvement: Faster collection of referenced CId content in project store
* fixed GC logging output stats (#458)Stefan Boberg2023-10-101-1/+1
| | | disk usage stats are now properly reported in log messages
* minor fixes to zencore (#457)Stefan Boberg2023-10-094-8/+3
| | | | * AssertException constructor should be explicit * removed pragma comment
* reject bad bucket reads (#456)Stefan Boberg2023-10-093-5/+16
| | | | * extended bad bucket rejection logic to include GET operations as well as PUTs
* minor: remove double includeDan Engelbrecht2023-10-061-1/+0
|
* fix memory usage in memlayer (#454)Dan Engelbrecht2023-10-062-15/+4
| | | * Bugfix: Remove double counting of memory usage in memcachelayer
* fix gc infinite loop (#453)Dan Engelbrecht2023-10-063-10/+26
| | | | * make sure we update last gc time even if gc fails * If we can't check if an oplog/project markerfile exists, assume it is not expired
* zenserver project restructuring (#442)Stefan Boberg2023-10-0610-1533/+1741
|
* reject known bad bucket names in structured cache (#452)v0.2.27-pre0Stefan Boberg2023-10-066-8/+82
| | | | | | | * added string_view helpers for ParseHexBytes/ParseHexNumber * reject known bad buckets in structured cache put handler (32-character hex bucket names are rejected) * also added bucket rejection logic to bucket discovery * added rejected_writes stat to HttpStructuredCache
* ZenCacheMemoryLayer should always store values using memory buffers (#451)Stefan Boberg2023-10-052-5/+9
| | | | | | | | this change fixes a problem where the memory cache layer can inadvertently prevent underlying block store files from being deleted * ensure we get memory buffers on all paths * added more context to error in IoBufferBuilder::ReadFromFileMaybe * fixed problematic pread call success check in IoBufferBuilder::ReadFromFileMaybe which would always report failure on Linux/MacOS
* fixed issue where IoBufferBuilder::ReadFromFileMaybe loses content type (#450)Stefan Boberg2023-10-052-2/+4
|
* check that block does not exists on disk before starting write to it (#449)Dan Engelbrecht2023-10-052-18/+46
| | | * check that block does not exists on disk before starting write to it
* Fix curruption of disk cache bucket index on GC (#448)Dan Engelbrecht2023-10-054-75/+93
| | | | | | | | | * make sure we hold the index lock when reading payload data in reclaim space * don't use index snapshot when updating index in reclaim space * check that things have not moved under our feet * don't touch m_Payloads without a lock * start write block index on the highest block index * we don't need to bump writeblockindex when stopping write to a block, we will bump appropriately when we start a new block * changelog
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-10-051-10/+22
|\
| * reduce lock in disklayer (#447)Dan Engelbrecht2023-10-051-10/+22
| | | | | | * Don't block all write access to all buckets when doing GatherReferences/CollectGarbage
* | trivial: log output typo in GCStefan Boberg2023-10-051-1/+1
|/
* added --quiet option to suppress default logging to stdout (#446)Stefan Boberg2023-10-045-5/+27
| | | | added --quiet option for zenserver to suppress default logging to stdout
* fix HandleRpcRequest::WriteResponseAsync trace scope when we are not running ↵Dan Engelbrecht2023-10-041-1/+2
| | | | async
* added thread names (#441)Stefan Boberg2023-10-043-1/+7
| | | added names to process handle GC thread and main thread for easier identification in crash dumps / process monitoring
* refactor comapactcas index (#443)Dan Engelbrecht2023-10-044-22/+73
| | | | | - Bugfix: Fix scrub messing up payload and access time in disk cache bucket when compacting index - Improvement: Split up disk cache bucket index into hash lookup and payload array to improve performance - Improvement: Reserve space up front for compact binary output when saving cache bucket manifest to improve performance
* removed websocket protocol support(#445)Stefan Boberg2023-10-048-2008/+11
| | | removed websocket support since it is not used right now and is unlikely to be used in the future
* factored out http parser from asio into separate files (#444)Stefan Boberg2023-10-033-450/+495
| | | | factored out http request parsing from httpasio into separate files to enable code to be reused for different transports
* faster accesstime save restore (#439)Dan Engelbrecht2023-10-039-279/+426
| | | | | | | | | | - Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting - Change format for faster read/write and reduced size on disk - Don't lock index while writing manifest to disk - Skip garbage collect if we are currently in a Flush operation - BlockStore::Flush no longer terminates currently writing block - Garbage collect references to currently writing block but keep the block as new data may be added - Fix BlockStore::Prune used disk space calculation - Don't materialize data in filecas when we just need the size
* clean up date formatting (#440)Stefan Boberg2023-10-021-4/+4
| | | | * clean up date formatting (previous code would include a newline)
* fix formatting of gc start messagev0.2.26-pre0Dan Engelbrecht2023-10-021-1/+1
|
* reduce time spent holding http.sys transaction lock (#437)Stefan Boberg2023-10-022-15/+26
| | | | | * changed where calls to IssueNewRequestMaybe are made to reduce per-transaction lock contention * minor: reduce stack frame for HttpSysTransaction::IoCompletionCallback
* Handle OOM and OOD more gracefully to not spam Sentry with error reports (#434)Dan Engelbrecht2023-10-0210-53/+259
| | | | | | - Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to reqeuster without reporting an error to Sentry - Improvement: If creating bucket fails when storing and item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry - Improvement: Make an explicit flush of the active block written to in blockstore flush - Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit
* Limit size of memory cache layer (#423)Dan Engelbrecht2023-10-027-91/+371
| | | | | | | | - Feature: Limit the size ZenCacheMemoryLayer may use - `--cache-memlayer-targetfootprint` option to set which size (in bytes) it should be limited to, zero to have it unbounded - `--cache-memlayer-maxage` option to set how long (in seconds) cache items should be kept in the memory cache Do more "standard" GC rather than clearing everything. Tries to purge memory on Get/Put on the fly if exceeding limit - not sure if we should have a polling thread instead of adding overhead to Get/Put (however light it may be).
* lightweight gc (#431)Dan Engelbrecht2023-10-0212-60/+204
| | | | | | - Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store - Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false" - Add `--skipcid` option to `zen gc` command, defaults to false - Add `--gc-lightweight-interval-seconds` option to zenserver
* Sentry username fix (#435)Stefan Boberg2023-10-011-2/+2
| | | * fix trailing null in Sentry username (GetUserName returns the length including NUL terminator)
* Correctly calculate the total number of RPC ops in the stats page for ↵Dan Engelbrecht2023-09-291-2/+2
| | | | | structured cache (#433) Change "chunks" title to "count" for RPC chunk requests in stats page for structured cache
* added handle to http.sys error response logging (#432)Stefan Boberg2023-09-291-2/+3
| | | * added handle to http.sys error response logging
* move badrequest count back to old scope for compatability with Zen DashboardDan Engelbrecht2023-09-281-2/+1
|
* adding more stats (#429)Dan Engelbrecht2023-09-2816-72/+551
| | | | | - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
* added more context to http response error message (#430)Stefan Boberg2023-09-281-10/+77
| | | when a http.sys request response API call fails, we now include more information including metadata about the contents of the reponse (i.e memory/file references including file names/size and chunk offsets/size) to help track down some odd error conditions seen in production
* prefer to handle cache RPC requests synchronously (#428)Stefan Boberg2023-09-273-8/+44
| | | | | * only handle RPC requests in a worker thread if we have an upstream. we may as well handle the request inline on the http_io thread if we're only dealing with local data since the response times should be pretty consistent in that case * http.sys: don't create async worker thread pool until it's needed (typically only if we have an upstream)
* chache upstream stats improved (#426)Dan Engelbrecht2023-09-271-10/+16
| | | | * cache upstream stats improved * fix exit for monitor thread
* flush block store blocks when finished (#425)Dan Engelbrecht2023-09-271-0/+1
| | | * flush block store block
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-09-2643-202/+700
|\
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-14 09:00:24 +0000