aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
...
* reject bad bucket reads (#456)Stefan Boberg2023-10-093-5/+16
| | | | * extended bad bucket rejection logic to include GET operations as well as PUTs
* minor: remove double includeDan Engelbrecht2023-10-061-1/+0
|
* fix memory usage in memlayer (#454)Dan Engelbrecht2023-10-062-15/+4
| | | * Bugfix: Remove double counting of memory usage in memcachelayer
* fix gc infinite loop (#453)Dan Engelbrecht2023-10-063-10/+26
| | | | * make sure we update last gc time even if gc fails * If we can't check if an oplog/project markerfile exists, assume it is not expired
* zenserver project restructuring (#442)Stefan Boberg2023-10-0610-1533/+1741
|
* reject known bad bucket names in structured cache (#452)v0.2.27-pre0Stefan Boberg2023-10-066-8/+82
| | | | | | | * added string_view helpers for ParseHexBytes/ParseHexNumber * reject known bad buckets in structured cache put handler (32-character hex bucket names are rejected) * also added bucket rejection logic to bucket discovery * added rejected_writes stat to HttpStructuredCache
* ZenCacheMemoryLayer should always store values using memory buffers (#451)Stefan Boberg2023-10-052-5/+9
| | | | | | | | this change fixes a problem where the memory cache layer can inadvertently prevent underlying block store files from being deleted * ensure we get memory buffers on all paths * added more context to error in IoBufferBuilder::ReadFromFileMaybe * fixed problematic pread call success check in IoBufferBuilder::ReadFromFileMaybe which would always report failure on Linux/MacOS
* fixed issue where IoBufferBuilder::ReadFromFileMaybe loses content type (#450)Stefan Boberg2023-10-052-2/+4
|
* check that block does not exists on disk before starting write to it (#449)Dan Engelbrecht2023-10-052-18/+46
| | | * check that block does not exists on disk before starting write to it
* Fix curruption of disk cache bucket index on GC (#448)Dan Engelbrecht2023-10-054-75/+93
| | | | | | | | | * make sure we hold the index lock when reading payload data in reclaim space * don't use index snapshot when updating index in reclaim space * check that things have not moved under our feet * don't touch m_Payloads without a lock * start write block index on the highest block index * we don't need to bump writeblockindex when stopping write to a block, we will bump appropriately when we start a new block * changelog
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-10-051-10/+22
|\
| * reduce lock in disklayer (#447)Dan Engelbrecht2023-10-051-10/+22
| | | | | | * Don't block all write access to all buckets when doing GatherReferences/CollectGarbage
* | trivial: log output typo in GCStefan Boberg2023-10-051-1/+1
|/
* added --quiet option to suppress default logging to stdout (#446)Stefan Boberg2023-10-045-5/+27
| | | | added --quiet option for zenserver to suppress default logging to stdout
* fix HandleRpcRequest::WriteResponseAsync trace scope when we are not running ↵Dan Engelbrecht2023-10-041-1/+2
| | | | async
* added thread names (#441)Stefan Boberg2023-10-043-1/+7
| | | added names to process handle GC thread and main thread for easier identification in crash dumps / process monitoring
* refactor comapactcas index (#443)Dan Engelbrecht2023-10-044-22/+73
| | | | | - Bugfix: Fix scrub messing up payload and access time in disk cache bucket when compacting index - Improvement: Split up disk cache bucket index into hash lookup and payload array to improve performance - Improvement: Reserve space up front for compact binary output when saving cache bucket manifest to improve performance
* removed websocket protocol support(#445)Stefan Boberg2023-10-048-2008/+11
| | | removed websocket support since it is not used right now and is unlikely to be used in the future
* factored out http parser from asio into separate files (#444)Stefan Boberg2023-10-033-450/+495
| | | | factored out http request parsing from httpasio into separate files to enable code to be reused for different transports
* faster accesstime save restore (#439)Dan Engelbrecht2023-10-039-279/+426
| | | | | | | | | | - Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting - Change format for faster read/write and reduced size on disk - Don't lock index while writing manifest to disk - Skip garbage collect if we are currently in a Flush operation - BlockStore::Flush no longer terminates currently writing block - Garbage collect references to currently writing block but keep the block as new data may be added - Fix BlockStore::Prune used disk space calculation - Don't materialize data in filecas when we just need the size
* clean up date formatting (#440)Stefan Boberg2023-10-021-4/+4
| | | | * clean up date formatting (previous code would include a newline)
* fix formatting of gc start messagev0.2.26-pre0Dan Engelbrecht2023-10-021-1/+1
|
* reduce time spent holding http.sys transaction lock (#437)Stefan Boberg2023-10-022-15/+26
| | | | | * changed where calls to IssueNewRequestMaybe are made to reduce per-transaction lock contention * minor: reduce stack frame for HttpSysTransaction::IoCompletionCallback
* Handle OOM and OOD more gracefully to not spam Sentry with error reports (#434)Dan Engelbrecht2023-10-0210-53/+259
| | | | | | - Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to reqeuster without reporting an error to Sentry - Improvement: If creating bucket fails when storing and item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry - Improvement: Make an explicit flush of the active block written to in blockstore flush - Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit
* Limit size of memory cache layer (#423)Dan Engelbrecht2023-10-027-91/+371
| | | | | | | | - Feature: Limit the size ZenCacheMemoryLayer may use - `--cache-memlayer-targetfootprint` option to set which size (in bytes) it should be limited to, zero to have it unbounded - `--cache-memlayer-maxage` option to set how long (in seconds) cache items should be kept in the memory cache Do more "standard" GC rather than clearing everything. Tries to purge memory on Get/Put on the fly if exceeding limit - not sure if we should have a polling thread instead of adding overhead to Get/Put (however light it may be).
* lightweight gc (#431)Dan Engelbrecht2023-10-0212-60/+204
| | | | | | - Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store - Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false" - Add `--skipcid` option to `zen gc` command, defaults to false - Add `--gc-lightweight-interval-seconds` option to zenserver
* Sentry username fix (#435)Stefan Boberg2023-10-011-2/+2
| | | * fix trailing null in Sentry username (GetUserName returns the length including NUL terminator)
* Correctly calculate the total number of RPC ops in the stats page for ↵Dan Engelbrecht2023-09-291-2/+2
| | | | | structured cache (#433) Change "chunks" title to "count" for RPC chunk requests in stats page for structured cache
* added handle to http.sys error response logging (#432)Stefan Boberg2023-09-291-2/+3
| | | * added handle to http.sys error response logging
* move badrequest count back to old scope for compatability with Zen DashboardDan Engelbrecht2023-09-281-2/+1
|
* adding more stats (#429)Dan Engelbrecht2023-09-2816-72/+551
| | | | | - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
* added more context to http response error message (#430)Stefan Boberg2023-09-281-10/+77
| | | when a http.sys request response API call fails, we now include more information including metadata about the contents of the reponse (i.e memory/file references including file names/size and chunk offsets/size) to help track down some odd error conditions seen in production
* prefer to handle cache RPC requests synchronously (#428)Stefan Boberg2023-09-273-8/+44
| | | | | * only handle RPC requests in a worker thread if we have an upstream. we may as well handle the request inline on the http_io thread if we're only dealing with local data since the response times should be pretty consistent in that case * http.sys: don't create async worker thread pool until it's needed (typically only if we have an upstream)
* chache upstream stats improved (#426)Dan Engelbrecht2023-09-271-10/+16
| | | | * cache upstream stats improved * fix exit for monitor thread
* flush block store blocks when finished (#425)Dan Engelbrecht2023-09-271-0/+1
| | | * flush block store block
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-09-2643-202/+700
|\
| * handle error in error log (#422)Dan Engelbrecht2023-09-251-15/+46
| | | | | | * do not allow exceptions to leak from Sentry error reporting or SentryAssertImpl::OnAssert
| * Add runtime status/control of logging (#419)Dan Engelbrecht2023-09-2211-29/+277
| | | | | | | | | | | | | | | | | | | | | | - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
| * added name to http.sys request queue (#417)Stefan Boberg2023-09-221-1/+33
| | | | | | | | | | | | * added name to http.sys request queue * changed so that rejected requests are answered with a 403 response instead of just dropping the connection
| * Collect all zen admin-related commands into admin.h/.cpp (#418)Dan Engelbrecht2023-09-2233-140/+117
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * move commands in scrub.h/cpp to admin_cmd.h/cpp * move job command into admin_cmd.h/.cpp * admin -> admin_cmd * bench -> bench_cmd * cache -> cache_cmd * copy -> copy_cmd * dedup -> dedup_cmd * hash -> hash_cmd * print -> print_cmd * projectstore -> projectstore_cmd * rpcreplay -> rpcreplay_cmd * serve -> serve_cmd * status -> status_cmd * top -> top_cmd * trace -> trace_cmd * up -> up_cmd * version -> version_cmd
| * add trace command to enable/disable tracing at runtime (#416)Dan Engelbrecht2023-09-227-38/+248
| | | | | | | | | | * add trace command to enable/disable tracing at runtime * rework tracing init/start/stop * changelog
* | sort commands for cleaner mergesStefan Boberg2023-09-261-2/+2
|/
* Improvement: Add names to background jobs for easier debugging (#412)Dan Engelbrecht2023-09-206-113/+176
| | | | Improvement: Background jobs now temporarily sets thread name to background job name while executing Improvement: Background jobs tracks worker thread id used while executing
* controlled zenserver shutdown (#413)Dan Engelbrecht2023-09-201-20/+36
| | | * Do controlled shutdown of zenserver catching any exception thrown during shutdown
* added info verb to vfs command help textStefan Boberg2023-09-201-1/+1
|
* VFS implementation for local storage service (#396)Stefan Boberg2023-09-2031-27/+2553
| | | currently, only Windows (using Projected File System) is supported
* increase jupiter oplog import/export connection timeout to 30 min (#411)Dan Engelbrecht2023-09-191-1/+1
| | | | * increase jupiter oplog import/export connection timeout to 30 min * changelog
* Add retry if FinalizeRef responds with non-empty "Needs" attachments (#409)Dan Engelbrecht2023-09-195-261/+377
| | | | | * Add retry if FinalizeRef responds with non-empty "Needs" attachments * better logging/progress report * changelog
* handle errors in spdlog gracefully (#410)Dan Engelbrecht2023-09-191-0/+29
| | | | * handle errors in spdlog gracefully - try to report and avoid termination * changelog
* add DiskWriteBlocker to structured cache store log writer (#408)Dan Engelbrecht2023-09-153-32/+45
| | | | | * add DiskWriteBlocker to structured cache store log writer * changelog
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-31 01:57:47 +0000