aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/cache
Commit message (Collapse)AuthorAgeFilesLines
...
* add `flush` command and more gc status info (#483)Dan Engelbrecht2023-10-182-5/+8
| | | | | | - Feature: New endpoint `/admin/flush ` to flush all storage - CAS, Cache and ProjectStore - Feature: New command `zen flush` to flush all storage - CAS, Cache and ProjectStore - Improved: Command `zen gc-status` now gives details about storage, when last GC occured, how long until next GC etc - Changed: Cache access and write log are disabled by default
* removed unnecessary vector in ZenCacheMemoryLayer::DropStefan Boberg2023-10-171-3/+2
|
* fix mem layer total count (#480)Dan Engelbrecht2023-10-172-6/+17
| | | * fix calculation of ZenCacheMemoryLayer::m_TotalSize
* minor - fix references size arrayDan Engelbrecht2023-10-171-0/+2
|
* don't call compact references if caching is not enabled (#478)Dan Engelbrecht2023-10-171-6/+12
|
* cache reference tracking (#455)Dan Engelbrecht2023-10-105-182/+577
| | | | | - Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC - disabled by default, enable with `--cache-reference-cache-enabled` - Improvement: Faster collection of referenced CId content in project store
* reject bad bucket reads (#456)Stefan Boberg2023-10-093-5/+16
| | | | * extended bad bucket rejection logic to include GET operations as well as PUTs
* fix memory usage in memlayer (#454)Dan Engelbrecht2023-10-062-15/+4
| | | * Bugfix: Remove double counting of memory usage in memcachelayer
* reject known bad bucket names in structured cache (#452)v0.2.27-pre0Stefan Boberg2023-10-065-8/+69
| | | | | | | * added string_view helpers for ParseHexBytes/ParseHexNumber * reject known bad buckets in structured cache put handler (32-character hex bucket names are rejected) * also added bucket rejection logic to bucket discovery * added rejected_writes stat to HttpStructuredCache
* ZenCacheMemoryLayer should always store values using memory buffers (#451)Stefan Boberg2023-10-051-3/+4
| | | | | | | | this change fixes a problem where the memory cache layer can inadvertently prevent underlying block store files from being deleted * ensure we get memory buffers on all paths * added more context to error in IoBufferBuilder::ReadFromFileMaybe * fixed problematic pread call success check in IoBufferBuilder::ReadFromFileMaybe which would always report failure on Linux/MacOS
* Fix curruption of disk cache bucket index on GC (#448)Dan Engelbrecht2023-10-052-45/+54
| | | | | | | | | * make sure we hold the index lock when reading payload data in reclaim space * don't use index snapshot when updating index in reclaim space * check that things have not moved under our feet * don't touch m_Payloads without a lock * start write block index on the highest block index * we don't need to bump writeblockindex when stopping write to a block, we will bump appropriately when we start a new block * changelog
* reduce lock in disklayer (#447)Dan Engelbrecht2023-10-051-10/+22
| | | * Don't block all write access to all buckets when doing GatherReferences/CollectGarbage
* fix HandleRpcRequest::WriteResponseAsync trace scope when we are not running ↵Dan Engelbrecht2023-10-041-1/+2
| | | | async
* refactor comapactcas index (#443)Dan Engelbrecht2023-10-041-3/+10
| | | | | - Bugfix: Fix scrub messing up payload and access time in disk cache bucket when compacting index - Improvement: Split up disk cache bucket index into hash lookup and payload array to improve performance - Improvement: Reserve space up front for compact binary output when saving cache bucket manifest to improve performance
* faster accesstime save restore (#439)Dan Engelbrecht2023-10-033-132/+252
| | | | | | | | | | - Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting - Change format for faster read/write and reduced size on disk - Don't lock index while writing manifest to disk - Skip garbage collect if we are currently in a Flush operation - BlockStore::Flush no longer terminates currently writing block - Garbage collect references to currently writing block but keep the block as new data may be added - Fix BlockStore::Prune used disk space calculation - Don't materialize data in filecas when we just need the size
* Handle OOM and OOD more gracefully to not spam Sentry with error reports (#434)Dan Engelbrecht2023-10-021-10/+25
| | | | | | - Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to reqeuster without reporting an error to Sentry - Improvement: If creating bucket fails when storing and item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry - Improvement: Make an explicit flush of the active block written to in blockstore flush - Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit
* Limit size of memory cache layer (#423)Dan Engelbrecht2023-10-024-46/+285
| | | | | | | | - Feature: Limit the size ZenCacheMemoryLayer may use - `--cache-memlayer-targetfootprint` option to set which size (in bytes) it should be limited to, zero to have it unbounded - `--cache-memlayer-maxage` option to set how long (in seconds) cache items should be kept in the memory cache Do more "standard" GC rather than clearing everything. Tries to purge memory on Get/Put on the fly if exceeding limit - not sure if we should have a polling thread instead of adding overhead to Get/Put (however light it may be).
* lightweight gc (#431)Dan Engelbrecht2023-10-021-0/+5
| | | | | | - Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store - Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false" - Add `--skipcid` option to `zen gc` command, defaults to false - Add `--gc-lightweight-interval-seconds` option to zenserver
* Correctly calculate the total number of RPC ops in the stats page for ↵Dan Engelbrecht2023-09-291-2/+2
| | | | | structured cache (#433) Change "chunks" title to "count" for RPC chunk requests in stats page for structured cache
* move badrequest count back to old scope for compatability with Zen DashboardDan Engelbrecht2023-09-281-2/+1
|
* adding more stats (#429)Dan Engelbrecht2023-09-288-44/+351
| | | | | - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
* prefer to handle cache RPC requests synchronously (#428)Stefan Boberg2023-09-271-4/+17
| | | | | * only handle RPC requests in a worker thread if we have an upstream. we may as well handle the request inline on the http_io thread if we're only dealing with local data since the response times should be pretty consistent in that case * http.sys: don't create async worker thread pool until it's needed (typically only if we have an upstream)
* Add runtime status/control of logging (#419)Dan Engelbrecht2023-09-223-16/+44
| | | | | | | | | | | - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
* VFS implementation for local storage service (#396)Stefan Boberg2023-09-204-3/+57
| | | currently, only Windows (using Projected File System) is supported
* add DiskWriteBlocker to structured cache store log writer (#408)Dan Engelbrecht2023-09-152-31/+43
| | | | | * add DiskWriteBlocker to structured cache store log writer * changelog
* add more trace scopes (#362)Dan Engelbrecht2023-09-154-130/+212
| | | | | * more trace scopes * Make sure ReplayLogEntries uses the correct size for oplog buffer * changelog
* More statistics for Cache, Project Store and Cid Store (#405)Dan Engelbrecht2023-09-142-5/+56
| | | | | Cache: requestcount, badrequestcount, writes Project Store: requestcount Cid Store: cidhits, cidmisses, cidwrites
* issue warning instead of assert on bad data in cid store (#400)Dan Engelbrecht2023-09-131-10/+14
| | | * issue warning instead of assert on bad data in cid store
* ZenCacheStore is now reference counted (#398)Stefan Boberg2023-09-132-2/+13
| | | this change also adds a GetNamespaces function which may be used to enumerate all currently known cache namespaces
* gracefully handle errors when writing cache log (#391)Dan Engelbrecht2023-09-111-29/+50
| | | | | | | * gracefully handle errors when writing cache log * changelog * fix log message
* add `cache-write-log` and `cache-access-log´ configuration options (#394)Dan Engelbrecht2023-09-111-0/+2
| | | * add `cache-write-log` and `cache-access-log´ configuration options
* Minor: Make sure to reset cache logging worker thread event to avoid ↵Dan Engelbrecht2023-08-241-0/+1
| | | | busy-looping looking for more work
* Cache process handles for FormatPackageMessage (#360)Dan Engelbrecht2023-08-172-6/+15
|
* single thread async cache log (#361)Dan Engelbrecht2023-08-173-124/+149
| | | | * rework cache store background jogging * correct capture for context
* skip upstream logic early if we have no upstream endpoints (#359)Dan Engelbrecht2023-08-173-37/+77
| | | | | * Skip upstream logic early if we have not upstream endpoints * make cache store logging of CbObjects async * changelog
* update vcpkg dependencies (#356)Dan Engelbrecht2023-08-111-1/+2
| | | | | | * bump vcpkg version * fmt lib 10 fixes * xmake dependencies (with linux workarounds) * changelog
* Increase retry logic (#325)Dan Engelbrecht2023-06-051-13/+19
| | | | * Increase timeout and number of retries in CacheBucket::PutStandaloneCacheValue when moving temporary file into place * changelog
* cache log sessionid (#297)Stefan Boberg2023-05-234-168/+244
| | | | | | | | | | | * implemented structured cache logging to be used as audit trail to help analyse potential cache pollution/corruption * added common header to all known log targets * made Oid::operator bool explicit to avoid logging/text format mishaps * HttpClient::operator bool -> explicit * changed cache logs to not rotate on start in order to retain more history * added CacheRequestContext * properly initialize request context * log session id and request id on zencacehstore get/put * changelog
* minor: fixed typos in log output (PUTCACEHRECORD -> PUTCACHERECORD)Stefan Boberg2023-05-221-2/+2
|
* fix for commented-out code which was never meant to be checked inStefan Boberg2023-05-171-6/+6
|
* Restructured structured cache store (#314)Stefan Boberg2023-05-179-2809/+2882
| | | | This change separates out the disk and memory storage strategies into separate cpp/h files to improve maintainability.
* Content scrubbing (#271)Stefan Boberg2023-05-161-135/+290
| | | Added zen scrub command which may be triggered via the zen CLI helper. This traverses storage and validates contents either by content hash and/or by structure. If unexpected data is encountered it is invalidated.
* Add `--gc-projectstore-duration-seconds` option (#281)Dan Engelbrecht2023-05-161-9/+9
| | | | | | * Add `--gc-projectstore-duration-seconds` option * Cleanup lua gc options parsing * Remove dead configuration values * changelog
* removed RefCounted from ZenCacheNamespaceStefan Boberg2023-05-151-2/+13
| | | | also added some class comments
* removed remnants of ZEN_USE_REF_TRACKINGStefan Boberg2023-05-151-19/+0
| | | | this code was originally meant to be used for GC but is no longer needed
* MaxOS build fix: ScrubStorage must be overrideStefan Boberg2023-05-151-10/+16
|
* added ScrubStorage to GcStorage base classStefan Boberg2023-05-151-2/+3
|
* added scrubbing related loggingStefan Boberg2023-05-151-0/+4
|
* minor GC API cleanupStefan Boberg2023-05-154-23/+23
| | | | | Scrub -> ScrubStorage Trigger -> TriggerGc (to make relationship to TriggerScrub clearer)
* Remove ZEN_CACHE_TRACKER etcStefan Boberg2023-05-154-437/+0
| | | this was code which was originally intended for use with GC but it's no longer useful
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-24 04:51:28 +0000