aboutsummaryrefslogtreecommitdiff
path: root/zenserver/cache/structuredcache.cpp
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-3159/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* oplog and cache stats (#244)Dan Engelbrecht2023-04-211-16/+252
| | | | | | | | | | | | | | | | * basic oplog stats * add GetValueStats to cache store * RwLock::ExclusiveLockScope -> RwLock::SharedLockScope * add rawhash and attachment count to CacheValueStats * added cache-stats and project-stats commands * add cast to make Mac overload detection happy * fix accept type in cache-stats command * Add options to project-stats command * use resource paths for stats in project store * use resource paths for stats in cache * fix cache-info and project-info url discriminator * more control over details$ output * cleanup * changelog
* send payloads as duplicated handles (#240)Dan Engelbrecht2023-03-141-17/+31
| | | | | | | | | * send payloads as duplicated handles if requestor provides process id and allows local file references. * linux/macos fixes * tests * fix access rights when duplicating handle * fix closing of duplicated handles on error * cleanup * changelog
* store cache rawhash and rawsize for unstructured cache values (#234)Dan Engelbrecht2023-02-231-30/+57
| | | | | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize. * Use CompressedBuffer::ValidateCompressedHeader when applicable * Make sure we rewrite the snapshot if we read an legacy existing index/log * changelog
* Enhanced rpc request recording (#229)Dan Engelbrecht2023-02-171-234/+51
| | | | | | | | | | | | | * rpc replay zen command * fix replay sessions for thread * recording start/stop as zen commands * move rpcrecording code to zenutil to remove code duplication * simplify recording http request threading * added more data logging to rpc replay * NotFound is an acceptable response for an rpc request * fix rpc replay command line parsing * rpc replay stats * Allow spawning of sub-process workers when replaying rpc recording * changelog
* Add info (GET) endpoints for structured cache (#211)Dan Engelbrecht2023-01-111-19/+141
| | | | | | | * Add GET requests on cache/namespace/bucket level * Add root route for project store requests (same as /list) * Add markerpath to oplog info object * Add totalsize, opcount and expired to oplog info * Changelog
* Fix log index snapshot (#210)Dan Engelbrecht2022-12-161-0/+1
| | | | | | * Fix log reading for structured cache store Make sure cache is flushed at exit * dont flush index to disk unless new entries have been written * changelog
* Path from handle perf improvement (#206)Dan Engelbrecht2022-12-081-8/+12
| | | | | | * Read recorded requests to memory before parsing This will more accurately simulate how requests comes in from a client * Make a fast path for GetFinalPathNameByHandleW Try to get the path with a fixes size buffer first to avoid always doing two calls to GetFinalPathNameByHandleW
* optimizations (#200)Dan Engelbrecht2022-12-071-55/+66
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Cache request record/replay (#198)Dan Engelbrecht2022-12-071-185/+385
| | | | | | | | | This adds recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request. `<host>/z$/exec$/start-recording?<disk-storage-path>` `<host>/z$/exec$/stop-recording` `<host>/z$/exec$/replay-recording?<thread-count>&<disk-storage-path>`
* Make sure we always store record/op before attachments (#195)Dan Engelbrecht2022-12-011-94/+137
| | | | | | | | * Make sure we always store record/op before attachments We don't want to store attachments first - a GC operation could then remove attachments if triggered before storing record/op * zen::Latch * Use latch to wait for attachments to be stored * use zen::latch when adding attachments from project oplog import * changelog
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-181-6/+9
| | | * Add `import-project` and `export-project` command line parsing
* Support file reference in package message (#184)Dan Engelbrecht2022-11-071-8/+26
| | | | | | | | * Fix packed message parsing for absolute path * Always enable are sharing when opening files as IoBuffers. * Allow control over sending partial files as localfile ref * Check "AcceptFlags" field in RPC message for allowing localfile ref in reply * make oplog entry add operations ZEN_DEBUG level logs * changelog
* Add "Accept" field in RPC request to gracefully handle requests from older ↵Dan Engelbrecht2022-10-131-21/+97
| | | | instances (#180)
* Format all rpc package responses using `FormatPackageMessageBuffer` to avoid ↵Dan Engelbrecht2022-09-291-26/+10
| | | | | memory copy (#174) When reading upstream, fall back to old rpc response to handle older instances.
* De/more upstream details (#168)v0.1.6-pre9v0.1.6-pre10Dan Engelbrecht2022-09-221-93/+166
| | | | | | * Pass along endpoint info for each upstream request * Add more timing details in log * more log details for single item upstream fetch * DISABLEDQUERY over SKIP
* rename URI chunk requests from value -> chunk (#166)Dan Engelbrecht2022-09-191-7/+7
|
* Implement proper GetCacheValues upstream (#155)Dan Engelbrecht2022-09-061-17/+34
| | | | * Implement proper GetCacheValues upstream * changelog
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-171-9/+9
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* namespace/bucket validation now uses AsciiSet for more efficient validation ↵Stefan Boberg2022-06-101-5/+7
| | | | (#127)
* Corrected some typos in log messagesStefan Boberg2022-06-081-3/+5
|
* add warnings on illegal namespace/bucket namesDan Engelbrecht2022-06-081-0/+5
|
* namespace dropDan Engelbrecht2022-05-251-9/+9
|
* fix mac compilation errorDan Engelbrecht2022-05-191-1/+1
|
* fix testsDan Engelbrecht2022-05-191-5/+5
|
* Keep Namespace out of CacheKey and store it on request levelDan Engelbrecht2022-05-191-96/+125
| | | | | RPC requests now has a Namespace field under Params instead of one Namespace per cache key Fall back to legacy upstream HTTP URI format if default namespace is requested
* align bucket naming rules with UE code baseDan Engelbrecht2022-05-181-3/+4
|
* review feedbackDan Engelbrecht2022-05-161-27/+26
|
* drop api/v2 prefix for non-legacy requestsDan Engelbrecht2022-05-161-112/+172
|
* use ns_ prefix on disk onlyDan Engelbrecht2022-05-161-5/+0
|
* use "default" as the default namespaceDan Engelbrecht2022-05-161-14/+14
| | | | remove ns_ prefix for namespaces on disk and in requests
* Validate max length for namespace nameDan Engelbrecht2022-05-121-0/+4
|
* keep compatability for valid bucket namesDan Engelbrecht2022-05-121-1/+1
|
* Add validation to namespace names that follows Jupiters rules.Dan Engelbrecht2022-05-121-225/+192
| | | | | Add unified validation of Namespace, Bucket and Hash for rpc requests. cleanup
* Tests for HttpRequestParseRelativeUriDan Engelbrecht2022-05-121-26/+142
|
* Add support for /api/v2/ URI requests with namespace supportDan Engelbrecht2022-05-121-25/+152
|
* revert Bucket - Key parameter changeDan Engelbrecht2022-05-121-2/+1
|
* cleanupDan Engelbrecht2022-05-111-26/+26
|
* parameterize namespace for upstream (first hack)Dan Engelbrecht2022-05-111-89/+151
|
* reverted unnecessary changesDan Engelbrecht2022-05-051-2/+2
|
* cleanup and review feedbackDan Engelbrecht2022-05-051-18/+24
|
* cleanupDan Engelbrecht2022-05-041-7/+6
|
* Add namespacecachestore layer to allow multiple structured cache namespacesDan Engelbrecht2022-05-041-24/+26
|
* mac compilation fixDan Engelbrecht2022-04-291-1/+1
|
* Use GetCacheValues when checking upstream to reduce number of calls to upstreamDan Engelbrecht2022-04-261-47/+101
| | | | Added some timing info to debug logs
* Value propagation fix - Read/Write ValueAPI as CompressedBinary type when ↵mattpetersepic2022-02-181-27/+44
| | | | | | writing to zen and horde upstreams. Return failure from HandleGetCacheRecord if the requested type does not match the cachetype. (#55) * Fix bug with getting values PUT to Jupiter as CompressedBinary. When getting CompressedBinary records from Jupiter, they are expected to now be a record with a reference to the compact binary. This has to be accounted for when performing upstream GETs. * HandleGetCacheRecord: avoid crashing on invalid type, and avoid sending back data that doesn't match the AcceptType.
* Fixed "Record* Record;" non-conformant compile errors from GCCMartin Ridgers2022-02-111-10/+10
|
* Merge pull request #52 from EpicGames/ValuePropagationFixzousar2022-02-091-1/+1
|\ | | | | Change Value propagation to Zen or Jupiter
| * Change Value propagation to Zen or Jupiterzousar2022-02-081-1/+1
| | | | | | | | This change ensures we retain the right content type of kCompressedBinary when propagating values from Zen->UpstreamZen. This is done via an RPC that posts a CbPackage. Furthermore when propagating from Zen->Jupiter, it composes its own referencing CbObject for them instead of sending a octet content type and Jupiter defining the referencing CbObject. When fetching Values from Jupiter, this new composed CbObject is still interpreted correctly by Zen.
* | Simplify HandleRpcGetCacheChunks (#53)mattpetersepic2022-02-091-260/+292
| | | | | | Refactor HandleRpcGetCacheChunks to reduce complexity. Port CacheStore tests from Unreal.
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 19:45:49 +0000