aboutsummaryrefslogtreecommitdiff
path: root/zenserver/upstream
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-0212-6196/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* oplog upload/download (#214)Dan Engelbrecht2023-02-092-0/+43
| | | | | | | | | | | | - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint.
* removed experimental mesh codeStefan Boberg2023-01-262-343/+0
| | | | should be replaced with a proper implementation later
* Changed so CompressedBuffer::DecodeRawHash returns IoHash just like on the ↵Stefan Boberg2022-12-121-1/+1
| | | | | | UE side (#208) removed all use of IoHash::FromBLAKE3() caused by interactions with CompressedBuffer APIs
* optimizations (#200)Dan Engelbrecht2022-12-072-36/+58
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-182-6/+7
| | | * Add `import-project` and `export-project` command line parsing
* Add "Accept" field in RPC request to gracefully handle requests from older ↵Dan Engelbrecht2022-10-131-0/+4
| | | | instances (#180)
* Use bucket/key to get inline value in upstream for chunks without a chunkid ↵Dan Engelbrecht2022-09-301-3/+6
| | | | (#176)
* Format all rpc package responses using `FormatPackageMessageBuffer` to avoid ↵Dan Engelbrecht2022-09-291-9/+23
| | | | | memory copy (#174) When reading upstream, fall back to old rpc response to handle older instances.
* De/more upstream details (#168)v0.1.6-pre9v0.1.6-pre10Dan Engelbrecht2022-09-222-100/+117
| | | | | | * Pass along endpoint info for each upstream request * Add more timing details in log * more log details for single item upstream fetch * DISABLEDQUERY over SKIP
* Add elapsed seconds per individual request from upstream (#167)Dan Engelbrecht2022-09-212-13/+35
|
* rename URI chunk requests from value -> chunk (#166)Dan Engelbrecht2022-09-194-21/+21
|
* fix log messageDan Engelbrecht2022-09-151-1/+1
|
* only try to parse payload hash if header exists (#163)Dan Engelbrecht2022-09-091-1/+8
| | | | * only try to parse payload hash if header exists * changelog
* Adjust errors vs warnings messages (#160)Dan Engelbrecht2022-09-081-20/+20
| | | | * demote a number of ZEN_ERROR to ZEN_WARN * changelog
* Added CloudCacheSession::GetInlineBlob to properly get CacheValues (#159)Dan Engelbrecht2022-09-073-4/+59
| | | | | * Added CloudCacheSession::GetInlineBlob to properly get CacheValues from Horde Issue #UE-162151 * validate uncompressed binary from Horde "application/x-jupiter-inline" response
* Implement proper GetCacheValues upstream (#155)Dan Engelbrecht2022-09-062-17/+256
| | | | * Implement proper GetCacheValues upstream * changelog
* remove legacy derived data interface for Jupiter (#152)Dan Engelbrecht2022-08-313-105/+6
| | | * remove legacy derived data interface for Jupiter
* De/fix crash on non responding upstream (#145)Dan Engelbrecht2022-08-193-27/+61
| | | * Fix ZenStructuredCacheClient lifetime issues
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-174-19/+7
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* Enable adding namespace to URI based upstream requestsDan Engelbrecht2022-06-081-24/+23
|
* keep "reason" from upstream response so we can present it even if the ↵Dan Engelbrecht2022-06-011-9/+19
| | | | request fails without outright error
* remove unused parameterDan Engelbrecht2022-05-311-7/+4
|
* Remove namespace from HTTP URI style request in upstream until shared ↵Dan Engelbrecht2022-05-311-16/+20
| | | | instances are deployed with version that support si
* Horde execute compressed input blobs (#109)Joe Kirchoff2022-05-271-40/+123
|
* Keep Namespace out of CacheKey and store it on request levelDan Engelbrecht2022-05-194-57/+135
| | | | | RPC requests now has a Namespace field under Params instead of one Namespace per cache key Fall back to legacy upstream HTTP URI format if default namespace is requested
* string_view vs string lifetime fixDan Engelbrecht2022-05-121-2/+2
|
* Use configured namespace in Jupiter if not explicit namespace is givenDan Engelbrecht2022-05-114-72/+76
| | | | | DdcNamespace -> DefaultDdcNamespace BlobStoreNamespace -> DefaultBlobStoreNamespace
* parameterize namespace for upstream (first hack)Dan Engelbrecht2022-05-114-103/+135
|
* cleanup and review feedbackDan Engelbrecht2022-05-051-1/+1
|
* cleanupDan Engelbrecht2022-05-043-7/+6
|
* Add namespacecachestore layer to allow multiple structured cache namespacesDan Engelbrecht2022-05-042-6/+7
|
* Initialize upstream apply in background thread (#88)v1.0.1.2Joe Kirchoff2022-05-032-0/+23
|
* Reduce risk of reallocating backing std::vector in CbWriter::AddBinaryDan Engelbrecht2022-04-281-1/+1
| | | | | Shard up g_MappingLock in IoBufferExtendedCore::Materialize() to reduce contention during high load Don't queue upstream cache records if we don't have any upstreams
* Compute tweaks (#78)v1.0.0.3Joe Kirchoff2022-04-191-22/+8
| | | | Remove force upload, not necessary. Add a few more timepoints
* Horde execute storage config (#75)Joe Kirchoff2022-04-142-8/+14
|
* Compute updates (#74)Joe Kirchoff2022-04-145-1316/+1469
|
* Simple file-based compute (#65)Joe Kirchoff2022-03-304-340/+445
|
* Enable Horde compute code on Linux & Mac (#61)Joe Kirchoff2022-03-222-91/+33
|
* Update horde compute to use Jupiter for storage (#60)Joe Kirchoff2022-03-174-121/+216
|
* Value propagation fix - Read/Write ValueAPI as CompressedBinary type when ↵mattpetersepic2022-02-182-5/+39
| | | | | | writing to zen and horde upstreams. Return failure from HandleGetCacheRecord if the requested type does not match the cachetype. (#55) * Fix bug with getting values PUT to Jupiter as CompressedBinary. When getting CompressedBinary records from Jupiter, they are expected to now be a record with a reference to the compact binary. This has to be accounted for when performing upstream GETs. * HandleGetCacheRecord: avoid crashing on invalid type, and avoid sending back data that doesn't match the AcceptType.
* Fix typo 'application/x-ue-cb-cbpkg' -> 'application/x-ue-cbpkg'Matt Peters2022-02-111-1/+1
|
* Merge pull request #52 from EpicGames/ValuePropagationFixzousar2022-02-093-101/+227
|\ | | | | Change Value propagation to Zen or Jupiter
| * prepare_commit to fix formattingzousar2022-02-091-44/+46
| |
| * Change Value propagation to Zen or Jupiterzousar2022-02-083-105/+229
| | | | | | | | This change ensures we retain the right content type of kCompressedBinary when propagating values from Zen->UpstreamZen. This is done via an RPC that posts a CbPackage. Furthermore when propagating from Zen->Jupiter, it composes its own referencing CbObject for them instead of sending a octet content type and Jupiter defining the referencing CbObject. When fetching Values from Jupiter, this new composed CbObject is still interpreted correctly by Zen.
* | Remove the backwards compatibility for the Zen CachePolicy changes no… (#49)mattpetersepic2022-02-081-3/+0
|/ | | Remove the backwards compatibility for the Zen CachePolicy changes now that there has been enough time for all internal users of the old protocol to update.
* Minor cleanup of free functions.Per Larsson2022-02-034-42/+47
|
* Merged main.Per Larsson2022-02-026-169/+215
|\
| * Some minor cleanupStefan Boberg2022-02-022-37/+21
| |
| * Mash -> MeshTrackerStefan Boberg2022-02-022-12/+18
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 10:20:46 +0000