From 87248f58b4551870af8f08aac3e38e8887e32073 Mon Sep 17 00:00:00 2001 From: Stefan Boberg Date: Mon, 13 Apr 2026 12:57:51 +0200 Subject: Add MemoryCidStore and ChunkStore interface (#940) This PR introduces an in-memory `CidStore` option primarily for use with compute, to avoid hitting disk for ephemeral data which is not really worth persisting. And in particular not worth paying the critical path cost of persistence. - **MemoryCidStore**: In-memory CidStore implementation backed by a hash map, optionally layered over a standard CidStore. Writes to the backing store are dispatched asynchronously via a dedicated flush thread to avoid blocking callers on disk I/O. Reads check memory first, then fall back to the backing store without caching the result. - **ChunkStore interface**: Extract `ChunkStore` abstract class (`AddChunk`, `ContainsChunk`, `FilterChunks`) and `FallbackChunkResolver` into `zenstore.h` so `HttpComputeService` can accept different storage backends for action inputs vs worker binaries. `CidStore` and `MemoryCidStore` both implement `ChunkStore`. - **Compute service wiring**: `HttpComputeService` takes two `ChunkStore&` params (action + worker). The compute server uses `MemoryCidStore` for actions (no disk persistence needed) and disk-backed `CidStore` for workers (cross-action reuse). The storage server passes its `CidStore` for both (unchanged behavior). --- src/zenstore/cidstore.cpp | 12 ++++++++++++ 1 file changed, 12 insertions(+) (limited to 'src/zenstore/cidstore.cpp') diff --git a/src/zenstore/cidstore.cpp b/src/zenstore/cidstore.cpp index b20d8f565..ac8a75a58 100644 --- a/src/zenstore/cidstore.cpp +++ b/src/zenstore/cidstore.cpp @@ -187,12 +187,24 @@ CidStore::Initialize(const CidStoreConfiguration& Config) m_Impl->Initialize(Config); } +CidStore::InsertResult +CidStore::AddChunk(const IoBuffer& ChunkData, const IoHash& RawHash) +{ + return m_Impl->AddChunk(ChunkData, RawHash, InsertMode::kMayBeMovedInPlace); +} + CidStore::InsertResult CidStore::AddChunk(const IoBuffer& ChunkData, const IoHash& RawHash, InsertMode Mode) { return m_Impl->AddChunk(ChunkData, RawHash, Mode); } +std::vector +CidStore::AddChunks(std::span ChunkDatas, std::span RawHashes) +{ + return m_Impl->AddChunks(ChunkDatas, RawHashes, InsertMode::kMayBeMovedInPlace); +} + std::vector CidStore::AddChunks(std::span ChunkDatas, std::span RawHashes, InsertMode Mode) { -- cgit v1.2.3