diff options
| author | Stefan Boberg <[email protected]> | 2026-04-13 12:57:51 +0200 |
|---|---|---|
| committer | GitHub Enterprise <[email protected]> | 2026-04-13 12:57:51 +0200 |
| commit | 87248f58b4551870af8f08aac3e38e8887e32073 (patch) | |
| tree | 2e0d7d18b5397ff7ebec3b5a30668af47dfb5941 /src/zencompute/include | |
| parent | 5.8.4-pre1 (diff) | |
| download | zen-87248f58b4551870af8f08aac3e38e8887e32073.tar.xz zen-87248f58b4551870af8f08aac3e38e8887e32073.zip | |
Add MemoryCidStore and ChunkStore interface (#940)
This PR introduces an in-memory `CidStore` option primarily for use with compute, to avoid hitting disk for ephemeral data which is not really worth persisting. And in particular not worth paying the critical path cost of persistence.
- **MemoryCidStore**: In-memory CidStore implementation backed by a hash map, optionally layered over a standard CidStore. Writes to the backing store are dispatched asynchronously via a dedicated flush thread to avoid blocking callers on disk I/O. Reads check memory first, then fall back to the backing store without caching the result.
- **ChunkStore interface**: Extract `ChunkStore` abstract class (`AddChunk`, `ContainsChunk`, `FilterChunks`) and `FallbackChunkResolver` into `zenstore.h` so `HttpComputeService` can accept different storage backends for action inputs vs worker binaries. `CidStore` and `MemoryCidStore` both implement `ChunkStore`.
- **Compute service wiring**: `HttpComputeService` takes two `ChunkStore&` params (action + worker). The compute server uses `MemoryCidStore` for actions (no disk persistence needed) and disk-backed `CidStore` for workers (cross-action reuse). The storage server passes its `CidStore` for both (unchanged behavior).
Diffstat (limited to 'src/zencompute/include')
| -rw-r--r-- | src/zencompute/include/zencompute/httpcomputeservice.h | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/src/zencompute/include/zencompute/httpcomputeservice.h b/src/zencompute/include/zencompute/httpcomputeservice.h index de85a295f..db3fce3c2 100644 --- a/src/zencompute/include/zencompute/httpcomputeservice.h +++ b/src/zencompute/include/zencompute/httpcomputeservice.h @@ -15,7 +15,7 @@ # include <memory> namespace zen { -class CidStore; +class ChunkStore; } namespace zen::compute { @@ -26,7 +26,8 @@ namespace zen::compute { class HttpComputeService : public HttpService, public IHttpStatsProvider, public IWebSocketHandler, public IComputeCompletionObserver { public: - HttpComputeService(CidStore& InCidStore, + HttpComputeService(ChunkStore& InActionStore, + ChunkStore& InWorkerStore, IHttpStatsService& StatsService, const std::filesystem::path& BaseDir, int32_t MaxConcurrentActions = 0); |