builds upload command (#278)

- Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API
  - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**)
  - `builds upload` upload a folder to Cloud Build API
	  - `--local-path` source folder to upload
	  - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given
	  - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated
	  - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated
	  - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path`
	  - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set
	  - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set
	  - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks
	  - `--allow-multipart` enable usage of multi-part http upload requests
	  - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path`
  - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**)
	  - `--local-path` target folder to download to
	  - `--build-id` an Oid in hex form for the Build identifier to use
	  - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name`
	  - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path`
	  - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET)
	  - `--allow-multipart` enable usage of multi-part http download reqeusts
  - `builds diff` download a folder from Cloud Build API
	- `--local-path` target folder to download to
	- `--compare-path` folder to compare target with
	- `--only-chunked` compare only files that would be chunked
  - `builds fetch-blob` fetch and validate a blob from remote store
	  - `--build-id` an Oid in hex form for the Build identifier to use
    - `--blob-hash` an IoHash in hex form identifying the blob to download
  - `builds validate part` fetch a build part and validate all referenced attachments
	  - `--build-id` an Oid in hex form for the Build identifier to use
	  - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name`
	  - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id`
  - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations
	  - `--local-path` source folder to upload
  - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`)
	  - `--url` Cloud Builds URL
    - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake
    - `--namespace` Builds Storage namespace
    - `--bucket` Builds Storage bucket
    - Authentication options for Cloud Build API
	    - Auth token
	      - `--access-token` http auth Cloud Storage access token
	      - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token
	      - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token
	    - OpenId authentication
	      - `--openid-provider-name` Open ID provider name
	      - `--openid-provider-url` Open ID provider url
	      - `--openid-client-id`Open ID client id
	      - `--openid-refresh-token` Open ID refresh token
	      - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials
	      - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials
	    - OAuth authentication
	      - `--oauth-url` OAuth provier url
	      - `--oauth-clientid` OAuth client id
	      - `--oauth-clientsecret` OAuth client secret
  - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`)
	  - `--storage-path` path to folder to store builds data
    - `--json-metadata` enable json output in store for all compact binary objects (off by default)
  - Output options for all builds commands
    - `--plain-progress` use plain line-by-line progress output
    - `--verbose`

7 files changed, 480 insertions, 5 deletions

diff --git a/src/zenutil/include/zenutil/buildstorage.h b/src/zenutil/include/zenutil/buildstorage.h
new file mode 100644
index 000000000..9c236310f
--- /dev/null
+++ b/src/zenutil/include/zenutil/buildstorage.h
@@ -0,0 +1,55 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zencore/compactbinary.h>
+#include <zenutil/chunkblock.h>
+
+namespace zen {
+
+class BuildStorage
+{
+public:
+	struct Statistics
+	{
+		std::atomic<uint64_t> TotalBytesRead	   = 0;
+		std::atomic<uint64_t> TotalBytesWritten	   = 0;
+		std::atomic<uint64_t> TotalRequestCount	   = 0;
+		std::atomic<uint64_t> TotalRequestTimeUs   = 0;
+		std::atomic<uint64_t> TotalExecutionTimeUs = 0;
+	};
+
+	virtual ~BuildStorage() {}
+
+	virtual CbObject ListBuilds(CbObject Query)								= 0;
+	virtual CbObject PutBuild(const Oid& BuildId, const CbObject& MetaData) = 0;
+	virtual CbObject GetBuild(const Oid& BuildId)							= 0;
+	virtual void	 FinalizeBuild(const Oid& BuildId)						= 0;
+
+	virtual std::pair<IoHash, std::vector<IoHash>> PutBuildPart(const Oid&		 BuildId,
+																const Oid&		 BuildPartId,
+																std::string_view PartName,
+																const CbObject&	 MetaData)											 = 0;
+	virtual CbObject							   GetBuildPart(const Oid& BuildId, const Oid& BuildPartId)							 = 0;
+	virtual std::vector<IoHash> FinalizeBuildPart(const Oid& BuildId, const Oid& BuildPartId, const IoHash& PartHash)				 = 0;
+	virtual void PutBuildBlob(const Oid& BuildId, const IoHash& RawHash, ZenContentType ContentType, const CompositeBuffer& Payload) = 0;
+	virtual std::vector<std::function<void()>> PutLargeBuildBlob(const Oid&												   BuildId,
+																 const IoHash&											   RawHash,
+																 ZenContentType											   ContentType,
+																 uint64_t												   PayloadSize,
+																 std::function<IoBuffer(uint64_t Offset, uint64_t Size)>&& Transmitter,
+																 std::function<void(uint64_t, bool)>&&					   OnSentBytes)					 = 0;
+
+	virtual IoBuffer						   GetBuildBlob(const Oid& BuildId, const IoHash& RawHash) = 0;
+	virtual std::vector<std::function<void()>> GetLargeBuildBlob(
+		const Oid&																			   BuildId,
+		const IoHash&																		   RawHash,
+		uint64_t																			   ChunkSize,
+		std::function<void(uint64_t Offset, const IoBuffer& Chunk, uint64_t BytesRemaining)>&& Receiver) = 0;
+
+	virtual void PutBlockMetadata(const Oid& BuildId, const IoHash& BlockRawHash, const CbObject& MetaData)				 = 0;
+	virtual std::vector<ChunkBlockDescription> FindBlocks(const Oid& BuildId)											 = 0;
+	virtual std::vector<ChunkBlockDescription> GetBlockMetadata(const Oid& BuildId, std::span<const IoHash> BlockHashes) = 0;
+};
+
+}  // namespace zen
diff --git a/src/zenutil/include/zenutil/chunkblock.h b/src/zenutil/include/zenutil/chunkblock.h
index 9b7414629..21107fb7c 100644
--- a/src/zenutil/include/zenutil/chunkblock.h
+++ b/src/zenutil/include/zenutil/chunkblock.h
@@ -12,21 +12,28 @@
 
 namespace zen {
 
-struct ChunkBlockDescription
+struct ThinChunkBlockDescription
 {
-	IoHash				  BlockHash;
-	std::vector<IoHash>	  ChunkHashes;
+	IoHash				BlockHash;
+	std::vector<IoHash> ChunkRawHashes;
+};
+
+struct ChunkBlockDescription : public ThinChunkBlockDescription
+{
+	uint64_t			  HeaderSize;
 	std::vector<uint32_t> ChunkRawLengths;
+	std::vector<uint32_t> ChunkCompressedLengths;
 };
 
 std::vector<ChunkBlockDescription> ParseChunkBlockDescriptionList(const CbObjectView& BlocksObject);
 ChunkBlockDescription			   ParseChunkBlockDescription(const CbObjectView& BlockObject);
 CbObject						   BuildChunkBlockDescription(const ChunkBlockDescription& Block, CbObjectView MetaData);
-
+ChunkBlockDescription			   GetChunkBlockDescription(const SharedBuffer& BlockPayload, const IoHash& RawHash);
 typedef std::function<std::pair<uint64_t, CompressedBuffer>(const IoHash& RawHash)> FetchChunkFunc;
 
 CompressedBuffer GenerateChunkBlock(std::vector<std::pair<IoHash, FetchChunkFunc>>&& FetchChunks, ChunkBlockDescription& OutBlock);
 bool			 IterateChunkBlock(const SharedBuffer&														   BlockPayload,
-								   std::function<void(CompressedBuffer&& Chunk, const IoHash& AttachmentHash)> Visitor);
+								   std::function<void(CompressedBuffer&& Chunk, const IoHash& AttachmentHash)> Visitor,
+								   uint64_t&																   OutHeaderSize);
 
 }  // namespace zen
diff --git a/src/zenutil/include/zenutil/chunkedcontent.h b/src/zenutil/include/zenutil/chunkedcontent.h
new file mode 100644
index 000000000..15c687462
--- /dev/null
+++ b/src/zenutil/include/zenutil/chunkedcontent.h
@@ -0,0 +1,256 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zencore/compactbinary.h>
+#include <zencore/compactbinarybuilder.h>
+#include <zencore/iohash.h>
+
+#include <filesystem>
+#include <vector>
+
+ZEN_THIRD_PARTY_INCLUDES_START
+#include <tsl/robin_map.h>
+ZEN_THIRD_PARTY_INCLUDES_END
+
+namespace zen {
+
+class CbWriter;
+class ChunkingController;
+class WorkerThreadPool;
+
+enum class SourcePlatform
+{
+	Windows = 0,
+	Linux	= 1,
+	MacOS	= 2,
+	_Count
+};
+
+std::string_view ToString(SourcePlatform Platform);
+SourcePlatform	 FromString(std::string_view Platform, SourcePlatform Default);
+SourcePlatform	 GetSourceCurrentPlatform();
+
+struct FolderContent
+{
+	SourcePlatform					   Platform = GetSourceCurrentPlatform();
+	std::vector<std::filesystem::path> Paths;
+	std::vector<uint64_t>			   RawSizes;
+	std::vector<uint32_t>			   Attributes;
+	std::vector<uint64_t>			   ModificationTicks;
+
+	bool operator==(const FolderContent& Rhs) const;
+
+	bool		AreKnownFilesEqual(const FolderContent& Rhs) const;
+	void		UpdateState(const FolderContent& Rhs, std::vector<uint32_t>& PathIndexesOufOfDate);
+	static bool AreFileAttributesEqual(const uint32_t Lhs, const uint32_t Rhs);
+};
+
+FolderContent GetUpdatedContent(const FolderContent&				Old,
+								const FolderContent&				New,
+								std::vector<std::filesystem::path>& OutDeletedPathIndexes);
+
+void		  SaveFolderContentToCompactBinary(const FolderContent& Content, CbWriter& Output);
+FolderContent LoadFolderContentToCompactBinary(CbObjectView Input);
+
+struct GetFolderContentStatistics
+{
+	std::atomic<uint64_t> FoundFileCount		= 0;
+	std::atomic<uint64_t> FoundFileByteCount	= 0;
+	std::atomic<uint64_t> AcceptedFileCount		= 0;
+	std::atomic<uint64_t> AcceptedFileByteCount = 0;
+	uint64_t			  ElapsedWallTimeUS		= 0;
+};
+
+FolderContent GetFolderContent(GetFolderContentStatistics&																Stats,
+							   const std::filesystem::path&																RootPath,
+							   std::function<bool(const std::string_view& RelativePath)>&&								AcceptDirectory,
+							   std::function<bool(std::string_view RelativePath, uint64_t Size, uint32_t Attributes)>&& AcceptFile,
+							   WorkerThreadPool&																		WorkerPool,
+							   int32_t																					UpdateInteralMS,
+							   std::function<void(bool IsAborted, std::ptrdiff_t PendingWork)>&&						UpdateCallback,
+							   std::atomic<bool>&																		AbortFlag);
+
+struct ChunkedContentData
+{
+	// To describe one asset with a particular RawHash, find the index of the hash in SequenceRawHashes
+	// ChunkCounts for that index will be the number of indexes in ChunkOrders that describe
+	// the sequence of chunks required to reconstruct the asset.
+	// Offset into ChunkOrders is based on how many entries in ChunkOrders the previous [n - 1] SequenceRawHashes uses
+	std::vector<IoHash>	  SequenceRawHashes;  // Raw hash for Chunk sequence
+	std::vector<uint32_t> ChunkCounts;		  // Chunk count of ChunkOrder for SequenceRawHashes[n]
+	std::vector<uint32_t> ChunkOrders;		  // Chunk sequence indexed into ChunkHashes, ChunkCounts[n] indexes per SequenceRawHashes[n]
+	std::vector<IoHash>	  ChunkHashes;		  // Unique chunk hashes
+	std::vector<uint64_t> ChunkRawSizes;	  // Unique chunk raw size for ChunkHash[n]
+};
+
+struct ChunkedFolderContent
+{
+	SourcePlatform					   Platform = GetSourceCurrentPlatform();
+	std::vector<std::filesystem::path> Paths;
+	std::vector<uint64_t>			   RawSizes;
+	std::vector<uint32_t>			   Attributes;
+	std::vector<IoHash>				   RawHashes;
+	ChunkedContentData				   ChunkedContent;
+};
+
+void				 SaveChunkedFolderContentToCompactBinary(const ChunkedFolderContent& Content, CbWriter& Output);
+ChunkedFolderContent LoadChunkedFolderContentToCompactBinary(CbObjectView Input);
+
+ChunkedFolderContent MergeChunkedFolderContents(const ChunkedFolderContent& Base, std::span<const ChunkedFolderContent> Overlays);
+ChunkedFolderContent DeletePathsFromChunkedContent(const ChunkedFolderContent& Base, std::span<const std::filesystem::path> DeletedPaths);
+
+struct ChunkingStatistics
+{
+	std::atomic<uint64_t> FilesProcessed	   = 0;
+	std::atomic<uint64_t> FilesChunked		   = 0;
+	std::atomic<uint64_t> BytesHashed		   = 0;
+	std::atomic<uint64_t> UniqueChunksFound	   = 0;
+	std::atomic<uint64_t> UniqueSequencesFound = 0;
+	std::atomic<uint64_t> UniqueBytesFound	   = 0;
+	uint64_t			  ElapsedWallTimeUS	   = 0;
+};
+
+ChunkedFolderContent ChunkFolderContent(ChunkingStatistics&												  Stats,
+										WorkerThreadPool&												  WorkerPool,
+										const std::filesystem::path&									  RootPath,
+										const FolderContent&											  Content,
+										const ChunkingController&										  InChunkingController,
+										int32_t															  UpdateInteralMS,
+										std::function<void(bool IsAborted, std::ptrdiff_t PendingWork)>&& UpdateCallback,
+										std::atomic<bool>&												  AbortFlag);
+
+struct ChunkedContentLookup
+{
+	struct ChunkLocation
+	{
+		uint32_t PathIndex;
+		uint64_t Offset;
+	};
+	tsl::robin_map<IoHash, uint32_t, IoHash::Hasher> ChunkHashToChunkIndex;
+	tsl::robin_map<IoHash, uint32_t, IoHash::Hasher> RawHashToSequenceRawHashIndex;
+	std::vector<uint32_t>							 SequenceRawHashIndexChunkOrderOffset;
+	std::vector<ChunkLocation>						 ChunkLocations;
+	std::vector<size_t>	  ChunkLocationOffset;	// ChunkLocations[ChunkLocationOffset[ChunkIndex]] -> start of sources for ChunkIndex
+	std::vector<uint32_t> ChunkLocationCounts;	// ChunkLocationCounts[ChunkIndex] count of chunk locations for ChunkIndex
+};
+
+ChunkedContentLookup BuildChunkedContentLookup(const ChunkedFolderContent& Content);
+
+inline std::pair<size_t, uint32_t>
+GetChunkLocationRange(const ChunkedContentLookup& Lookup, uint32_t ChunkIndex)
+{
+	return std::make_pair(Lookup.ChunkLocationOffset[ChunkIndex], Lookup.ChunkLocationCounts[ChunkIndex]);
+}
+
+inline std::span<const ChunkedContentLookup::ChunkLocation>
+GetChunkLocations(const ChunkedContentLookup& Lookup, uint32_t ChunkIndex)
+{
+	std::pair<size_t, uint32_t> Range = GetChunkLocationRange(Lookup, ChunkIndex);
+	return std::span<const ChunkedContentLookup::ChunkLocation>(Lookup.ChunkLocations).subspan(Range.first, Range.second);
+}
+
+namespace compactbinary_helpers {
+	template<typename Type>
+	void WriteArray(std::span<const Type> Values, std::string_view ArrayName, CbWriter& Output)
+	{
+		Output.BeginArray(ArrayName);
+		for (const Type Value : Values)
+		{
+			Output << Value;
+		}
+		Output.EndArray();
+	}
+
+	template<typename Type>
+	void WriteArray(const std::vector<Type>& Values, std::string_view ArrayName, CbWriter& Output)
+	{
+		WriteArray(std::span<const Type>(Values), ArrayName, Output);
+	}
+
+	template<>
+	inline void WriteArray(std::span<const std::filesystem::path> Values, std::string_view ArrayName, CbWriter& Output)
+	{
+		Output.BeginArray(ArrayName);
+		for (const std::filesystem::path& Path : Values)
+		{
+			Output.AddString((const char*)Path.generic_u8string().c_str());
+		}
+		Output.EndArray();
+	}
+
+	template<>
+	inline void WriteArray(const std::vector<std::filesystem::path>& Values, std::string_view ArrayName, CbWriter& Output)
+	{
+		WriteArray(std::span<const std::filesystem::path>(Values), ArrayName, Output);
+	}
+
+	inline void WriteBinaryAttachmentArray(std::span<const IoHash> Values, std::string_view ArrayName, CbWriter& Output)
+	{
+		Output.BeginArray(ArrayName);
+		for (const IoHash& Hash : Values)
+		{
+			Output.AddBinaryAttachment(Hash);
+		}
+		Output.EndArray();
+	}
+
+	inline void WriteBinaryAttachmentArray(const std::vector<IoHash>& Values, std::string_view ArrayName, CbWriter& Output)
+	{
+		WriteArray(std::span<const IoHash>(Values), ArrayName, Output);
+	}
+
+	inline void ReadArray(std::string_view ArrayName, CbObjectView Input, std::vector<uint32_t>& Result)
+	{
+		CbArrayView Array = Input[ArrayName].AsArrayView();
+		Result.reserve(Array.Num());
+		for (CbFieldView ItemView : Array)
+		{
+			Result.push_back(ItemView.AsUInt32());
+		}
+	}
+
+	inline void ReadArray(std::string_view ArrayName, CbObjectView Input, std::vector<uint64_t>& Result)
+	{
+		CbArrayView Array = Input[ArrayName].AsArrayView();
+		Result.reserve(Array.Num());
+		for (CbFieldView ItemView : Array)
+		{
+			Result.push_back(ItemView.AsUInt64());
+		}
+	}
+
+	inline void ReadArray(std::string_view ArrayName, CbObjectView Input, std::vector<std::filesystem::path>& Result)
+	{
+		CbArrayView Array = Input[ArrayName].AsArrayView();
+		Result.reserve(Array.Num());
+		for (CbFieldView ItemView : Array)
+		{
+			std::u8string_view U8Path = ItemView.AsU8String();
+			Result.push_back(std::filesystem::path(U8Path));
+		}
+	}
+
+	inline void ReadArray(std::string_view ArrayName, CbObjectView Input, std::vector<IoHash>& Result)
+	{
+		CbArrayView Array = Input[ArrayName].AsArrayView();
+		Result.reserve(Array.Num());
+		for (CbFieldView ItemView : Array)
+		{
+			Result.push_back(ItemView.AsHash());
+		}
+	}
+
+	inline void ReadBinaryAttachmentArray(std::string_view ArrayName, CbObjectView Input, std::vector<IoHash>& Result)
+	{
+		CbArrayView Array = Input[ArrayName].AsArrayView();
+		Result.reserve(Array.Num());
+		for (CbFieldView ItemView : Array)
+		{
+			Result.push_back(ItemView.AsBinaryAttachment());
+		}
+	}
+
+}  // namespace compactbinary_helpers
+
+}  // namespace zen
diff --git a/src/zenutil/include/zenutil/chunkingcontroller.h b/src/zenutil/include/zenutil/chunkingcontroller.h
new file mode 100644
index 000000000..fe4fc1bb5
--- /dev/null
+++ b/src/zenutil/include/zenutil/chunkingcontroller.h
@@ -0,0 +1,55 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zencore/compactbinary.h>
+
+#include <zenutil/chunkedfile.h>
+
+#include <atomic>
+#include <filesystem>
+
+namespace zen {
+
+const std::vector<std::string_view> DefaultChunkingExcludeExtensions = {".exe", ".dll", ".pdb", ".self"};
+
+const ChunkedParams DefaultChunkedParams = {.MinSize = ((8u * 1u) * 1024u) - 128u,
+											.MaxSize = 128u * 1024u,
+											.AvgSize = ((8u * 4u) * 1024u) + 128u};
+
+const size_t DefaultChunkingFileSizeLimit = DefaultChunkedParams.MaxSize;
+
+const uint32_t DefaultFixedChunkingChunkSize = 16u * 1024u * 1024u;
+
+struct ChunkedInfoWithSource;
+
+class ChunkingController
+{
+public:
+	virtual ~ChunkingController() {}
+
+	// Return true if the input file was processed. If true is returned OutChunked will contain the chunked info
+	virtual bool			 ProcessFile(const std::filesystem::path& InputPath,
+										 uint64_t					  RawSize,
+										 ChunkedInfoWithSource&		  OutChunked,
+										 std::atomic<uint64_t>&		  BytesProcessed) const = 0;
+	virtual std::string_view GetName() const										  = 0;
+	virtual CbObject		 GetParameters() const									  = 0;
+};
+
+std::unique_ptr<ChunkingController> CreateBasicChunkingController(
+	std::span<const std::string_view> ExcludeExtensions	 = DefaultChunkingExcludeExtensions,
+	uint64_t						  ChunkFileSizeLimit = DefaultChunkingFileSizeLimit,
+	const ChunkedParams&			  ChunkingParams	 = DefaultChunkedParams);
+std::unique_ptr<ChunkingController> CreateBasicChunkingController(CbObjectView Parameters);
+
+std::unique_ptr<ChunkingController> CreateChunkingControllerWithFixedChunking(
+	std::span<const std::string_view> ExcludeExtensions		 = DefaultChunkingExcludeExtensions,
+	uint64_t						  ChunkFileSizeLimit	 = DefaultChunkingFileSizeLimit,
+	const ChunkedParams&			  ChunkingParams		 = DefaultChunkedParams,
+	uint32_t						  FixedChunkingChunkSize = DefaultFixedChunkingChunkSize);
+std::unique_ptr<ChunkingController> CreateChunkingControllerWithFixedChunking(CbObjectView Parameters);
+
+std::unique_ptr<ChunkingController> CreateChunkingController(std::string_view Name, CbObjectView Parameters);
+
+}  // namespace zen
diff --git a/src/zenutil/include/zenutil/filebuildstorage.h b/src/zenutil/include/zenutil/filebuildstorage.h
new file mode 100644
index 000000000..c95fb32e6
--- /dev/null
+++ b/src/zenutil/include/zenutil/filebuildstorage.h
@@ -0,0 +1,16 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zencore/logging.h>
+#include <zenutil/buildstorage.h>
+
+namespace zen {
+class HttpClient;
+
+std::unique_ptr<BuildStorage> CreateFileBuildStorage(const std::filesystem::path& StoragePath,
+													 BuildStorage::Statistics&	  Stats,
+													 bool						  EnableJsonOutput,
+													 double						  LatencySec	= 0.0,
+													 double						  DelayPerKBSec = 0.0);
+}  // namespace zen
diff --git a/src/zenutil/include/zenutil/jupiter/jupiterbuildstorage.h b/src/zenutil/include/zenutil/jupiter/jupiterbuildstorage.h
new file mode 100644
index 000000000..89fc70140
--- /dev/null
+++ b/src/zenutil/include/zenutil/jupiter/jupiterbuildstorage.h
@@ -0,0 +1,17 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zencore/logging.h>
+#include <zenutil/buildstorage.h>
+
+namespace zen {
+class HttpClient;
+
+std::unique_ptr<BuildStorage> CreateJupiterBuildStorage(LoggerRef					 InLog,
+														HttpClient&					 InHttpClient,
+														BuildStorage::Statistics&	 Stats,
+														std::string_view			 Namespace,
+														std::string_view			 Bucket,
+														const std::filesystem::path& TempFolderPath);
+}  // namespace zen
diff --git a/src/zenutil/include/zenutil/parallellwork.h b/src/zenutil/include/zenutil/parallellwork.h
new file mode 100644
index 000000000..7a8218c51
--- /dev/null
+++ b/src/zenutil/include/zenutil/parallellwork.h
@@ -0,0 +1,69 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+#pragma once
+
+#include <zencore/thread.h>
+#include <zencore/workthreadpool.h>
+
+#include <atomic>
+
+namespace zen {
+
+class ParallellWork
+{
+public:
+	ParallellWork(std::atomic<bool>& AbortFlag) : m_AbortFlag(AbortFlag), m_PendingWork(1) {}
+
+	~ParallellWork()
+	{
+		// Make sure to call Wait before destroying
+		ZEN_ASSERT(m_PendingWork.Remaining() == 0);
+	}
+
+	void ScheduleWork(WorkerThreadPool&																WorkerPool,
+					  std::function<void(std::atomic<bool>& AbortFlag)>&&							Work,
+					  std::function<void(const std::exception& Ex, std::atomic<bool>& AbortFlag)>&& OnError)
+	{
+		m_PendingWork.AddCount(1);
+		try
+		{
+			WorkerPool.ScheduleWork([this, Work = std::move(Work), OnError = std::move(OnError)] {
+				try
+				{
+					Work(m_AbortFlag);
+				}
+				catch (const std::exception& Ex)
+				{
+					OnError(Ex, m_AbortFlag);
+				}
+				m_PendingWork.CountDown();
+			});
+		}
+		catch (const std::exception&)
+		{
+			m_PendingWork.CountDown();
+			throw;
+		}
+	}
+
+	void Abort() { m_AbortFlag = true; }
+
+	bool IsAborted() const { return m_AbortFlag.load(); }
+
+	void Wait(int32_t UpdateInteralMS, std::function<void(bool IsAborted, std::ptrdiff_t PendingWork)>&& UpdateCallback)
+	{
+		ZEN_ASSERT(m_PendingWork.Remaining() > 0);
+		m_PendingWork.CountDown();
+		while (!m_PendingWork.Wait(UpdateInteralMS))
+		{
+			UpdateCallback(m_AbortFlag.load(), m_PendingWork.Remaining());
+		}
+	}
+	Latch& PendingWork() { return m_PendingWork; }
+
+private:
+	std::atomic<bool>& m_AbortFlag;
+	Latch			   m_PendingWork;
+};
+
+}  // namespace zen