builds upload command (#278)

- Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API
  - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**)
  - `builds upload` upload a folder to Cloud Build API
	  - `--local-path` source folder to upload
	  - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given
	  - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated
	  - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated
	  - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path`
	  - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set
	  - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set
	  - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks
	  - `--allow-multipart` enable usage of multi-part http upload requests
	  - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path`
  - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**)
	  - `--local-path` target folder to download to
	  - `--build-id` an Oid in hex form for the Build identifier to use
	  - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name`
	  - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path`
	  - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET)
	  - `--allow-multipart` enable usage of multi-part http download reqeusts
  - `builds diff` download a folder from Cloud Build API
	- `--local-path` target folder to download to
	- `--compare-path` folder to compare target with
	- `--only-chunked` compare only files that would be chunked
  - `builds fetch-blob` fetch and validate a blob from remote store
	  - `--build-id` an Oid in hex form for the Build identifier to use
    - `--blob-hash` an IoHash in hex form identifying the blob to download
  - `builds validate part` fetch a build part and validate all referenced attachments
	  - `--build-id` an Oid in hex form for the Build identifier to use
	  - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name`
	  - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id`
  - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations
	  - `--local-path` source folder to upload
  - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`)
	  - `--url` Cloud Builds URL
    - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake
    - `--namespace` Builds Storage namespace
    - `--bucket` Builds Storage bucket
    - Authentication options for Cloud Build API
	    - Auth token
	      - `--access-token` http auth Cloud Storage access token
	      - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token
	      - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token
	    - OpenId authentication
	      - `--openid-provider-name` Open ID provider name
	      - `--openid-provider-url` Open ID provider url
	      - `--openid-client-id`Open ID client id
	      - `--openid-refresh-token` Open ID refresh token
	      - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials
	      - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials
	    - OAuth authentication
	      - `--oauth-url` OAuth provier url
	      - `--oauth-clientid` OAuth client id
	      - `--oauth-clientsecret` OAuth client secret
  - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`)
	  - `--storage-path` path to folder to store builds data
    - `--json-metadata` enable json output in store for all compact binary objects (off by default)
  - Output options for all builds commands
    - `--plain-progress` use plain line-by-line progress output
    - `--verbose`

6 files changed, 54 insertions, 43 deletions

diff --git a/src/zenserver/projectstore/buildsremoteprojectstore.cpp b/src/zenserver/projectstore/buildsremoteprojectstore.cpp
index e4e91104c..fbb9bc344 100644
--- a/src/zenserver/projectstore/buildsremoteprojectstore.cpp
+++ b/src/zenserver/projectstore/buildsremoteprojectstore.cpp
@@ -344,7 +344,12 @@ public:
 										   m_BuildId);
 			return Result;
 		}
-		Result.Blocks = std::move(Blocks.value());
+		Result.Blocks.reserve(Blocks.value().size());
+		for (ChunkBlockDescription& BlockDescription : Blocks.value())
+		{
+			Result.Blocks.push_back(ThinChunkBlockDescription{.BlockHash	  = BlockDescription.BlockHash,
+															  .ChunkRawHashes = std::move(BlockDescription.ChunkRawHashes)});
+		}
 		return Result;
 	}
 
diff --git a/src/zenserver/projectstore/fileremoteprojectstore.cpp b/src/zenserver/projectstore/fileremoteprojectstore.cpp
index 5a21a7540..98e292d91 100644
--- a/src/zenserver/projectstore/fileremoteprojectstore.cpp
+++ b/src/zenserver/projectstore/fileremoteprojectstore.cpp
@@ -192,8 +192,8 @@ public:
 			return GetKnownBlocksResult{{.ErrorCode		 = static_cast<int>(HttpResponseCode::NoContent),
 										 .ElapsedSeconds = LoadResult.ElapsedSeconds + Timer.GetElapsedTimeUs() * 1000}};
 		}
-		std::vector<ChunkBlockDescription> KnownBlocks = GetBlocksFromOplog(LoadResult.ContainerObject, ExistingBlockHashes);
-		GetKnownBlocksResult			   Result{{.ElapsedSeconds = LoadResult.ElapsedSeconds + Timer.GetElapsedTimeUs() * 1000}};
+		std::vector<ThinChunkBlockDescription> KnownBlocks = GetBlocksFromOplog(LoadResult.ContainerObject, ExistingBlockHashes);
+		GetKnownBlocksResult				   Result{{.ElapsedSeconds = LoadResult.ElapsedSeconds + Timer.GetElapsedTimeUs() * 1000}};
 		Result.Blocks = std::move(KnownBlocks);
 		return Result;
 	}
diff --git a/src/zenserver/projectstore/jupiterremoteprojectstore.cpp b/src/zenserver/projectstore/jupiterremoteprojectstore.cpp
index 2b6a437d1..e5839ad3b 100644
--- a/src/zenserver/projectstore/jupiterremoteprojectstore.cpp
+++ b/src/zenserver/projectstore/jupiterremoteprojectstore.cpp
@@ -193,7 +193,7 @@ public:
 			return GetKnownBlocksResult{{.ErrorCode		 = static_cast<int>(HttpResponseCode::NoContent),
 										 .ElapsedSeconds = LoadResult.ElapsedSeconds + ExistsResult.ElapsedSeconds}};
 		}
-		std::vector<ChunkBlockDescription> KnownBlocks = GetBlocksFromOplog(LoadResult.ContainerObject, ExistingBlockHashes);
+		std::vector<ThinChunkBlockDescription> KnownBlocks = GetBlocksFromOplog(LoadResult.ContainerObject, ExistingBlockHashes);
 
 		GetKnownBlocksResult Result{
 			{.ElapsedSeconds = LoadResult.ElapsedSeconds + ExistsResult.ElapsedSeconds + Timer.GetElapsedTimeUs() * 1000.0}};
diff --git a/src/zenserver/projectstore/projectstore.cpp b/src/zenserver/projectstore/projectstore.cpp
index f6f7eba99..53df12b14 100644
--- a/src/zenserver/projectstore/projectstore.cpp
+++ b/src/zenserver/projectstore/projectstore.cpp
@@ -8628,7 +8628,11 @@ TEST_CASE("project.store.block")
 	}
 	ChunkBlockDescription Block;
 	CompressedBuffer	  BlockBuffer = GenerateChunkBlock(std::move(Chunks), Block);
-	CHECK(IterateChunkBlock(BlockBuffer.Decompress(), [](CompressedBuffer&&, const IoHash&) {}));
+	uint64_t			  HeaderSize;
+	CHECK(IterateChunkBlock(
+		BlockBuffer.Decompress(),
+		[](CompressedBuffer&&, const IoHash&) {},
+		HeaderSize));
 }
 
 TEST_CASE("project.store.iterateoplog")
diff --git a/src/zenserver/projectstore/remoteprojectstore.cpp b/src/zenserver/projectstore/remoteprojectstore.cpp
index b4b2c6fc4..a7263da83 100644
--- a/src/zenserver/projectstore/remoteprojectstore.cpp
+++ b/src/zenserver/projectstore/remoteprojectstore.cpp
@@ -516,21 +516,23 @@ namespace remotestore_impl {
 							return;
 						}
 
-						bool StoreChunksOK = IterateChunkBlock(
-							BlockPayload,
-							[&WantedChunks, &WriteAttachmentBuffers, &WriteRawHashes, &Info](CompressedBuffer&& Chunk,
-																							 const IoHash&		AttachmentRawHash) {
-								if (WantedChunks.contains(AttachmentRawHash))
-								{
-									WriteAttachmentBuffers.emplace_back(Chunk.GetCompressed().Flatten().AsIoBuffer());
-									IoHash	 RawHash;
-									uint64_t RawSize;
-									ZEN_ASSERT(CompressedBuffer::ValidateCompressedHeader(WriteAttachmentBuffers.back(), RawHash, RawSize));
-									ZEN_ASSERT(RawHash == AttachmentRawHash);
-									WriteRawHashes.emplace_back(AttachmentRawHash);
-									WantedChunks.erase(AttachmentRawHash);
-								}
-							});
+						uint64_t BlockHeaderSize = 0;
+						bool	 StoreChunksOK	 = IterateChunkBlock(
+							  BlockPayload,
+							  [&WantedChunks, &WriteAttachmentBuffers, &WriteRawHashes, &Info](CompressedBuffer&& Chunk,
+																							   const IoHash&	  AttachmentRawHash) {
+								  if (WantedChunks.contains(AttachmentRawHash))
+								  {
+									  WriteAttachmentBuffers.emplace_back(Chunk.GetCompressed().Flatten().AsIoBuffer());
+									  IoHash   RawHash;
+									  uint64_t RawSize;
+									  ZEN_ASSERT(CompressedBuffer::ValidateCompressedHeader(WriteAttachmentBuffers.back(), RawHash, RawSize));
+									  ZEN_ASSERT(RawHash == AttachmentRawHash);
+									  WriteRawHashes.emplace_back(AttachmentRawHash);
+									  WantedChunks.erase(AttachmentRawHash);
+								  }
+							  },
+							  BlockHeaderSize);
 
 						if (!StoreChunksOK)
 						{
@@ -1101,11 +1103,11 @@ GetBlockHashesFromOplog(CbObjectView ContainerObject)
 	return BlockHashes;
 }
 
-std::vector<ChunkBlockDescription>
+std::vector<ThinChunkBlockDescription>
 GetBlocksFromOplog(CbObjectView ContainerObject, std::span<const IoHash> IncludeBlockHashes)
 {
 	using namespace std::literals;
-	std::vector<ChunkBlockDescription>	   Result;
+	std::vector<ThinChunkBlockDescription> Result;
 	CbArrayView							   BlocksArray = ContainerObject["blocks"sv].AsArrayView();
 	tsl::robin_set<IoHash, IoHash::Hasher> IncludeSet;
 	IncludeSet.insert(IncludeBlockHashes.begin(), IncludeBlockHashes.end());
@@ -1128,7 +1130,7 @@ GetBlocksFromOplog(CbObjectView ContainerObject, std::span<const IoHash> Include
 			{
 				ChunkHashes.push_back(ChunkField.AsHash());
 			}
-			Result.push_back({.BlockHash = BlockHash, .ChunkHashes = std::move(ChunkHashes)});
+			Result.push_back(ThinChunkBlockDescription{.BlockHash = BlockHash, .ChunkRawHashes = std::move(ChunkHashes)});
 		}
 	}
 	return Result;
@@ -1144,7 +1146,7 @@ BuildContainer(CidStore&																	ChunkStore,
 			   bool																			BuildBlocks,
 			   bool																			IgnoreMissingAttachments,
 			   bool																			AllowChunking,
-			   const std::vector<ChunkBlockDescription>&									KnownBlocks,
+			   const std::vector<ThinChunkBlockDescription>&								KnownBlocks,
 			   WorkerThreadPool&															WorkerPool,
 			   const std::function<void(CompressedBuffer&&, ChunkBlockDescription&&)>&		AsyncOnBlock,
 			   const std::function<void(const IoHash&, TGetAttachmentBufferFunc&&)>&		OnLargeAttachment,
@@ -1386,7 +1388,7 @@ BuildContainer(CidStore&																	ChunkStore,
 			return {};
 		}
 
-		auto FindReuseBlocks = [](const std::vector<ChunkBlockDescription>&			KnownBlocks,
+		auto FindReuseBlocks = [](const std::vector<ThinChunkBlockDescription>&		KnownBlocks,
 								  const std::unordered_set<IoHash, IoHash::Hasher>& Attachments,
 								  JobContext*										OptionalContext) -> std::vector<size_t> {
 			std::vector<size_t> ReuseBlockIndexes;
@@ -1399,14 +1401,14 @@ BuildContainer(CidStore&																	ChunkStore,
 
 				for (size_t KnownBlockIndex = 0; KnownBlockIndex < KnownBlocks.size(); KnownBlockIndex++)
 				{
-					const ChunkBlockDescription& KnownBlock			  = KnownBlocks[KnownBlockIndex];
-					size_t						 BlockAttachmentCount = KnownBlock.ChunkHashes.size();
+					const ThinChunkBlockDescription& KnownBlock			  = KnownBlocks[KnownBlockIndex];
+					size_t							 BlockAttachmentCount = KnownBlock.ChunkRawHashes.size();
 					if (BlockAttachmentCount == 0)
 					{
 						continue;
 					}
 					size_t FoundAttachmentCount = 0;
-					for (const IoHash& KnownHash : KnownBlock.ChunkHashes)
+					for (const IoHash& KnownHash : KnownBlock.ChunkRawHashes)
 					{
 						if (Attachments.contains(KnownHash))
 						{
@@ -1447,8 +1449,8 @@ BuildContainer(CidStore&																	ChunkStore,
 		std::vector<size_t> ReusedBlockIndexes	  = FindReuseBlocks(KnownBlocks, FoundHashes, OptionalContext);
 		for (size_t KnownBlockIndex : ReusedBlockIndexes)
 		{
-			const ChunkBlockDescription& KnownBlock = KnownBlocks[KnownBlockIndex];
-			for (const IoHash& KnownHash : KnownBlock.ChunkHashes)
+			const ThinChunkBlockDescription& KnownBlock = KnownBlocks[KnownBlockIndex];
+			for (const IoHash& KnownHash : KnownBlock.ChunkRawHashes)
 			{
 				if (UploadAttachments.erase(KnownHash) == 1)
 				{
@@ -1784,8 +1786,8 @@ BuildContainer(CidStore&																	ChunkStore,
 		std::vector<size_t> ReusedBlockFromChunking = FindReuseBlocks(KnownBlocks, ChunkedHashes, OptionalContext);
 		for (size_t KnownBlockIndex : ReusedBlockIndexes)
 		{
-			const ChunkBlockDescription& KnownBlock = KnownBlocks[KnownBlockIndex];
-			for (const IoHash& KnownHash : KnownBlock.ChunkHashes)
+			const ThinChunkBlockDescription& KnownBlock = KnownBlocks[KnownBlockIndex];
+			for (const IoHash& KnownHash : KnownBlock.ChunkRawHashes)
 			{
 				if (ChunkedHashes.erase(KnownHash) == 1)
 				{
@@ -1803,7 +1805,7 @@ BuildContainer(CidStore&																	ChunkStore,
 			Blocks.reserve(ReuseBlockCount);
 			for (auto It = ReusedBlockIndexes.begin(); It != UniqueKnownBlocksEnd; It++)
 			{
-				Blocks.push_back(KnownBlocks[*It]);
+				Blocks.push_back({KnownBlocks[*It]});
 			}
 			remotestore_impl::ReportMessage(OptionalContext,
 											fmt::format("Reused {} attachments from {} blocks", ReusedAttachmentCount, ReuseBlockCount));
@@ -1919,9 +1921,9 @@ BuildContainer(CidStore&																	ChunkStore,
 				  {
 					  // We can share the lock as we are not resizing the vector and only touch BlockHash at our own index
 					  RwLock::SharedLockScope _(BlocksLock);
-					  Blocks[BlockIndex].ChunkHashes.insert(Blocks[BlockIndex].ChunkHashes.end(),
-															BlockAttachmentHashes.begin(),
-															BlockAttachmentHashes.end());
+					  Blocks[BlockIndex].ChunkRawHashes.insert(Blocks[BlockIndex].ChunkRawHashes.end(),
+															   BlockAttachmentHashes.begin(),
+															   BlockAttachmentHashes.end());
 				  }
 				  uint64_t NowMS = Timer.GetElapsedTimeMs();
 				  ZEN_INFO("Assembled block {} with {} chunks in {} ({})",
@@ -2167,7 +2169,7 @@ BuildContainer(CidStore&																	ChunkStore,
 			{
 				for (const ChunkBlockDescription& B : Blocks)
 				{
-					ZEN_ASSERT(!B.ChunkHashes.empty());
+					ZEN_ASSERT(!B.ChunkRawHashes.empty());
 					if (BuildBlocks)
 					{
 						ZEN_ASSERT(B.BlockHash != IoHash::Zero);
@@ -2177,7 +2179,7 @@ BuildContainer(CidStore&																	ChunkStore,
 							OplogContinerWriter.AddBinaryAttachment("rawhash"sv, B.BlockHash);
 							OplogContinerWriter.BeginArray("chunks"sv);
 							{
-								for (const IoHash& RawHash : B.ChunkHashes)
+								for (const IoHash& RawHash : B.ChunkRawHashes)
 								{
 									OplogContinerWriter.AddHash(RawHash);
 								}
@@ -2193,7 +2195,7 @@ BuildContainer(CidStore&																	ChunkStore,
 					{
 						OplogContinerWriter.BeginArray("chunks"sv);
 						{
-							for (const IoHash& RawHash : B.ChunkHashes)
+							for (const IoHash& RawHash : B.ChunkRawHashes)
 							{
 								OplogContinerWriter.AddBinaryAttachment(RawHash);
 							}
@@ -2389,7 +2391,7 @@ SaveOplog(CidStore&				 ChunkStore,
 		OnBlock = UploadBlock;
 	}
 
-	std::vector<ChunkBlockDescription> KnownBlocks;
+	std::vector<ThinChunkBlockDescription> KnownBlocks;
 
 	uint64_t TransferWallTimeMS = 0;
 
diff --git a/src/zenserver/projectstore/remoteprojectstore.h b/src/zenserver/projectstore/remoteprojectstore.h
index 1ef0416b7..1210afc7c 100644
--- a/src/zenserver/projectstore/remoteprojectstore.h
+++ b/src/zenserver/projectstore/remoteprojectstore.h
@@ -66,7 +66,7 @@ public:
 
 	struct GetKnownBlocksResult : public Result
 	{
-		std::vector<ChunkBlockDescription> Blocks;
+		std::vector<ThinChunkBlockDescription> Blocks;
 	};
 
 	struct RemoteStoreInfo
@@ -166,7 +166,7 @@ RemoteProjectStore::Result LoadOplog(CidStore&			  ChunkStore,
 									 bool				  CleanOplog,
 									 JobContext*		  OptionalContext);
 
-std::vector<IoHash>				   GetBlockHashesFromOplog(CbObjectView ContainerObject);
-std::vector<ChunkBlockDescription> GetBlocksFromOplog(CbObjectView ContainerObject, std::span<const IoHash> IncludeBlockHashes);
+std::vector<IoHash>					   GetBlockHashesFromOplog(CbObjectView ContainerObject);
+std::vector<ThinChunkBlockDescription> GetBlocksFromOplog(CbObjectView ContainerObject, std::span<const IoHash> IncludeBlockHashes);
 
 }  // namespace zen