builds upload command (#278)

- Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API
  - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**)
  - `builds upload` upload a folder to Cloud Build API
	  - `--local-path` source folder to upload
	  - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given
	  - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated
	  - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated
	  - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path`
	  - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set
	  - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set
	  - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks
	  - `--allow-multipart` enable usage of multi-part http upload requests
	  - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path`
  - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**)
	  - `--local-path` target folder to download to
	  - `--build-id` an Oid in hex form for the Build identifier to use
	  - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name`
	  - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path`
	  - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET)
	  - `--allow-multipart` enable usage of multi-part http download reqeusts
  - `builds diff` download a folder from Cloud Build API
	- `--local-path` target folder to download to
	- `--compare-path` folder to compare target with
	- `--only-chunked` compare only files that would be chunked
  - `builds fetch-blob` fetch and validate a blob from remote store
	  - `--build-id` an Oid in hex form for the Build identifier to use
    - `--blob-hash` an IoHash in hex form identifying the blob to download
  - `builds validate part` fetch a build part and validate all referenced attachments
	  - `--build-id` an Oid in hex form for the Build identifier to use
	  - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name`
	  - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id`
  - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations
	  - `--local-path` source folder to upload
  - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`)
	  - `--url` Cloud Builds URL
    - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake
    - `--namespace` Builds Storage namespace
    - `--bucket` Builds Storage bucket
    - Authentication options for Cloud Build API
	    - Auth token
	      - `--access-token` http auth Cloud Storage access token
	      - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token
	      - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token
	    - OpenId authentication
	      - `--openid-provider-name` Open ID provider name
	      - `--openid-provider-url` Open ID provider url
	      - `--openid-client-id`Open ID client id
	      - `--openid-refresh-token` Open ID refresh token
	      - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials
	      - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials
	    - OAuth authentication
	      - `--oauth-url` OAuth provier url
	      - `--oauth-clientid` OAuth client id
	      - `--oauth-clientsecret` OAuth client secret
  - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`)
	  - `--storage-path` path to folder to store builds data
    - `--json-metadata` enable json output in store for all compact binary objects (off by default)
  - Output options for all builds commands
    - `--plain-progress` use plain line-by-line progress output
    - `--verbose`

1 files changed, 140 insertions, 0 deletions

diff --git a/src/zencore/filesystem.cpp b/src/zencore/filesystem.cpp
index 5716d1255..8279fb952 100644
--- a/src/zencore/filesystem.cpp
+++ b/src/zencore/filesystem.cpp
@@ -1469,6 +1469,36 @@ GetModificationTickFromHandle(void* NativeHandle, std::error_code& Ec)
 	return 0;
 }
 
+uint64_t
+GetModificationTickFromPath(const std::filesystem::path& Filename)
+{
+	// PathFromHandle
+	void* Handle;
+#if ZEN_PLATFORM_WINDOWS
+	Handle = CreateFileW(Filename.c_str(), GENERIC_READ, FILE_SHARE_READ, nullptr, OPEN_EXISTING, 0, nullptr);
+	if (Handle == INVALID_HANDLE_VALUE)
+	{
+		ThrowLastError(fmt::format("Failed to open file {} to check modification tick.", Filename));
+	}
+	auto _ = MakeGuard([Handle]() { CloseHandle(Handle); });
+#else
+	int Fd = open(Filename.c_str(), O_RDONLY | O_CLOEXEC);
+	if (Fd <= 9)
+	{
+		ThrowLastError(fmt::format("Failed to open file {} to check modification tick.", Filename));
+	}
+	Handle	  = (void*)uintptr_t(Fd);
+	auto	_ = MakeGuard([Handle]() { close(int(uintptr_t(Handle))); });
+#endif
+	std::error_code Ec;
+	uint64_t		ModificatonTick = GetModificationTickFromHandle(Handle, Ec);
+	if (Ec)
+	{
+		ThrowSystemError(Ec.value(), Ec.message());
+	}
+	return ModificatonTick;
+}
+
 std::filesystem::path
 GetRunningExecutablePath()
 {
@@ -1895,6 +1925,116 @@ PickDefaultSystemRootDirectory()
 #endif	// ZEN_PLATFORM_WINDOWS
 }
 
+#if ZEN_PLATFORM_WINDOWS
+
+uint32_t
+GetFileAttributes(const std::filesystem::path& Filename)
+{
+	DWORD Attributes = ::GetFileAttributes(Filename.native().c_str());
+	if (Attributes == INVALID_FILE_ATTRIBUTES)
+	{
+		ThrowLastError(fmt::format("failed to get attributes of file {}", Filename));
+	}
+	return (uint32_t)Attributes;
+}
+
+void
+SetFileAttributes(const std::filesystem::path& Filename, uint32_t Attributes)
+{
+	if (::SetFileAttributes(Filename.native().c_str(), Attributes) == 0)
+	{
+		ThrowLastError(fmt::format("failed to set attributes of file {}", Filename));
+	}
+}
+
+#endif	// ZEN_PLATFORM_WINDOWS
+
+#if ZEN_PLATFORM_LINUX || ZEN_PLATFORM_MAC
+
+uint32_t
+GetFileMode(const std::filesystem::path& Filename)
+{
+	struct stat Stat;
+	int			err = stat(Filename.native().c_str(), &Stat);
+	if (err)
+	{
+		ThrowLastError(fmt::format("Failed to get mode of file {}", Filename));
+	}
+	return (uint32_t)Stat.st_mode;
+}
+
+void
+SetFileMode(const std::filesystem::path& Filename, uint32_t Attributes)
+{
+	int err = chmod(Filename.native().c_str(), (mode_t)Attributes);
+	if (err)
+	{
+		ThrowLastError(fmt::format("Failed to set mode of file {}", Filename));
+	}
+}
+
+#endif	// ZEN_PLATFORM_LINUX || ZEN_PLATFORM_MAC
+
+#if ZEN_PLATFORM_WINDOWS
+const uint32_t FileAttributesSystemReadOnlyFlag = FILE_ATTRIBUTE_READONLY;
+#else
+const uint32_t FileAttributesSystemReadOnlyFlag = 0x00000001;
+#endif	// ZEN_PLATFORM_WINDOWS
+
+const uint32_t FileModeWriteEnableFlags = 0222;
+
+bool
+IsFileAttributeReadOnly(uint32_t FileAttributes)
+{
+#if ZEN_PLATFORM_WINDOWS
+	return (FileAttributes & FileAttributesSystemReadOnlyFlag) != 0;
+#else
+	return (FileAttributes & 0x00000001) != 0;
+#endif	// ZEN_PLATFORM_WINDOWS
+}
+
+bool
+IsFileModeReadOnly(uint32_t FileMode)
+{
+	return (FileMode & FileModeWriteEnableFlags) == 0;
+}
+
+uint32_t
+MakeFileAttributeReadOnly(uint32_t FileAttributes, bool ReadOnly)
+{
+	return ReadOnly ? (FileAttributes | FileAttributesSystemReadOnlyFlag) : (FileAttributes & ~FileAttributesSystemReadOnlyFlag);
+}
+
+uint32_t
+MakeFileModeReadOnly(uint32_t FileMode, bool ReadOnly)
+{
+	return ReadOnly ? (FileMode & ~FileModeWriteEnableFlags) : (FileMode | FileModeWriteEnableFlags);
+}
+
+bool
+SetFileReadOnly(const std::filesystem::path& Filename, bool ReadOnly)
+{
+#if ZEN_PLATFORM_WINDOWS
+	uint32_t CurrentAttributes = GetFileAttributes(Filename);
+	uint32_t NewAttributes	   = MakeFileAttributeReadOnly(CurrentAttributes, ReadOnly);
+	if (CurrentAttributes != NewAttributes)
+	{
+		SetFileAttributes(Filename, NewAttributes);
+		return true;
+	}
+#endif	// ZEN_PLATFORM_WINDOWS
+#if ZEN_PLATFORM_LINUX || ZEN_PLATFORM_MAC
+	uint32_t CurrentMode = GetFileMode(Filename);
+	uint32_t NewMode	 = MakeFileModeReadOnly(CurrentMode, ReadOnly);
+	if (CurrentMode != NewMode)
+	{
+		SetFileMode(Filename, NewMode);
+		return true;
+	}
+#endif	// ZEN_PLATFORM_LINUX || ZEN_PLATFORM_MAC
+	return false;
+}
+
 //////////////////////////////////////////////////////////////////////////
 //
 // Testing related code follows...