Documentation updates (#882)

Restructured the docs folder in preparation for more docs. Improved the contents a bit.

1 files changed, 143 insertions, 0 deletions

diff --git a/docs/cache.md b/docs/cache.md
new file mode 100644
index 000000000..a916945fa
--- /dev/null
+++ b/docs/cache.md
@@ -0,0 +1,143 @@
+# Structured Cache (DDC)
+
+The Cache service is the primary storage interface for the Derived Data Cache
+(DDC). It stores and retrieves derived data produced by the Unreal Engine
+cooker — cooked assets, shader bytecode, texture compilations, and other build
+artifacts.
+
+The cache is designed to allow the client to avoid doing computationally intensive
+work by using the output produced by another machine/user or a previous local run.
+
+## Concepts
+
+### Namespace
+
+A logical partition within the cache. Each namespace (e.g. `ns_ue.ddc`) has its
+own set of buckets and independent storage accounting. Namespaces allow different
+types of cached data to be managed separately.
+
+### Bucket
+
+A subdivision within a namespace. Buckets group cache entries by the type of
+derived data they contain (e.g. `bulkdatalist`, `animationsequence`,
+`shaderbytecode`). Bucket names are lowercase alphanumeric strings.
+
+### Key
+
+Cache entries are identified by a 40-character hex hash within a bucket. The
+full key path is `{bucket}/{keyhash}`. The key itself has no prescribed
+derivation, but typically the client summarises all inputs to a transformation
+and feeds the result into a hash function such as **IoHash** (BLAKE3/160) to
+produce it.
+
+The inputs conceptually include the code performing the transformation, though
+not in a literal sense — that would be impractical. Instead, one or more version
+numbers are typically included as a proxy for the code itself.
+
+It is important that all inputs influencing the outcome of the transformation
+are accounted for. Missing inputs can lead to non-determinism in the cooker and
+editor, where stale cached results are used instead of recomputing.
+
+### Cache Record
+
+A cache record is a structured entry stored in Compact Binary format. Records
+contain metadata describing the cached result along with references to one or
+more attachments that hold the actual payload data. When a client performs a
+cache lookup, the record is returned first, and the client can then fetch the
+referenced attachments as needed.
+
+### Cache Value
+
+A cache value is an unstructured (opaque) entry stored as a raw binary blob.
+Unlike records, values do not carry structured metadata or separate attachment
+references — the entire payload is stored and retrieved as a single unit. Values
+are used for simpler caching scenarios where the overhead of structured records
+is unnecessary.
+
+### Attachment
+
+A binary payload (chunk) associated with a cache entry, identified by the
+content hash of the data. Attachments are stored in CAS and referenced by
+the cache entry. The full attachment path is `{bucket}/{keyhash}/{valuehash}`.
+
+## API
+
+**Base URI:** `/z$/`
+
+### Service and Namespace Info
+
+```
+GET /z$/                            # Service info (namespaces, storage size)
+GET /z$/{namespace}                 # Namespace info (buckets, sizes, config)
+GET /z$/{namespace}?bucketsizes=*   # Include per-bucket storage sizes
+```
+
+### Bucket Operations
+
+```
+GET    /z$/{namespace}/{bucket}     # Bucket info (size, entry count)
+DELETE /z$/{namespace}/{bucket}     # Drop all entries in a bucket
+DELETE /z$/{namespace}              # Drop entire namespace
+```
+
+### Record Operations
+
+Records are structured cache entries (Compact Binary format).
+
+```
+GET /z$/{namespace}/{bucket}/{keyhash}      # Get cached record
+PUT /z$/{namespace}/{bucket}/{keyhash}      # Store cached record
+```
+
+### Attachment Operations
+
+Attachments are binary chunks referenced by cache entries.
+
+```
+GET /z$/{namespace}/{bucket}/{keyhash}/{valuehash}  # Get attachment
+PUT /z$/{namespace}/{bucket}/{keyhash}/{valuehash}  # Store attachment
+```
+
+### Batch Operations (RPC)
+
+For high-throughput scenarios, the RPC endpoint supports batch get/put
+operations using the Compact Binary protocol:
+
+```
+POST /z$/{namespace}/$rpc           # Batch RPC operations
+```
+
+Supported RPC operations:
+
+- `PutCacheRecords` — batch store of structured records
+- `GetCacheRecords` — batch retrieve of structured records
+- `PutCacheValues` — batch store of unstructured values
+- `GetCacheValues` — batch retrieve of unstructured values
+- `GetCacheChunks` — batch retrieve of chunks with upstream fallback
+
+### Detailed Inventory
+
+```
+GET /z$/details$                                    # All cache contents
+GET /z$/details$/{namespace}                        # Namespace inventory
+GET /z$/details$/{namespace}/{bucket}               # Bucket inventory
+GET /z$/details$/{namespace}/{bucket}/{key}         # Single key details
+```
+
+Add `?csv=true` for CSV output, `?details=true` for size information, or
+`?attachmentdetails=true` for full attachment listings.
+
+## Upstream Caching
+
+The cache service supports tiered caching through upstream connections. When a
+cache miss occurs locally, the request can be forwarded to an upstream storage
+instance (e.g. a shared build farm cache). Successfully retrieved content is
+stored locally for future requests.
+
+## Web Dashboard
+
+The Cache page in the dashboard shows:
+
+- Per-namespace breakdown with disk and memory usage
+- Per-bucket entry counts and storage sizes
+- Cache hit/miss statistics