diff options
| author | Dan Engelbrecht <[email protected]> | 2022-12-07 11:21:41 +0100 |
|---|---|---|
| committer | GitHub <[email protected]> | 2022-12-07 02:21:41 -0800 |
| commit | 100c8f966b1c5b2fb190748f0177600562d1c5fe (patch) | |
| tree | fc85e350dea47330149a1d42eb7a6c7ae0a06111 /zenstore/compactcas.cpp | |
| parent | Cache request record/replay (#198) (diff) | |
| download | zen-100c8f966b1c5b2fb190748f0177600562d1c5fe.tar.xz zen-100c8f966b1c5b2fb190748f0177600562d1c5fe.zip | |
optimizations (#200)
* Use direct file read and direct buffer allocation for small IoBuffer materalization
* Reduce range of materialized data in CompositeBuffer reading
CompressedBuffer header reading often only need a small part and not the whole file
* reduce lock contention in IoBuffer::Materialize
* Reduce parsing of compressed headers
Validate header type at decompression
* faster CreateDirectories - start from leaf going up and recurse back
* optimized BufferHeader::IsValid
* Add ValidateCompressedHeader to use when we don't need the actual compressed data
Validate that we always get compressed data in CidStore::AddChunk
* changelog
Diffstat (limited to 'zenstore/compactcas.cpp')
| -rw-r--r-- | zenstore/compactcas.cpp | 13 |
1 files changed, 9 insertions, 4 deletions
diff --git a/zenstore/compactcas.cpp b/zenstore/compactcas.cpp index 7507a82f6..8679eb95e 100644 --- a/zenstore/compactcas.cpp +++ b/zenstore/compactcas.cpp @@ -269,9 +269,11 @@ CasContainerStrategy::Scrub(ScrubContext& Ctx) } IoBuffer Buffer(IoBuffer::Wrap, Data, Size); - if (CompressedBuffer Compressed = CompressedBuffer::FromCompressed(SharedBuffer(Buffer)); Compressed) + IoHash RawHash; + uint64_t RawSize; + if (CompressedBuffer::ValidateCompressedHeader(Buffer, RawHash, RawSize)) { - if (IoHash::FromBLAKE3(Compressed.GetRawHash()) != Hash) + if (RawHash != Hash) { // Hash mismatch BadKeys.push_back(Hash); @@ -295,10 +297,13 @@ CasContainerStrategy::Scrub(ScrubContext& Ctx) const IoHash& Hash = ChunkIndexToChunkHash[ChunkIndex]; IoBuffer Buffer(IoBuffer::BorrowedFile, File.GetBasicFile().Handle(), Offset, Size); + + IoHash RawHash; + uint64_t RawSize; // TODO: Add API to verify compressed buffer without having to memorymap the whole file - if (CompressedBuffer Compressed = CompressedBuffer::FromCompressed(SharedBuffer(Buffer)); Compressed) + if (CompressedBuffer::ValidateCompressedHeader(Buffer, RawHash, RawSize)) { - if (IoHash::FromBLAKE3(Compressed.GetRawHash()) != Hash) + if (RawHash != Hash) { // Hash mismatch BadKeys.push_back(Hash); |