aboutsummaryrefslogtreecommitdiff
path: root/src/zencore/compress.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Add test suites (#799)Stefan Boberg2026-03-021-0/+4
| | | | | | | | | | | | | Makes all test cases part of a test suite. Test suites are named after the module and the name of the file containing the implementation of the test. * This allows for better and more predictable filtering of which test cases to run which should also be able to reduce the time CI spends in tests since it can filter on the tests for that particular module. Also improves `xmake test` behaviour: * instead of an explicit list of projects just enumerate the test projects which are available based on build system state * also introduces logic to avoid running `xmake config` unnecessarily which would invalidate the existing build and do lots of unnecessary work since dependencies were invalidated by the updated config * also invokes build only for the chosen test targets As a bonus, also adds `xmake sln --open` which allows opening IDE after generation of solution/xmake project is done.
* fix zero length stream decompress (#685)Dan Engelbrecht2025-12-101-1/+1
| | | * fix validation during stream decompress of zero-size rawsize
* use scope guards to secure that BasicFile::Detach is called even on ↵Dan Engelbrecht2025-12-011-10/+5
| | | | exception (#672)
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-271-15/+46
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* fix stream decompress (#668)Dan Engelbrecht2025-11-271-10/+14
| | | * fix source file offset in DecompressToStream implementations
* RawOffset can be anything and we expect an empty buffer to be returned along ↵Dan Engelbrecht2025-11-261-4/+19
| | | | with RawSize = 0 if the offset was out of bounds for the value. (#666)
* fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-031-14/+36
| | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
* streaming none compressor (#414)Dan Engelbrecht2025-06-021-3/+36
| | | * add proper streaming to none compressor type
* improved post upload/download summary (#308)Dan Engelbrecht2025-03-181-50/+80
| | | | | | | | | * added ValidateStatistics and improved post upload summary * improved download statistics * smoother stats update when compressing * better feedback during stream compresss/decompress * don't capture TotalPartWriteCount by reference * disk stats cleanup * multi-test-download overall timer
* pick up existing cache (#299)Dan Engelbrecht2025-03-101-16/+35
| | | | | | - Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run - Improvement: Faster abort during stream compression - Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write - Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails
* streaming compress (#295)Dan Engelbrecht2025-03-051-2/+169
| | | | - Improvement: Validate hash of decompressed data inline with streaming decompression - Improvement: Do streaming compression of large blobs to improve memory and I/O performance
* stream decompress (#293)Dan Engelbrecht2025-03-041-0/+186
| | | | * clean up latency parameters and slow down rate updates * add DecompressToStream
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-53/+161
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* safer partial requests (#82)Dan Engelbrecht2024-05-171-34/+46
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* compressed header reading opt (#33)Dan Engelbrecht2024-04-031-52/+86
| | | * refactor so we don't have to re-read data from source to get block sizes
* non memory copy compressed range (#13)Dan Engelbrecht2024-03-201-0/+192
| | | | | * Add CompressedBuffer::GetRange that references source data rather than make a memory copy * Use Compressed.CopyRange in project store GetChunkRange * docs for CompressedBuffer::CopyRange and CompressedBuffer::GetRange
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-141-9/+21
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* header only compressed buffers are valid (#604)Dan Engelbrecht2023-12-121-2/+2
| | | - Bugfix: Allow attachments that contains a raw size of zero
* streaming decompression support (#142)Stefan Boberg2023-05-231-20/+786
| | | Added CompressedBufferReader support from UE. This provides some streaming decompression support which can be employed to reduce memory and other resource usage.
* cleaned up how Oodle is incorporatedStefan Boberg2023-05-021-4/+1
| | | | changed win64 to be more in line with other platforms wrt xmake config
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-0/+1353
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-16 20:45:54 +0000