aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* precommitzousar2024-04-041-1/+6
|
* Add optional gcpath param to import-oplog cmdzousar2024-04-042-2/+10
|
* Categorize the clean argument on oplog-import as a general argzousar2024-04-042-3/+3
|
* improved assert (#37)Dan Engelbrecht2024-04-0445-167/+591
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* hardening parsepackagemessage (#38)Dan Engelbrecht2024-04-042-37/+74
| | | * hardening of ParsePackageMessage and extended details when malformed attachments are detected
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-034-20/+34
| | | * Validate size of found chunks in cas/cache
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-037-211/+114
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* compressed header reading opt (#33)Dan Engelbrecht2024-04-031-52/+86
| | | * refactor so we don't have to re-read data from source to get block sizes
* Fix sentry using wrong folder path when data path contains non-anscii ↵Dan Engelbrecht2024-03-281-2/+2
| | | | | characters (#32) * Fix sentry using wrong folder path when data path contains non-ascii characters
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-288-68/+198
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* add "fieldnames" query param for GetProjectFiles/GetProjectChunkInfos (#29)Dan Engelbrecht2024-03-283-23/+137
| | | | | | | | - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/chunkinfos` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `rawhash` and `rawsize` (translates to `?fieldnames=id,rawhash,rawsize`) - Use `?fieldnames=*` to get all the fields - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/files` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `clientpath` and `serverpath` (translates to `?fieldnames=id,clientpath,serverpath`), `filter=client` only applies if `fieldnames` is not given as a parameter - Use `?fieldnames=*` to get all the fields
* Faster reading of compressed buffer headers by not materializing entire ↵Dan Engelbrecht2024-03-281-22/+52
| | | | source buffer (#28)
* Get raw size for compressed chunks correctly for ↵Dan Engelbrecht2024-03-271-1/+7
| | | | `/prj/{project}/oplog/{log}/chunkinfos` (#27)
* make existing block log clearerDan Engelbrecht2024-03-261-1/+1
|
* fix order of parameters in block check result message (#26)Dan Engelbrecht2024-03-261-1/+1
| | | * fix order of parametes in block check result message
* add yaml serialization support (#3)Stefan Boberg2024-03-267-783/+1201
| | | | | | | this change adds serialization of payloads as YAML, but not parsing. The implementation is somewhat based on the JSON path, and may be collapsed eventually as it is possible to serialize JSON format using the same code it also separates out the JSON serialization into a separate file for ease of maintenance any HTTP request response may be formatted as yaml by using a `.yaml` suffix or an `Accept: text/yaml` header
* add filter to projectstore entries request (#25)Dan Engelbrecht2024-03-263-6/+101
| | | | | | | * Add HttpServerRequest::Decode to decode http request parameters * Add support to filter projectstore `entries` request using the `fieldfilter` where the wanted fields are comma (,) delimited * Add support for responding with compressed payloads for projectstore `entries` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `files` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `chunkinfo` requests by adding AcceptType `compressed-binary` to the request header
* remove redundant json11 includesStefan Boberg2024-03-262-2/+0
|
* consistent paths encoding (#24)Dan Engelbrecht2024-03-255-50/+52
| | | | * Don't encode filesystem path to UTF8 unless stored in compactbinary string * Be consistent where we encode/decode paths to UTF8
* add a limit to the number of times we attempt to finalize (#22)Dan Engelbrecht2024-03-253-84/+137
| | | | | | - Improvement: Add limit to the number of times we attempt to finalize and exported oplog - Improvement: Switch to large thread pool when executing oplog export/import - Improvement: Clean up reporting of missing attachments in oplog export/import - Improvement: Remove double-reporting of abort reason for oplog export/import
* use batch request for checking existing blocks as Jupiter is now fixed (#20)v5.4.2-pre15v5.4.2-pre14Dan Engelbrecht2024-03-231-41/+10
|
* re-enable partial cache chunks (#21)v5.4.2-pre9v5.4.2-pre12v5.4.2-pre11v5.4.2-pre10Dan Engelbrecht2024-03-224-28/+40
| | | | * Separate chunk raw hash from section hash (how to find the fragment attachment) * fix partial get cache value tests
* check existance of reused blocks (#18)Dan Engelbrecht2024-03-225-43/+167
| | | | | * Add HasAttachments to RemoteProjectStore so we can query if attachment blocks actually exist * use individual requests for compressed blob check in Jupiter * remove weird 1000.5 to 1000.0 when converting seconds to milliseconds
* disable partial getcachechunk responses (#19)Dan Engelbrecht2024-03-221-1/+7
|
* harden attach sponsor process (#14)Dan Engelbrecht2024-03-217-46/+70
| | | | | - Improvement: Delay exiting due to no sponsor processes by one second to handle race conditions - Improvement: Safer IsProcessRunning check - Improvement: make sure we can RequestApplicationExit safely from any thread
* add support for responding with partial cache chunks (#11)Dan Engelbrecht2024-03-216-17/+268
| | | * add support for responding with partial cache chunks
* improved process monitoring behaviour with invalid pids (#16)Stefan Boberg2024-03-214-14/+42
|
* non memory copy compressed range (#13)Dan Engelbrecht2024-03-206-44/+277
| | | | | * Add CompressedBuffer::GetRange that references source data rather than make a memory copy * Use Compressed.CopyRange in project store GetChunkRange * docs for CompressedBuffer::CopyRange and CompressedBuffer::GetRange
* read jupiter token from file (#10)Dan Engelbrecht2024-03-183-5/+65
| | | | | * Add ability to specify a json file for cloud access token OidcToken.exe in UE can be asked to produce such a file with the -OutFile option * avoid division by zero when reporting progress
* add a retry when trying to create the lock file (#9)Dan Engelbrecht2024-03-181-2/+8
| | | * add a retry when trying to create the lock file if UE is trying to read it at the same time
* make sure zenserver reacts and exist on SIGTERM signal (#8)Dan Engelbrecht2024-03-152-0/+7
| | | | | * make sure zenserver reacts and exist on SIGTERM signal * add zen tag to all runners * temp disable mac codesigning
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-1415-654/+1950
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* clean up test linking (#4)Dan Engelbrecht2024-03-1423-632/+412
| | | | | | | - Improvement: Add zenhttp-test and zenutil-test - Improvement: Moved cachepolicy test to cachepolicy.cpp - Improvement: Renamed cachestore tests from z$ to cachestore - Improvement: Moved test linking so test for a lib is linked by <lib>-test - Improvement: Removed HttpRequestParseRelativeUri in httpstructuredcache.cpp and use the one in cacherequests.h instead
* HTTP request logging (#6)Stefan Boberg2024-03-144-31/+108
| | | this change adds support for tracing http payloads when using the asio path. This was already supported when using the `--http=plugin` path and this change moves some code into a shared class for reuse.
* fix potential partially written files (#2)Dan Engelbrecht2024-03-135-17/+47
| | | | * Make sure WriteFile() does not leave incomplete files * use TemporaryFile and MoveTemporaryIntoPlace to avoid leaving partial files on error
* http request parser safety (#664)Dan Engelbrecht2024-03-122-3/+9
| | | | * make sure we don't add more headers than we support * don't capture for loop variables by reference for async work
* fix zenserver state macos (#669)Dan Engelbrecht2024-03-103-8/+48
| | | | | | - Shared memory for zenserver state may hang around after all zenserver processes exit - make sure we find a valid entry in `zen up` before bailing - Httpasio add retry for desired port - Httpasio only call listen() once - Httpasio explicitly close acceptor sockets
* Add WriteMeasuredVarUInt to avoid measuring ints twice before writing (#665)Dan Engelbrecht2024-03-062-26/+53
| | | | Avoid double resize of buffer in CbWriter::SetName and CbWriter::AddBinary Add WriteMeasuredVarUInt to avoid measuring ints twice before writing
* some asio HTTP server tweaks for correctness (#663)Stefan Boberg2024-03-051-9/+11
|
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-287-62/+145
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* add disk caching to block move (#661)Dan Engelbrecht2024-02-271-23/+36
| | | * add disk caching to block move
* improved block store logging and more gcv2 tests (#659)Dan Engelbrecht2024-02-277-114/+215
| | | | * improved gc/blockstore logging * more gcv2 tests
* remove reference caching (#658)Dan Engelbrecht2024-02-277-698/+197
| | | * remove reference caching
* hashing fixes (#657)Dan Engelbrecht2024-02-2611-97/+173
| | | | | * move structuredcachestore tests to zenstore-test * Don't materialize entire files when hashing if it is a large files * rewrite CompositeBuffer::Mid to never materialize buffers
* adding context to http.sys error messageStefan Boberg2024-02-201-5/+15
| | | | added some context to http.sys API call error reporting
* Keep track of added ops during GCV2 instead of rescanning full oplog when ↵Dan Engelbrecht2024-02-132-8/+25
| | | | added ops are detected (#652)
* Added --copy-log, --copy-cache-log and copy-http-log option to zen logs ↵Dan Engelbrecht2024-02-125-20/+81
| | | | | | command (#651) * be safe and use ToText() rather than AsText() when getting http responses * Added `--copy-log`, `--copy-cache-log` and `copy-http-log` option to zen logs command to copy logs from a local running zen server instance
* Save compressed large attachments to temporary files on disk (#650)Dan Engelbrecht2024-02-123-189/+219
| | | | | | | | | * Save large compressed large attachments to temporary files on disk * bump oplog block max size up to 64Mb again * Make sure CompositeBuffer::AppendBuffers actually moves inputs when it should * removed parallell execution of fetching payload for block assembly it was not actually helping and added complexity * make sure we move/release payload buffers as soon as possible * make sure we don't read in full large attachments to memory when computing hash
* Add testcase and fix for ASIO ipv4 localhost support (#649)zousar2024-02-062-14/+86
| | | - Bugfix: Implement two listening sockets in ASIO (ipv4+ipv6) when either we start with `--http-forceloopback` or we resort to that mode because of a failure to bind to the "any" address
* compress large attachments on demand (#647)Dan Engelbrecht2024-02-056-390/+561
| | | | | | | - Improvement: Speed up oplog export by fetching/compressing big attachments on demand - Improvement: Speed up oplog export by batch-fetcing small attachments - Improvement: Speed up oplog import by batching writes of oplog ops - Improvement: Tweak oplog export default block size and embed size limit - Improvement: Add more messaging and progress during oplog import/export
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 00:47:03 +0000