aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/remoteprojectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-031-3519/+0
| | | * move remoteproject code to remotestorelib
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-021-0/+197
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* gracefully handle missing chunks when exporting an oplog (#526)Dan Engelbrecht2025-09-291-5/+19
| | | | - Improvement: Gracefully handle missing chunks when exporting an oplog - Bugfix: Add object validation when replaying oplog to avoid crash on malformed data
* limit number of chunks per block (#508)Dan Engelbrecht2025-09-231-2/+7
| | | | | - Improvement: Chunk block generation for `zen oplog-export` and `zen builds upload` command are now limited to max 4000 chunks per block zen builds list-block testing function to check results for recently used blocks in a context
* more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-221-0/+18
| | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
* new in-memory storage strategy for oplogs (#490)Dan Engelbrecht2025-09-151-1/+1
| | | - Improvement: Revised project oplog in-memory representation which reduces load times and memory usage
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-756/+787
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-10/+13
| | | * add validation of compact binary payloads before reading them
* oplog output improvements (#474)Dan Engelbrecht2025-08-221-12/+13
| | | | | | * include session id in oplog import/export output * use Mbit/sec for transfer rates * fix elapsed time conversion in buildremotprojectstore * changelog
* long filename support (#330)Dan Engelbrecht2025-03-311-3/+3
| | | - Bugfix: Long file paths now works correctly on Windows
* builds upload command (#278)Dan Engelbrecht2025-02-261-35/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-48/+7
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* move WriteToTempFile to basicfile.h (#283)Dan Engelbrecht2025-02-121-60/+40
| | | add helper constructors to BasicFile
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-149/+77
|
* export oplog to zen target url (#274)Dan Engelbrecht2025-01-161-1/+1
| | | | * fix oplog target url for oplog export to remote zenserver * verify that chunking is allowed before chunking loose files
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-131-1/+4
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* Builds API remote project store (#258)Dan Engelbrecht2024-12-121-149/+174
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-1241/+1450
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-281-332/+419
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* fix crash on corrupt oplog block (#223)Dan Engelbrecht2024-11-211-0/+16
| | | | * bail attachment import on corrupt attachment block * changelog
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-061-41/+60
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-5/+5
| | | | blocking client requests (#134)
* stop exceptions from leaking on threaded work (#102)Dan Engelbrecht2024-08-061-16/+25
| | | | * catch exceptions in threaded work * don't abort all project file/chunk info fetch for single failure
* Use a smaller thread pool for network operations when doing oplog import to ↵Dan Engelbrecht2024-05-301-1/+1
| | | | | reduce risk NIC/router failure (#89) Medium worker pool now uses a minimum of 2 threads (up from 1)
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-261-4/+20
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* fix oplog import during gcv2 (#62)v5.5.0-pre3v5.5.0-pre2Dan Engelbrecht2024-04-251-72/+90
| | | | - Bugfix: Always pre-cache oplog when creating project store GCv2 referencer - Bugfix: Correctly capture attachments imported with oplog to void them being GCd before oplog is written
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-221-55/+84
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* fix LogRemoteStoreStatsDetails (#53)Dan Engelbrecht2024-04-201-2/+3
|
* import oplog improvements (#54)Dan Engelbrecht2024-04-201-90/+195
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* safer oplog import (#52)de/safer-oplog-importDan Engelbrecht2024-04-181-52/+120
| | | | * reference cache gc update capture * When importing oplogs we now import all attachments first and (optionally clean) write the oplog on success
* gc v2 disk freed space fix and oplog stats report improvement (#45)Dan Engelbrecht2024-04-151-77/+131
| | | | | - Bugfix: Correctly calculate size freed/data moved from blocks in GCv2 - Improvement: Reduced details in remote store stats for oplog export/import to user - Improvement: Transfer speed for oplog export/import is now an overall number rather than average of speed per single request
* remote project store stats (#44)Dan Engelbrecht2024-04-101-19/+56
| | | | | * add remote oplog store statistics * block chunking when uploading oplog to zenserver (mirroring) * make sure we can move temporary dechunked file into cas store
* improved assert (#37)Dan Engelbrecht2024-04-041-3/+3
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-031-1/+1
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-281-2/+2
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* make existing block log clearerDan Engelbrecht2024-03-261-1/+1
|
* fix order of parameters in block check result message (#26)Dan Engelbrecht2024-03-261-1/+1
| | | * fix order of parametes in block check result message
* add a limit to the number of times we attempt to finalize (#22)Dan Engelbrecht2024-03-251-81/+133
| | | | | | - Improvement: Add limit to the number of times we attempt to finalize and exported oplog - Improvement: Switch to large thread pool when executing oplog export/import - Improvement: Clean up reporting of missing attachments in oplog export/import - Improvement: Remove double-reporting of abort reason for oplog export/import
* check existance of reused blocks (#18)Dan Engelbrecht2024-03-221-24/+71
| | | | | * Add HasAttachments to RemoteProjectStore so we can query if attachment blocks actually exist * use individual requests for compressed blob check in Jupiter * remove weird 1000.5 to 1000.0 when converting seconds to milliseconds
* read jupiter token from file (#10)Dan Engelbrecht2024-03-181-5/+12
| | | | | * Add ability to specify a json file for cloud access token OidcToken.exe in UE can be asked to produce such a file with the -OutFile option * avoid division by zero when reporting progress
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-141-582/+1070
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* Save compressed large attachments to temporary files on disk (#650)Dan Engelbrecht2024-02-121-182/+216
| | | | | | | | | * Save large compressed large attachments to temporary files on disk * bump oplog block max size up to 64Mb again * Make sure CompositeBuffer::AppendBuffers actually moves inputs when it should * removed parallell execution of fetching payload for block assembly it was not actually helping and added complexity * make sure we move/release payload buffers as soon as possible * make sure we don't read in full large attachments to memory when computing hash
* compress large attachments on demand (#647)Dan Engelbrecht2024-02-051-324/+386
| | | | | | | - Improvement: Speed up oplog export by fetching/compressing big attachments on demand - Improvement: Speed up oplog export by batch-fetcing small attachments - Improvement: Speed up oplog import by batching writes of oplog ops - Improvement: Tweak oplog export default block size and embed size limit - Improvement: Add more messaging and progress during oplog import/export
* improve oplog export logging (#644)Dan Engelbrecht2024-01-311-96/+207
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-291-4/+5
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-251-21/+34
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-241-29/+29
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* oplog import/export improvements (#634)Dan Engelbrecht2024-01-231-159/+305
| | | | * improve feedback from oplog import/export * improve oplog save performance
* add --ignore-missing-attachments to oplog-import command (#637)Dan Engelbrecht2024-01-221-34/+59
|
* global thread worker pools (#577)Dan Engelbrecht2023-11-291-12/+4
| | | - Improvement: Use two global worker thread pools instead of ad-hoc creation of worker pools
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-18 04:14:59 +0000