aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp
Commit message (Collapse)AuthorAgeFilesLines
* review fixesDmytro Ivanov2025-04-221-1/+1
|
* Added config, versioning and logging for pluginsDmytro Ivanov2025-04-223-106/+167
|
* filesystem retry fixes (#354)Dan Engelbrecht2025-04-101-5/+5
| | | | | | | | | * add more forgiving retries with filesystem * fall back to FindFirstFile if access prevents us from using GetFileAttributes * only validate hash if we have a complete payload in http client * changelog
* use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-023-1/+113
| | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
* long filename support (#330)Dan Engelbrecht2025-03-311-1/+1
| | | - Bugfix: Long file paths now works correctly on Windows
* optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-1/+1
| | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* zen build cache service (#318)Dan Engelbrecht2025-03-263-47/+70
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* build list filters (#313)Dan Engelbrecht2025-03-192-3/+3
| | | | | - Feature: `zen builds list` command has new options - `--query-path` - path to a .json (json format) or .cbo (compact binary object format) with the search query to use - `--result-path` - path to a .json (json format) or .cbo (compact binary object format) to write output result to, if omitted json format will be output to console
* Build command tweaks (#301)Stefan Boberg2025-03-111-7/+10
| | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* partial block fetch (#298)Dan Engelbrecht2025-03-071-66/+95
| | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-064-18/+38
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* limit and validate responses before logging the text (#292)Stefan Boberg2025-03-042-1/+93
| | | Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
* refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-031-0/+5
| | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-4/+23
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* improved builds api interface in jupiter (#281)Dan Engelbrecht2025-02-123-140/+178
| | | | | * multipart upload/download iterface in jupiter * review fixes
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-222-0/+105
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-168-7/+1066
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-134-4/+8
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* more memory tagging and fixes (#263)Stefan Boberg2024-12-162-4/+94
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Memory tracking improvements (#262)Stefan Boberg2024-12-111-0/+14
| | | | | * added LLM tag to properly tag RPC allocations * annotated some more httpsys functions with memory tags * only emit memory scope events if the active tag is different from the new tag
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-18/+60
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* auth fixes (#260)Dan Engelbrecht2024-12-103-32/+43
| | | | * fix so we can replace an openid provider that was read from disk file * fix OidcClient lifetime issues in authmg
* Unity build fixes (#253)Stefan Boberg2024-12-051-8/+0
| | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+19
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-281-2/+2
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* add validation of payload responses in http client (#240)Dan Engelbrecht2024-11-271-220/+292
| | | if response payload does not validate properly do a retry if applicable
* workspace share security (#192)Dan Engelbrecht2024-10-231-1/+4
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* safer path from handle (#195)Dan Engelbrecht2024-10-161-2/+12
| | | * remove PathFromHandle that throws to give better context on failures
* made fmt formatter format function const (#162)Stefan Boberg2024-09-201-2/+2
| | | this appears to be required as of fmt v11
* Add zenserver session id to Sentry context (#133)Dan Engelbrecht2024-08-221-3/+1
|
* don't assert that we have moved bytes if source block is zero size (#97)Dan Engelbrecht2024-06-142-2/+2
| | | | * don't assert that we have moved bytes if source block is zero size * handle invalid session ids gracefully
* workspace shares (#84)Dan Engelbrecht2024-05-292-0/+20
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-1/+2
| | | * use write and move in place for safer writing of files
* import oplog improvements (#54)Dan Engelbrecht2024-04-201-70/+158
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* improved assert (#37)Dan Engelbrecht2024-04-048-25/+56
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-032-18/+40
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* add yaml serialization support (#3)Stefan Boberg2024-03-261-0/+10
| | | | | | | this change adds serialization of payloads as YAML, but not parsing. The implementation is somewhat based on the JSON path, and may be collapsed eventually as it is possible to serialize JSON format using the same code it also separates out the JSON serialization into a separate file for ease of maintenance any HTTP request response may be formatted as yaml by using a `.yaml` suffix or an `Accept: text/yaml` header
* add filter to projectstore entries request (#25)Dan Engelbrecht2024-03-262-0/+36
| | | | | | | * Add HttpServerRequest::Decode to decode http request parameters * Add support to filter projectstore `entries` request using the `fieldfilter` where the wanted fields are comma (,) delimited * Add support for responding with compressed payloads for projectstore `entries` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `files` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `chunkinfo` requests by adding AcceptType `compressed-binary` to the request header
* HTTP request logging (#6)Stefan Boberg2024-03-144-31/+108
| | | this change adds support for tracing http payloads when using the asio path. This was already supported when using the `--http=plugin` path and this change moves some code into a shared class for reuse.
* http request parser safety (#664)Dan Engelbrecht2024-03-121-1/+7
| | | | * make sure we don't add more headers than we support * don't capture for loop variables by reference for async work
* fix zenserver state macos (#669)Dan Engelbrecht2024-03-101-5/+24
| | | | | | - Shared memory for zenserver state may hang around after all zenserver processes exit - make sure we find a valid entry in `zen up` before bailing - Httpasio add retry for desired port - Httpasio only call listen() once - Httpasio explicitly close acceptor sockets
* some asio HTTP server tweaks for correctness (#663)Stefan Boberg2024-03-051-9/+11
|
* adding context to http.sys error messageStefan Boberg2024-02-201-5/+15
| | | | added some context to http.sys API call error reporting
* Add testcase and fix for ASIO ipv4 localhost support (#649)zousar2024-02-061-14/+59
| | | - Bugfix: Implement two listening sockets in ASIO (ipv4+ipv6) when either we start with `--http-forceloopback` or we resort to that mode because of a failure to bind to the "any" address
* Actually thow exception if we can't parse the jobid when starting async job ↵Dan Engelbrecht2024-01-311-2/+7
| | | | | | (#645) Return system error code on exception throw in zen command Clean up HttpClient::Response::ErrorMessage to remove redundant ": "
* improve oplog export logging (#644)Dan Engelbrecht2024-01-311-9/+9
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-291-1/+1
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-251-1/+4
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Use proper format for range request header (#640)Dan Engelbrecht2024-01-241-4/+10
| | | | Clear header callback after use Use separate temp-vector for headers
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-242-141/+424
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 07:21:16 +0000