aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp
Commit message (Collapse)AuthorAgeFilesLines
* make sure we properly shut down any pending http requests at exitDan Engelbrecht2025-08-201-0/+1
|
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-122-3/+27
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* list build part content (#462)Dan Engelbrecht2025-08-112-6/+20
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* de/stringbuilder safety (#456)Dan Engelbrecht2025-08-051-1/+1
| | | | | | - Improvement: Safeguard FormatCallstack to not throw exceptions when building the callstack string - Improvement: Limit thread name length when setting it for debugger use - Improvemnet: Don't allow assert callbacks to throw exception - Improvement: When formatting log output for malformed attachments in a package message, allow the string buffer to grow instead of throwing exception
* crash in composite buffer stream (#431)Dan Engelbrecht2025-06-101-1/+1
| | | * fix BufferedReadFileStream calculating buffer end wrong
* use unique tmp name for auth token file (#426)Dan Engelbrecht2025-06-091-3/+3
| | | * use Oid to generate unique name since std::tmpnam is not good practice
* http client streaming upload (#413)Dan Engelbrecht2025-06-021-27/+283
| | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* handle exception with batch work (#401)Dan Engelbrecht2025-05-191-2/+4
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* extend log on failed httpsys response (#394)Dan Engelbrecht2025-05-132-8/+22
| | | | | * extend log on failed httpsys response * fix formatting for "Desired port is in use, retrying" * add warning log if port is remapped
* close servers in multi server (#371)Dmytro Ivanov2025-04-241-0/+4
|
* add retry on internal error / bad gateway (#370)Dan Engelbrecht2025-04-241-1/+3
| | | * do http client retry on internal error and bad gateway
* Make plugin loading errors non fatal (#364)Dmytro Ivanov2025-04-233-29/+46
| | | | make plugin loading errors non fatal
* review fixesDmytro Ivanov2025-04-221-1/+1
|
* Added config, versioning and logging for pluginsDmytro Ivanov2025-04-223-106/+167
|
* filesystem retry fixes (#354)Dan Engelbrecht2025-04-101-5/+5
| | | | | | | | | * add more forgiving retries with filesystem * fall back to FindFirstFile if access prevents us from using GetFileAttributes * only validate hash if we have a complete payload in http client * changelog
* use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-023-1/+113
| | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
* long filename support (#330)Dan Engelbrecht2025-03-311-1/+1
| | | - Bugfix: Long file paths now works correctly on Windows
* optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-1/+1
| | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* zen build cache service (#318)Dan Engelbrecht2025-03-263-47/+70
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* build list filters (#313)Dan Engelbrecht2025-03-192-3/+3
| | | | | - Feature: `zen builds list` command has new options - `--query-path` - path to a .json (json format) or .cbo (compact binary object format) with the search query to use - `--result-path` - path to a .json (json format) or .cbo (compact binary object format) to write output result to, if omitted json format will be output to console
* Build command tweaks (#301)Stefan Boberg2025-03-111-7/+10
| | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* partial block fetch (#298)Dan Engelbrecht2025-03-071-66/+95
| | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-064-18/+38
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* limit and validate responses before logging the text (#292)Stefan Boberg2025-03-042-1/+93
| | | Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
* refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-031-0/+5
| | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-4/+23
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* improved builds api interface in jupiter (#281)Dan Engelbrecht2025-02-123-140/+178
| | | | | * multipart upload/download iterface in jupiter * review fixes
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-222-0/+105
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-168-7/+1066
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-134-4/+8
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* more memory tagging and fixes (#263)Stefan Boberg2024-12-162-4/+94
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Memory tracking improvements (#262)Stefan Boberg2024-12-111-0/+14
| | | | | * added LLM tag to properly tag RPC allocations * annotated some more httpsys functions with memory tags * only emit memory scope events if the active tag is different from the new tag
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-18/+60
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* auth fixes (#260)Dan Engelbrecht2024-12-103-32/+43
| | | | * fix so we can replace an openid provider that was read from disk file * fix OidcClient lifetime issues in authmg
* Unity build fixes (#253)Stefan Boberg2024-12-051-8/+0
| | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+19
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-281-2/+2
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* add validation of payload responses in http client (#240)Dan Engelbrecht2024-11-271-220/+292
| | | if response payload does not validate properly do a retry if applicable
* workspace share security (#192)Dan Engelbrecht2024-10-231-1/+4
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* safer path from handle (#195)Dan Engelbrecht2024-10-161-2/+12
| | | * remove PathFromHandle that throws to give better context on failures
* made fmt formatter format function const (#162)Stefan Boberg2024-09-201-2/+2
| | | this appears to be required as of fmt v11
* Add zenserver session id to Sentry context (#133)Dan Engelbrecht2024-08-221-3/+1
|
* don't assert that we have moved bytes if source block is zero size (#97)Dan Engelbrecht2024-06-142-2/+2
| | | | * don't assert that we have moved bytes if source block is zero size * handle invalid session ids gracefully
* workspace shares (#84)Dan Engelbrecht2024-05-292-0/+20
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-1/+2
| | | * use write and move in place for safer writing of files
* import oplog improvements (#54)Dan Engelbrecht2024-04-201-70/+158
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* improved assert (#37)Dan Engelbrecht2024-04-048-25/+56
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-032-18/+40
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* add yaml serialization support (#3)Stefan Boberg2024-03-261-0/+10
| | | | | | | this change adds serialization of payloads as YAML, but not parsing. The implementation is somewhat based on the JSON path, and may be collapsed eventually as it is possible to serialize JSON format using the same code it also separates out the JSON serialization into a separate file for ease of maintenance any HTTP request response may be formatted as yaml by using a `.yaml` suffix or an `Accept: text/yaml` header
* add filter to projectstore entries request (#25)Dan Engelbrecht2024-03-262-0/+36
| | | | | | | * Add HttpServerRequest::Decode to decode http request parameters * Add support to filter projectstore `entries` request using the `fieldfilter` where the wanted fields are comma (,) delimited * Add support for responding with compressed payloads for projectstore `entries` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `files` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `chunkinfo` requests by adding AcceptType `compressed-binary` to the request header
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 23:37:46 +0000