aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp
Commit message (Collapse)AuthorAgeFilesLines
...
* various TSAN/ASAN/LeakAnalyzer fixes (#622)Stefan Boberg2023-12-191-1/+3
| | | | | | | | | * fix JobQueue test threading issue. The inner job queued with `QueueJob` would reference `I` from inside the captured closure which would subsequently disappear * made sure application exit is thread safe * don't try to access string data out of bounds * keep-alive flag is accessed from multiple threads * fix memory leaks in Zen upstream client code * TSAN fixes for Event
* http plugin dependency fixes (#590)Stefan Boberg2023-12-051-3/+11
| | | these changes clean up module dependencies and allow the transports subtree to be built standalone (in the UE tree for instance)
* HTTP plugin request debug logging (#587)Stefan Boberg2023-12-0511-128/+207
| | | | | | * added log level control/query to LoggerRef * added debug logging to http plugin implementation * added GetDebugName() to transport plugin interfaces * added debug name to log output
* WinIoThreadPool teardown cleaned up (#580)Stefan Boberg2023-11-301-0/+5
| | | previously would not shut down the threadpool properly which would leave threads around. This was not a problem in practice for our usage since we keep the thread pool alive for the duration of the process but it's better to clean up properly.
* minor: updated transport plugin default ports to match the new defaultsStefan Boberg2023-11-201-3/+3
|
* Make object store endpoint S3 compatible. (#535)Per Larsson2023-11-151-0/+17
| | | | | * Make object store endpoint S3 compatible. * Removed XML pretty printing and set object store endpoint disabled by default.
* package dependency clean-ups (#531)Stefan Boberg2023-11-131-2/+2
| | | | | | | | | | this change just cleans up dependency declarations in xmake.lua files, discovered while exploring a more to xrepo which catches dependency problems since it will not just place all includes in a single directory, unlike vcpkg. * removed spurious asio dependency from zenserver-test * removed rocksdb reference * add missing asio package dependency * removed catch2 reference (no longer available) * added explicit cpr reference * made some zencore package dependencies public (this is necessary because some public zencore headers pull in package headers. If you use a more strict package manager than vcpkg then you get compilation errors whenever these includes are pulled in unless you declare the dependency explicitly)
* option for zenserver - `--http-forceloopback` (#516)Dan Engelbrecht2023-11-096-30/+55
| | | | * New option for zenserver - `--http-forceloopback` which forces opening of the server http server using loopback (local) connection (UE-199776) * add fallback to local connection for asio if we get access denied on public port
* removed duplicate and unused constantsStefan Boberg2023-11-091-8/+0
|
* disk layer gc and error/warnings cleanup (#515)Dan Engelbrecht2023-11-081-1/+1
| | | | | | | - Improvement: Use GC reserve when writing index/manifest for a disk cache bucket when disk is low when available - Improvement: Demote errors to warning for issues that are not critical and we handle gracefully - Improvement: Treat more out of memory errors from windows as Out Of Memory errors Fixed wrong sizeof() statement for compactcas index (luckily the two structs are of same size)
* factored out some compiler definitions etc into zenbase (#517)Stefan Boberg2023-11-073-3/+3
| | | | | this is a header-only library which mostly contains definitions to support different platforms and compilers. It is part of the zen codebase but is intended to be consumable separately to zenbase etc to support standalone transport plug-ins and similar.
* spdlog implementation hiding (#498)Stefan Boberg2023-11-0610-40/+39
| | | | | | | | | this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future. As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link. Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future. The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
* added missing includes (#504)Stefan Boberg2023-10-273-0/+7
| | | | | this change adds some includes to files which "inherit" includes from elsewhere this was exposed on another branch when removing some heavy dependencies from central headers
* improve error handling when parsing http requests (#489)Dan Engelbrecht2023-10-201-6/+36
| | | * catch exceptions in HttpRequestParser::OnMessageComplete() callback
* added HttpClient::GetBaseUriStefan Boberg2023-10-201-1/+2
| | | | mostly useful when reporting errors
* Cache (rpc) activitity recording improvements (#482)Stefan Boberg2023-10-201-1/+1
| | | | | | | this adds a new RPC recording path aimed at more continuous recording and analysis of recorded sessions the new strategy is implemented alongside the original in order to retain the ability to read the older format the main difference between v2 and v1 is that the new strategy splits the recording into segments which are independent from each other. This is done to enable long running sessions with automatic disk cleanup (not implemented yet), appending to an existing recording (not implemented) and/or partial analysis and processing. The recorder will start a new segment when some criteria is fulfilled, including the number of files in the segment directory, disk footprint etc
* minor: fixed some typosStefan Boberg2023-10-171-3/+2
|
* support for multiple http servers (#473)Stefan Boberg2023-10-1311-361/+484
| | | | | | * added support for having multiple http servers active in one session * added configuration API to pluggable transports * removed pimpl pattern from some pluggable transports implementations
* improved http.sys initialization diagnostics and amended logic for dedicated ↵Stefan Boberg2023-10-131-40/+71
| | | | servers (#475)
* restructured zenhttp (#472)Stefan Boberg2023-10-1315-259/+4
| | | separating the http server implementations into a directory and moved diagsvcs into zenserver since it's somewhat hard-coded for it
* Merge remote-tracking branch 'origin/main' into zs/default-port-changeStefan Boberg2023-10-127-27/+83
|\
| * restructured transports SDK for easier UE integration (#470)Stefan Boberg2023-10-121-1/+1
| |
| * adjust resource usage for dedicated servers (#466)Stefan Boberg2023-10-126-26/+82
| | | | | | | | | | | | | | | | | | when dedicated mode is enabled via `--dedicated` or `server.dedicated` then we tune http.sys server settings to be more suitable for a shared server initially we tune two things * the thread pool used to service I/O requests allows a larger number of threads to be created when needed. The minimum thread count is unchanged but in dedicated server mode we double the maximum number of threads allowed * the http.sys request queue length (`HttpServerQueueLengthProperty`) is increased to 50,000 in dedicated mode. The regular default is 1,000
* | Change default port to 8558zousar2023-10-112-4/+4
|/ | | | Changes the default port without altering config for shared instances.
* pluggable asio transport (#460)Stefan Boberg2023-10-1113-188/+562
| | | | | | | | | added pluggable transport based on asio. This is in an experimental state and is not yet a replacement for httpasio even though that is the ultimate goal also moved plugin API header into dedicated part of the tree to clarify that it is meant to be usable in isolation, without any dependency on zencore et al moved transport implementations into dedicated source directory in zenhttp note that this adds code to the build but nothing should change at runtime since the instantiation of the new code is conditional and is inactive by default
* hide HttpAsioServer interface behind factory function (#463)Stefan Boberg2023-10-113-24/+29
|
* updated plugin API class names (#462)Stefan Boberg2023-10-118-79/+78
| | | | | | | | | * renamed some interfaces to improve pluggable transport API TransportConnectionInterface -> TransportConnection TransportPluginInterface -> TransportPlugin TransportServerConnectionHandler -> TransportServerConnection TransportServerInterface -> TransportServer
* suppress no-op auth debug loggingStefan Boberg2023-10-101-1/+6
|
* experimental pluggable transport support (#436)Stefan Boberg2023-10-1012-41/+1670
| | | this change adds a `--http=plugin` mode where we support pluggable transports. Currently this defaults to a barebones blocking winsock implementation but there is also support for dynamic loading of transport plugins, which will be further developed in the near future.
* removed websocket protocol support(#445)Stefan Boberg2023-10-044-1906/+1
| | | removed websocket support since it is not used right now and is unlikely to be used in the future
* factored out http parser from asio into separate files (#444)Stefan Boberg2023-10-033-450/+495
| | | | factored out http request parsing from httpasio into separate files to enable code to be reused for different transports
* reduce time spent holding http.sys transaction lock (#437)Stefan Boberg2023-10-022-15/+26
| | | | | * changed where calls to IssueNewRequestMaybe are made to reduce per-transaction lock contention * minor: reduce stack frame for HttpSysTransaction::IoCompletionCallback
* Handle OOM and OOD more gracefully to not spam Sentry with error reports (#434)Dan Engelbrecht2023-10-022-4/+39
| | | | | | - Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to reqeuster without reporting an error to Sentry - Improvement: If creating bucket fails when storing and item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry - Improvement: Make an explicit flush of the active block written to in blockstore flush - Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit
* added handle to http.sys error response logging (#432)Stefan Boberg2023-09-291-2/+3
| | | * added handle to http.sys error response logging
* adding more stats (#429)Dan Engelbrecht2023-09-281-0/+5
| | | | | - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
* added more context to http response error message (#430)Stefan Boberg2023-09-281-10/+77
| | | when a http.sys request response API call fails, we now include more information including metadata about the contents of the reponse (i.e memory/file references including file names/size and chunk offsets/size) to help track down some odd error conditions seen in production
* prefer to handle cache RPC requests synchronously (#428)Stefan Boberg2023-09-272-4/+27
| | | | | * only handle RPC requests in a worker thread if we have an upstream. we may as well handle the request inline on the http_io thread if we're only dealing with local data since the response times should be pretty consistent in that case * http.sys: don't create async worker thread pool until it's needed (typically only if we have an upstream)
* Add runtime status/control of logging (#419)Dan Engelbrecht2023-09-222-3/+3
| | | | | | | | | | | - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
* added name to http.sys request queue (#417)Stefan Boberg2023-09-221-1/+33
| | | | | | * added name to http.sys request queue * changed so that rejected requests are answered with a 403 response instead of just dropping the connection
* add more trace scopes (#362)Dan Engelbrecht2023-09-152-8/+23
| | | | | * more trace scopes * Make sure ReplayLogEntries uses the correct size for oplog buffer * changelog
* bugfix: http.sys async response handling race (#406)Stefan Boberg2023-09-151-1/+14
| | | | under heavy load, an async response could end up deleting the HTTP transaction object before the issuing call had completed
* http and httpsys config options (#401)Dan Engelbrecht2023-09-146-26/+45
| | | | * Added `--http-threads`, `--httpsys-async-work-threads`, `--httpsys-enable-request-logging` and `--httpsys-enable-async-response` command line options to zenserver * remove unused CreateHttpSysServer
* Make sure error logging or destructors don't throw exception when trying to ↵Dan Engelbrecht2023-09-121-14/+21
| | | | | get file name from handle (#393) - Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle
* add console logging to zen command (#389)Dan Engelbrecht2023-09-082-4/+18
| | | | properly set trace log level if IsVerbose add log category to http client
* Extend http client (#387)Dan Engelbrecht2023-09-083-93/+559
| | | * extend http client with configuration, headers, parameters and disk streaming upload/download
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-1/+2
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* oplog mirror support (#367)Stefan Boberg2023-08-212-1/+24
| | | | feature: added oplog-mirror command. this can be invoked to export oplog contents to corresponding files
* Cache process handles for FormatPackageMessage (#360)Dan Engelbrecht2023-08-173-131/+146
|
* handle exception in asio header parsing (#348)Dan Engelbrecht2023-08-091-50/+57
| | | | * make sure we return an error code instead of throwing exception if header parsing fails * changelog
* various zenhttp fixes from sb/protoStefan Boberg2023-06-3011-27/+98
| | | | | | | | | * Made HttpHealthService use locks to serialize access to state * Added ToString(HttpResponseCode HttpCode) * Added support for JS source maps * Moved IHttpStatsProvider/IHttpStatsService * Enabled enumeration of stats providers * Disabled build of HttpTestingService unless ZEN_WITH_TESTS is defined
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-20 13:24:39 +0000