aboutsummaryrefslogtreecommitdiff
path: root/zenserver
Commit message (Collapse)AuthorAgeFilesLines
* Refactor CacheBuckets to allow for storing RawHash/RawSize (#232)v0.2.4-pre0v0.2.4Dan Engelbrecht2023-02-202-98/+258
| | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize.
* Enhanced rpc request recording (#229)Dan Engelbrecht2023-02-173-262/+62
| | | | | | | | | | | | | * rpc replay zen command * fix replay sessions for thread * recording start/stop as zen commands * move rpcrecording code to zenutil to remove code duplication * simplify recording http request threading * added more data logging to rpc replay * NotFound is an acceptable response for an rpc request * fix rpc replay command line parsing * rpc replay stats * Allow spawning of sub-process workers when replaying rpc recording * changelog
* Experimental ObjectStore/CDN like endpointPer Larsson2023-02-165-1/+321
|
* Move knowledge of UE env variable from zenserver to zen command line tool (#227)Dan Engelbrecht2023-02-131-14/+3
| | | | * move knowledge of UE env variable from zenserver to zen command line tool * move env-fetching code to GetEnvVariable
* no sentry report on port conflict (#228)Dan Engelbrecht2023-02-131-4/+8
| | | | * Exit with failure code on port conflict rather than reporting crash to Sentry * changelog
* oplog upload/download (#214)Dan Engelbrecht2023-02-0914-63/+2798
| | | | | | | | | | | | - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint.
* Add support for "packagedata" mapping in oplog entries (#224)Dan Engelbrecht2023-02-071-3/+13
|
* Merge pull request #219 from EpicGames/linux-buildMartin Ridgers2023-02-061-0/+11
|\ | | | | Scripts for building Linux binaries using UE's Linux toolchain.
| * Commented out breakpad_client xmake.lua adjustment - not applicable yetMartin Ridgers2023-02-031-0/+3
| | | | | | | | | | | | Later baselines of vcpkg will change to a newer version of sentry-native that will produce libbreakpad_client. This block of xmake script will then be required to coerce xmake to order libraries correctly
| * Fixed symbol resolution error because xmake reverses dependent librariesMartin Ridgers2023-02-021-0/+8
| |
* | remove legacy `export-project` and `import-project` (#222)Dan Engelbrecht2023-02-021-203/+0
| |
* | Reduce lock scopes in oplog (#220)Dan Engelbrecht2023-02-022-90/+123
|/
* Clean up project store file structure (#218)Dan Engelbrecht2023-02-013-1/+1
| | | | * move project store to separate folder * moved import/export project commands into projectstore cmd files
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2023-01-262-113/+167
|\
| * Retain memory buffer when replaying oplog (#216)Dan Engelbrecht2023-01-261-3/+7
| | | | | | | | * Retain memory buffer when replaying oplog * changelog
| * Better error responses/logging in project store request (#217)Dan Engelbrecht2023-01-262-110/+160
| | | | | | | | * Better error responses/logging in project store request * changelog
* | mergeStefan Boberg2023-01-269-1073/+25
|\|
| * removed HttpLaunchService and related codeStefan Boberg2023-01-243-643/+0
| | | | | | | | this was used for testing but is no longer wanted in this form
| * removed dead/experimental file system related codeStefan Boberg2023-01-246-430/+0
| |
| * Added version information to server executable (#215)Stefan Boberg2023-01-242-0/+25
| | | | | | | | | | This change adds some version and product information to the Windows executable, in order to make it easier for users to understand the purpose of the zenserver.exe process when they see it in Task Manager and on disk It also adds back the executable icon which was lost when we transitioned from the previous sln build process
* | removed experimental mesh codeStefan Boberg2023-01-265-382/+0
|/ | | | should be replaced with a proper implementation later
* fix gc logging (#213)Dan Engelbrecht2023-01-131-7/+7
| | | | | | * Don't output time to next GC if time is "infinite". * Do immediate check of GC status on thread startup instead of waiting montior intervall first. * set up reasonable gc defaults * changelog
* zen command line tool improvements (#212)Dan Engelbrecht2023-01-131-0/+1
| | | | | | | | | | | | - Feature: zen command line tool `cache-info` to show cache, namespace or bucket info - Feature: zen command line tool `project-info` to show store, project or oplog info - Feature: zen command line tool `project-drop` to drop project or oplog - Feature: zen command line tool `gc` to trigger a GC run - Feature: zen command line tool `gc-info` to check status of GC - Improvement: zen command line tool now fails on any unrecognized arguments - Improvement: zen command line tool now displays extra help for all sub-commands - Improvement: host address can now be configured for zen command line tool `drop` command changelog
* Add info (GET) endpoints for structured cache (#211)Dan Engelbrecht2023-01-116-60/+415
| | | | | | | * Add GET requests on cache/namespace/bucket level * Add root route for project store requests (same as /list) * Add markerpath to oplog info object * Add totalsize, opcount and expired to oplog info * Changelog
* Fix log index snapshot (#210)Dan Engelbrecht2022-12-163-12/+19
| | | | | | * Fix log reading for structured cache store Make sure cache is flushed at exit * dont flush index to disk unless new entries have been written * changelog
* oplog level GC (#209)Dan Engelbrecht2022-12-142-43/+209
| | | | | | | | Adds check for marker file supplied by UE to see if an oplog is expired (user has deleted the corresponding cooked folder). Fixed concurrency vulnerabilities is project store related to oplogs. * fix concurrency vulnerabilities * propagate lifetime file path * oplog level gc * changelog
* Changed so CompressedBuffer::DecodeRawHash returns IoHash just like on the ↵Stefan Boberg2022-12-123-6/+5
| | | | | | UE side (#208) removed all use of IoHash::FromBLAKE3() caused by interactions with CompressedBuffer APIs
* Path from handle perf improvement (#206)Dan Engelbrecht2022-12-081-8/+12
| | | | | | * Read recorded requests to memory before parsing This will more accurately simulate how requests comes in from a client * Make a fast path for GetFinalPathNameByHandleW Try to get the path with a fixes size buffer first to avoid always doing two calls to GetFinalPathNameByHandleW
* Fix logging of number of read entries from log/index file for stores (#204)Dan Engelbrecht2022-12-081-5/+4
| | | | * Fix logging of number of read entries from log/index file for stores * changelog
* optimizations (#200)Dan Engelbrecht2022-12-078-149/+201
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Cache request record/replay (#198)Dan Engelbrecht2022-12-073-213/+457
| | | | | | | | | This adds recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request. `<host>/z$/exec$/start-recording?<disk-storage-path>` `<host>/z$/exec$/stop-recording` `<host>/z$/exec$/replay-recording?<thread-count>&<disk-storage-path>`
* Use Iso8601 format for logging start and end message (#202)Dan Engelbrecht2022-12-071-2/+3
| | | | * Use Iso8601 format for logging start and end message * changelog
* Size based gc trigger (#197)Dan Engelbrecht2022-12-026-30/+39
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* reduce gc log spam (INFO -> DEBUG) (#199)Dan Engelbrecht2022-12-022-33/+34
|
* Make sure we always store record/op before attachments (#195)Dan Engelbrecht2022-12-012-116/+172
| | | | | | | | * Make sure we always store record/op before attachments We don't want to store attachments first - a GC operation could then remove attachments if triggered before storing record/op * zen::Latch * Use latch to wait for attachments to be stored * use zen::latch when adding attachments from project oplog import * changelog
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-244-54/+98
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* add "version" command with "detailed" option (#192)Dan Engelbrecht2022-11-211-0/+15
| | | specify host url to query running service version, otherwise you get zen command version
* bump cache large object size limit to 128Kb (#191)Dan Engelbrecht2022-11-211-1/+1
| | | | * bump cache large object size limit to 128Kb * changelog
* changelog and removed obsolete commentDan Engelbrecht2022-11-181-1/+0
|
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-185-220/+665
| | | * Add `import-project` and `export-project` command line parsing
* move BasicFile to zenutil to remove zenstore dependency from zen command (#190)Dan Engelbrecht2022-11-173-3/+2
|
* Configure OpenID providers from cmd line and Lua cfg. (#189)Per Larsson2022-11-145-6/+75
|
* Support file reference in package message (#184)Dan Engelbrecht2022-11-074-15/+35
| | | | | | | | * Fix packed message parsing for absolute path * Always enable are sharing when opening files as IoBuffers. * Allow control over sending partial files as localfile ref * Check "AcceptFlags" field in RPC message for allowing localfile ref in reply * make oplog entry add operations ZEN_DEBUG level logs * changelog
* fix concurrency issues in projectstore and enable GC (#181)Dan Engelbrecht2022-10-172-276/+644
| | | | | | | | | | | | | | * Fix concurreny issues when deleting projects/oplogs * remove rocksdb test code * project store unit tests * safer deletion of oplogs/projects * reference count ProjectStore::Project to handle lifetime during GC * Don't open all project oplogs unless we need them * Don't scrub expired projects * Don't gather references from expired projects * added logging details for GC * release lock as soon as folder is moved * more tests for project store * changelog
* Add "Accept" field in RPC request to gracefully handle requests from older ↵Dan Engelbrecht2022-10-133-23/+105
| | | | instances (#180)
* disable project store GC (#179)Dan Engelbrecht2022-10-121-1/+2
|
* De/oplog gc lifetime (#178)Dan Engelbrecht2022-10-102-47/+141
| | | | | | | * Make sure we don't use invalidated iterators in projectstore.cpp * project store keeps track of project file and will garbage collect data for a project if the project file no longer exist * Implement GC of projects in project store - still need to fix lifetime issues for Project instances * Add INFO log if project file path is empty in projectstore * changelog
* De/reduce buffer creation in parsepackedmessage (#175)Dan Engelbrecht2022-09-301-7/+7
| | | | | * Don't create call CreateBuffer for attachement data that we only read and not keep * changelog * don't read oplog attachments into memory just to do a redundant store of them
* Use bucket/key to get inline value in upstream for chunks without a chunkid ↵Dan Engelbrecht2022-09-301-3/+6
| | | | (#176)
* Format all rpc package responses using `FormatPackageMessageBuffer` to avoid ↵Dan Engelbrecht2022-09-292-35/+33
| | | | | memory copy (#174) When reading upstream, fall back to old rpc response to handle older instances.
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-25 06:53:17 +0000