| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
| |
- Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
|
| | |
|
| |
|
|
|
| |
* added FLLMTag which can be used to register memory tags outside of core
* changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency
* instrumented some subsystems with dynamic tags
|
| |
|
|
|
|
|
| |
minor clean-up
`zencore/memory.h` used to contain a variety of things including `Malloc` support along with `MemoryView` etc
since the memory allocator stuff moved into `zencore/memory/memory.h` there was basically only `MemoryView` and `MemCmp` in there which seemed better to split out into separate headers to avoid overloading `memory.h`
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future.
To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks.
This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`:
* `mimalloc` - mimalloc (default, same as before)
* `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows.
* `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows.
* `ansi` - fallback to default system allocator
|
| |
|
|
|
|
|
|
|
| |
* removed unused memory classes
* added align.h alignment helpers used in upcoming changes
* added char16_t StringLength
* avoid memory alloc in SetCurrentThreadName
* added command line parsing helpers to zencore/commandline.h
* removed IoBuffer direct VirtualAlloc path
|
| |
|
| |
* remove PathFromHandle that throws to give better context on failures
|
| |
|
| |
* Make IoBufferExtendedCore::Materialize error reporting match Materialize
|
| |
|
| |
Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
|
| |
|
|
|
| |
* Don't try to validate empty compact binary
* Don't try to free fake memory buffer pointer in iobuffer destructor
|
| |
|
| |
- Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
|
| |
|
|
|
| |
- Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs
- Improvement: Add ability to block a set of CAS entries from GC in project store
- Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
|
| |
|
|
|
| |
* move structuredcachestore tests to zenstore-test
* Don't materialize entire files when hashing if it is a large files
* rewrite CompositeBuffer::Mid to never materialize buffers
|
| |
|
|
|
| |
* Change BasicFile::Read to throw exception like IoBuffer
- Don't ASSERT on dwNumberOfBytesRead == NumberOfBytesToRead - throw exception with details instead
- Use proper return type for pread()
|
| |
|
|
| |
* fix leak in IoBuffer for manifested small chunk. previously it would null out the `m_DataPtr` member on every path from `IoBufferExtendedCore::~IoBufferExtendedCore()` but it only really makes sense to null it out when the buffer has been memory mapped
|
| | |
|
| |
|
|
|
| |
this change adds some includes to files which "inherit" includes from elsewhere
this was exposed on another branch when removing some heavy dependencies from central headers
|
| |
|
|
|
|
|
|
| |
this change fixes a problem where the memory cache layer can inadvertently prevent underlying block store files from being deleted
* ensure we get memory buffers on all paths
* added more context to error in IoBufferBuilder::ReadFromFileMaybe
* fixed problematic pread call success check in IoBufferBuilder::ReadFromFileMaybe which would always report failure on Linux/MacOS
|
| | |
|
| |
|
|
|
|
| |
- Improvement: Catch Out Of Memory and Out Of Disk exceptions and report back to reqeuster without reporting an error to Sentry
- Improvement: If creating bucket fails when storing and item in the structured cache, log a warning and propagate error to requester without reporting an error to Sentry
- Improvement: Make an explicit flush of the active block written to in blockstore flush
- Improvement: Make sure cache and cas MakeIndexSnapshot does not throw exception on failure which would cause and abnormal termniation at exit
|
| |
|
|
|
| |
* more trace scopes
* Make sure ReplayLogEntries uses the correct size for oplog buffer
* changelog
|
| |
|
|
|
| |
get file name from handle (#393)
- Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle
|
| |
|
|
|
|
|
|
|
|
| |
* stream large downloads from jupiter to temporary file
* rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag
* changelog
* log number of attachments to download
* add delay on jupiter request failure when retrying
* make sure we upload all attachments even if Needs are empty when ForceUpload is true
release TempAttachment as soon as it is used
* sort attachments so we get predictable blocks for the same oplog
|
| |
|
|
|
|
| |
(#370)
* use better hash function for better distribution in IoBuffer g_MappingLocks
* changelog
|
| |
|
|
| |
* use exception when allocations fail rather than asserts
* changelog
|
| |
|
|
| |
no actual code changes, just comments
|
| |
|
|
|
|
| |
this verifies that attempting to map a range outside the underlying file
fails. Posix appears to offer different semantics so it
silently accepts it still however.
|
| |
|
| |
ATL has been used here and there as a convenience. Given that this is a legacy component and not always something which gets installed along with the compiler we hereby remove the dependency altogether in favour of our own simple wrappers
|
| |
|
|
| |
* better logging/exception when reading file in IoBuffer::Materialize fails
* changelog
|
| |
|
|
| |
* added FileSizeFromHandle function
* added file size to error message when MapViewOfFile fails
|
|
|
* moved source directories into `/src`
* updated bundle.lua for new `src` path
* moved some docs, icon
* removed old test trees
|