Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a test of memory use when logging a lot of big images #1372

Merged
merged 5 commits into from
Feb 22, 2023

Conversation

emilk
Copy link
Member

@emilk emilk commented Feb 21, 2023

Investigating #1242

RERUN_TRACK_ALLOCATIONS=1 cargo r -p test_image_memory

It turns out that when we log normal RGBA images we store the data once in the ArrowMsg, and then that data is deep-cloned when put in the store. I think it is because of this:

#[derive(Clone, Debug, PartialEq, ArrowField, ArrowSerialize, ArrowDeserialize)]
#[arrow_field(type = "dense")]
pub enum TensorData {
    U8(Vec<u8>),
    U16(Buffer<u16>),
    U32(Buffer<u32>),
    U64(Buffer<u64>),
    // ---
    I8(Buffer<i8>),
    I16(Buffer<i16>),
    I32(Buffer<i32>),
    I64(Buffer<i64>),
    // ---
    // TODO(#854): Native F16 support for arrow tensors
    //F16(Vec<arrow2::types::f16>),
    F32(Buffer<f32>),
    F64(Buffer<f64>),
    JPEG(Vec<u8>),
}

We use the ref-counted Buffer for everything except for 8-bit tensors (most images) and for jpegs.

We have a similar problem for the Mesh3D component.

@emilk emilk added 🔨 testing testing and benchmarks 📉 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself labels Feb 21, 2023
@teh-cmc teh-cmc self-requested a review February 22, 2023 09:18
crates/re_sdk/src/lib.rs Outdated Show resolved Hide resolved
tests/rust/test_image_memory/src/main.rs Outdated Show resolved Hide resolved
@emilk emilk merged commit 6b9aab0 into main Feb 22, 2023
@emilk emilk deleted the emilk/test-image-memory-use branch February 22, 2023 15:55
emilk added a commit that referenced this pull request Mar 2, 2023
* Add a test of memory use when logging a lot of big images

* copy-pasta

Co-authored-by: Clement Rey <[email protected]>

* copy-pasta

Co-authored-by: Clement Rey <[email protected]>

* Add store archiving to the stack shortening

* fix feature name

---------

Co-authored-by: Clement Rey <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📉 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself 🔨 testing testing and benchmarks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants