Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use Vec instead of DashMap on LabelDict.type_to_label #343

Open
huang12zheng opened this issue Mar 14, 2024 · 1 comment
Open

use Vec instead of DashMap on LabelDict.type_to_label #343

huang12zheng opened this issue Mar 14, 2024 · 1 comment

Comments

@huang12zheng
Copy link

pub struct LabelDict {
    label_to_type: DashMap<String, u16>,
    type_to_label: DashMap<u16, String>,
    id: AtomicU16,
}

use Vec or some lockfree Vec, or arena instead of DashMap on type_to_label.

@cfsfine
Copy link

cfsfine commented Mar 14, 2024

Hello! Thank you for interest in tugraph-analytics.
CStore(also geaflow-cstore) is a single-machine storage engine that theoretically supports multi-threaded writing on a single shard. LabelDict needs to reside in memory. When the written data has a new label, the registration interface of LabelDict needs to be called, so no matter Whether you use DashMap or other lock free structures, you need to use concurrency-safe data structures. DashMap may not necessarily perform the best here, but so far it is not a performance bottleneck.
You also mentioned Arena. CStore supports cross-language calls. The premise of using pre-allocated memory is that the data is only used within CStore and does not have data transmission with the upper-layer system. But Label exist data interaction with the upper-layer system, so if use Arena, there will be life cycle problems, and copying (from the memory pool to outside the memory pool) cannot be avoided, and the performance loss will be greater.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants