Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added union method to HLL #3293

Merged
merged 2 commits into from
Aug 18, 2024
Merged

Added union method to HLL #3293

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
92157f6
Exposed method to compute union cardinality of two HLL counters with …
LucaCappelletti94 Aug 15, 2024
ed67c55
Formatted code
LucaCappelletti94 Aug 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions src/core/src/sketch/hyperloglog/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,13 @@
estimators::mle(&counts, self.p, self.q, 0.01) as usize
}

pub fn union(&self, other: &HyperLogLog) -> usize {

Check warning on line 89 in src/core/src/sketch/hyperloglog/mod.rs

View check run for this annotation

Codecov / codecov/patch

src/core/src/sketch/hyperloglog/mod.rs#L89 

Added line #L89 was not covered by tests
let (only_a, only_b, intersection) =
estimators::joint_mle(&self.registers, &other.registers, self.p, self.q);

only_a + only_b + intersection
}

pub fn similarity(&self, other: &HyperLogLog) -> f64 {
let (only_a, only_b, intersection) =
estimators::joint_mle(&self.registers, &other.registers, self.p, self.q);
Expand Down Expand Up @@ -272,13 +279,12 @@
const N_UNIQUE_H1: usize = 500741;
const N_UNIQUE_H2: usize = 995845;
const N_UNIQUE_U: usize = 995845;
const INTERSECTION: usize = 500838;

const SIMILARITY: f64 = 0.502783;
const CONTAINMENT_H1: f64 = 1.;
const CONTAINMENT_H2: f64 = 0.502783;

const INTERSECTION: usize = 500838;

let mut filename = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
filename.push("../../tests/test-data/genome-s10.fa.gz");

Expand Down Expand Up @@ -321,6 +327,9 @@
let abs_error = (1. - (hll2.cardinality() as f64 / N_UNIQUE_H2 as f64)).abs();
assert!(abs_error < ERR_RATE, "{}", abs_error);

let abs_error = (1. - (hll1.union(&hll2) as f64 / N_UNIQUE_U as f64)).abs();
assert!(abs_error < ERR_RATE, "{}", abs_error);

let similarity = hll1.similarity(&hll2);
let abs_error = (1. - (similarity / SIMILARITY)).abs();
assert!(abs_error < ERR_RATE, "{} {}", similarity, SIMILARITY);
Expand Down
Loading