Invalid line coverage for multiple partially covered derive macros

[treiher]

If multiple derive macros are used, and one is covered and one is uncovered, the results shown in the llvm-cov output will not match the output produced by json.get_uncovered_lines(). While llvm-cov considers the line to be uncovered, json.get_uncovered_lines() considers the line to be covered. So even if there is a missing line, --show-missing-lines will not show the affected line and --fail-uncovered-lines 0 will have no effect.

Here is a reproducer:

use strum::{Display, EnumIter};

#[derive(Display, EnumIter, Clone, Debug, PartialEq)]
pub enum Foo {
    A = 0,
    B = 1,
}

#[cfg(test)]
mod tests {
    use strum::IntoEnumIterator;

    use super::*;

    #[test]
    fn test_foo_serde() {
        for foo in Foo::iter() {
            assert_eq!(foo, foo.clone());
        }
    }
}

Filename       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
src/lib.rs           9                 2    77.78%           3                 1    66.67%           6                 1    83.33%           0                 0         -
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                9                 2    77.78%           3                 1    66.67%           6                 1    83.33%           0                 0         -

The function created by the strum::EnumIter macro is tested, while the function created by the strum::Display macro is not. llvm-cov considers line 3 to be uncovered, but this line is considered to be covered by json.get_uncovered_lines(). This issue is caused by the following code, which was added to fix #181:

cargo-llvm-cov/src/json.rs

Lines 268 to 270 in b7e5eb5

	if let Some(covered_lines) = covered_files.get(file_name) {
	uncovered_lines.retain(|&x| !covered_lines.contains(&x));
	}

Unfortunately, I don't see a simple solution which wouldn't add a regression. Any ideas?

Using the JSON output for --fail-uncovered-lines would already be an improvement, so that at least the exit code would be correct, even if some lines in --show-missing-lines are missing. What do you think?

[taiki-e]

cc @vmiklos

[taiki-e]

(It doesn't resolve the underlying issue, but I guess the case itself could be resolved by using #[automatically_derived] in strum, considering rust-lang/rust#120185.)

[vmiklos]

I think the root of the trouble is that llvm's JSON output tells us the line coverage percentage, so if it's complete or not is precise, but what lines are covered is not in the JSON explicitly. llvm's code is quite complex, tries to deal with c++ templates and macros..

My code here is obviously quite naive, but then with some iterations it turned out to be quite usable in practice.

So if your problem is about the percentage reported, that's from llvm. If the percentage is <100, but you're not happy with the reported reported lines, then the JSON parsing code (in rust) can be tweaked.

I guess the only important point is to make sure the percentage keeps coming from llvm, not our own calculation. Does this help?

[vmiklos]

FWIW I filed llvm/llvm-project#126307, in case llvm-cov would give us which lines are not covered, then we could drop our own calculation, so you could not end up with a silly case where llvm-cov and 'cargo llvm-cov' disagrees if a line is covered or not.