Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sensors: harden string parsing from BPF events #1276

Merged
merged 1 commit into from
Aug 2, 2023

Conversation

mtardy
Copy link
Member

@mtardy mtardy commented Jul 26, 2023

Initially found this issue by trying to fuzz parts of the handler. You can run the fuzzer with the following command:
go test -fuzz=Fuzz_parseString -run=Fuzz_parseString ./pkg/sensors/tracing/ Also added some unit test cases under Test_parseString.

Previous parsing had flaws that could make Tetragon panic or be OOM killed in certain circumstances. In reality, the string size was already bound by BPF copy_string function to 1024 but it could have theoretically returned a size of -2 as an error code which would have made it panic.

Also remove some code duplication that was rewriting the string parsing instead of using the common function.

@mtardy mtardy added the area/userspace Related to userspace Tetragon logic label Jul 26, 2023
Comment on lines +980 to +986
// If no size then path walk was not possible and file was
// either a mount point or not a "file" at all which can
// happen if running without any filters and kernel opens an
// anonymous inode. For this lets just report its on "/" all
// though pid filtering will mostly catch this.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I get this comment so maybe we should not apply this for gt.GenericFileType, gt.GenericFdType, gt.GenericKiocb: and case gt.GenericPathType:. I just reestablished it to have exactly the same behavior.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the case I'm trying to handle is the anonymous file case where the value returned here is empty. I guess its fine to do for all the other cases as well. What we need to be sure is we report something so the logs at least have an entry.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I see, I just have one last question, in the patched that introduced that you wrote:

When using fd_install hook to monitor program open* if the open is against
the mount then our path walking logic will return the size=0. The user
side then interprets this as an error and prints an ugly warning.

And the thing I wonder is, if the size is indeed 0 and it's correctly writing a 4 bytes 0, decoding should be okay and not error:


But I guess you encountered the error? Because the only situation I see that could trigger this would be that BPF sends a message less than 4 bytes in total for that arg, thus we would get an EOF.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm that patch is from some long ago time I don't recall what I did. If sending a 4B zerod arg is the right thing lets do that. A lot of code has been fixed/moved around probably from when I wrote that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok maybe this is never triggered anymore

@mtardy mtardy marked this pull request as ready for review July 26, 2023 13:06
@mtardy mtardy requested a review from a team as a code owner July 26, 2023 13:06
@mtardy mtardy requested a review from tpapagian July 26, 2023 13:06
@mtardy mtardy force-pushed the pr/mtardy/fuzz-handle-string branch from cf55f07 to f51ab2d Compare July 31, 2023 08:34
Initially found this issue by trying to fuzz parts of the handler. You
can run the fuzzer with the following command:
`go test -fuzz=Fuzz_parseString -run=Fuzz_parseString ./pkg/sensors/tracing/`
Also added some unit test cases under `Test_parseString`.

Previous parsing had flaws that could make Tetragon panic or be OOM
killed in certain circumstances. In reality, the string size was already
bound by BPF copy_string function to 1024 but it could have
theoretically returned a size of -2 as an error code which would have
made it panic.

Also remove some code duplication that was rewriting the string parsing
instead of using the common function.

Signed-off-by: Mahe Tardy <[email protected]>
@mtardy mtardy force-pushed the pr/mtardy/fuzz-handle-string branch from f51ab2d to 94faed1 Compare July 31, 2023 08:45
@mtardy mtardy requested a review from kkourt August 1, 2023 08:11
Copy link
Contributor

@kkourt kkourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

t.Errorf("unexpected error %v", err)
}
if out != "pizza" {
t.Errorf("got %q, want %q", out, "pizza")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍕!

err = binary.Read(r, binary.LittleEndian, &outputStr)

// limit the size of the string to avoid huge memory allocation and OOM kill in case of issue
if size > maxStringSize {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is defensive right? We should not get such a string size from the BPF kernel program?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes see MAX_STRING here

size = probe_read_str(&args[4], MAX_STRING, (char *)arg);

@jrfastab
Copy link
Contributor

jrfastab commented Aug 2, 2023

I'm happy with this. One question though just to be sure I understood this correctly.

@jrfastab jrfastab merged commit 9393851 into main Aug 2, 2023
22 checks passed
@jrfastab jrfastab deleted the pr/mtardy/fuzz-handle-string branch August 2, 2023 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/userspace Related to userspace Tetragon logic
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants