Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random segfaults in some unit tests #1718

Open
mcm001 opened this issue Jan 13, 2025 · 3 comments
Open

Random segfaults in some unit tests #1718

mcm001 opened this issue Jan 13, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@mcm001
Copy link
Contributor

mcm001 commented Jan 13, 2025

Describe the bug
A clear and concise description of what the bug is and what the expected behavior should have been.

To Reproduce

See: https://github.com/PhotonVision/photonvision/actions/runs/12738416647/job/35500709893?pr=1715 and similar for random setfaults

Additional context

Running a photonlibTest from #1714 :

Failure 1:

[==========] 52 tests from 7 test suites ran. (1505 ms total)
[  PASSED  ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==77094==ERROR: AddressSanitizer: SEGV on unknown address 0x000069800042 (pc 0x155550999d79 bp 0x1555499ff5a0 sp 0x1555499fed18 T9)
==77094==The signal is caused by a READ memory access.
    #0 0x155550999d79  (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
    #1 0x155554a942e9 in MemcmpInterceptorCommon(void*, int (*)(void const*, void const*, unsigned long), void const*, void const*, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:881
    #2 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:892
    #3 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:887
    #4 0x15555549ac00 in std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/include/c++/11/bits/char_traits.h:389
    #5 0x15555549ac00 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const /usr/include/c++/11/bits/basic_string.tcc:1444
    #6 0x15555549ac00 in bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) /usr/include/c++/11/bits/basic_string.h:6250
    #7 0x15555549ac00 in GetSinkStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:163
    #8 0x15555549ac00 in UpdateStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:228
    #9 0x15555549ac00 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:464
    #10 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
    #11 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
    #12 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
    #13 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
    #14 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
    #15 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
    #16 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
    #17 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
    #18 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #19 0x1555514dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #20 0x155550894ac2 in start_thread nptl/pthread_create.c:442
    #21 0x15555092684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x199d79) 
Thread T9 created by T0 here:
    #0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)

==77094==ABORTING

Failure 2:

[==========] 52 tests from 7 test suites ran. (1515 ms total)
[  PASSED  ] 52 tests.
terminate called after throwing an instance of 'std::bad_function_call'
  what():  bad_function_call
./run-photonlibpy-to-crash.sh: line 31: 97288 Aborted                 "$PROGRAM"
(.venv) matt@photonvision:~/Documents/GitHub/photonvision$ 

Failure 3:

[==========] 52 tests from 7 test suites ran. (1529 ms total)
[  PASSED  ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==74081==ERROR: AddressSanitizer: SEGV on unknown address 0x00006980003b (pc 0x155554a2cc37 bp 0x00006980004b sp 0x1555499fed20 T9)
==74081==The signal is caused by a WRITE memory access.
    #0 0x155554a2cc37 in bool __sanitizer::atomic_compare_exchange_strong<__sanitizer::atomic_uint8_t>(__sanitizer::atomic_uint8_t volatile*, __sanitizer::atomic_uint8_t::Type*, __sanitizer::atomic_uint8_t::Type, __sanitizer::memory_order) ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:80
    #1 0x155554a2cc37 in __asan::Allocator::AtomicallySetQuarantineFlagIfAllocated(__asan::AsanChunk*, void*, __sanitizer::BufferedStackTrace*) ../../../../src/libsanitizer/asan/asan_allocator.cpp:621
    #2 0x155554a2cc37 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) ../../../../src/libsanitizer/asan/asan_allocator.cpp:697
    #3 0x155554ab720c in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172
    #4 0x155555498f70 in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /usr/include/c++/11/ext/new_allocator.h:145
    #5 0x155555498f70 in std::allocator<char>::deallocate(char*, unsigned long) /usr/include/c++/11/bits/allocator.h:199
    #6 0x155555498f70 in std::allocator_traits<std::allocator<char> >::deallocate(std::allocator<char>&, char*, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:496
    #7 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long) /usr/include/c++/11/bits/basic_string.h:245
    #8 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() /usr/include/c++/11/bits/basic_string.h:240
    #9 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() /usr/include/c++/11/bits/basic_string.h:672
    #10 0x155555498f70 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:88
    #11 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:149
    #12 0x155555498f70 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:163
    #13 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:196
    #14 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) /usr/include/c++/11/bits/alloc_traits.h:848
    #15 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~vector() /usr/include/c++/11/bits/stl_vector.h:680
    #16 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_move_assign(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, std::integral_constant<bool, true>) /usr/include/c++/11/bits/stl_vector.h:1821
    #17 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator=(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) /usr/include/c++/11/bits/stl_vector.h:714
    #18 0x155555498f70 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:463
    #19 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
    #20 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
    #21 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
    #22 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
    #23 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
    #24 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
    #25 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
    #26 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
    #27 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #28 0x1555514dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #29 0x155550894ac2 in start_thread nptl/pthread_create.c:442
    #30 0x15555092684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:80 in bool __sanitizer::atomic_compare_exchange_strong<__sanitizer::atomic_uint8_t>(__sanitizer::atomic_uint8_t volatile*, __sanitizer::atomic_uint8_t::Type*, __sanitizer::atomic_uint8_t::Type, __sanitizer::memory_order)
Thread T9 created by T0 here:
    #0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)

==74081==ABORTING

Failure 4:

[  PASSED  ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==89459==ERROR: AddressSanitizer: SEGV on unknown address 0x000069800050 (pc 0x155550999d79 bp 0x1555499ff5a0 sp 0x1555499fed18 T9)
==89459==The signal is caused by a READ memory access.
    #0 0x155550999d79  (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
    #1 0x155554a942e9 in MemcmpInterceptorCommon(void*, int (*)(void const*, void const*, unsigned long), void const*, void const*, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:881
    #2 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:892
    #3 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:887
    #4 0x15555549ac00 in std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/include/c++/11/bits/char_traits.h:389
    #5 0x15555549ac00 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const /usr/include/c++/11/bits/basic_string.tcc:1444
    #6 0x15555549ac00 in bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) /usr/include/c++/11/bits/basic_string.h:6250
    #7 0x15555549ac00 in GetSinkStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:163
    #8 0x15555549ac00 in UpdateStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:228
    #9 0x15555549ac00 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:464
    #10 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
    #11 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
    #12 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
    #13 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
    #14 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
    #15 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
    #16 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
    #17 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
    #18 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #19 0x1555514dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #20 0x155550894ac2 in start_thread nptl/pthread_create.c:442
    #21 0x15555092684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x199d79) 
Thread T9 created by T0 here:
    #0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)

==89459==ABORTING
(.venv) matt@photonvision:~/Documents/GitHub/photonvision$
@mcm001 mcm001 added the bug Something isn't working label Jan 13, 2025
@Sam948-byte
Copy link
Contributor

This error also occurred here, showing that it's not limited to the MacOS Host Build.

https://github.com/PhotonVision/photonvision/actions/runs/12729129088/job/35480552247?pr=1700

@mcm001
Copy link
Contributor Author

mcm001 commented Jan 13, 2025

I think i found something spooky. here, we iterate over m_addresses from a function called by NotifierThread.

The instance could only come from this magic static here. The callback comes from m_videoListener here. The callback does capture this. Not sure about destruction order or how to fix this, but this -seems- like an allwpilib side issue that will need to be patched there.

@PhotonVision PhotonVision deleted a comment from Sam948-byte Jan 13, 2025
@mcm001
Copy link
Contributor Author

mcm001 commented Jan 13, 2025

(ack whoops!)

@mcm001 mcm001 mentioned this issue Jan 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mcm001 @Sam948-byte