[C++] How to fix error in arrow-s3fs-test Test #43164
Comments
llama90
changed the title
|
Hello, @kou Could you please provide any advice or guidance on this matter? I tried to find relevant CI scripts and documentation to resolve this issue. The part that triggers the error message is not in the arrow's code, but in the file generated after the build. Could this be a bug in |
|
Could you show full error log? |
|
Please take this. Thank you! lldb ./arrow-s3fs-test
(lldb) target create "./arrow-s3fs-test"
Current executable set to '/Users/lama/workspace/arrow-new/cpp/cmake-build-debug/debug/arrow-s3fs-test' (arm64).
(lldb) run
Process 17773 launched: '/Users/lama/workspace/arrow-new/cpp/cmake-build-debug/debug/arrow-s3fs-test' (arm64)
Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug/_deps/googletest-src/googletest/src/gtest_main.cc
[==========] Running 69 tests from 7 test suites.
[----------] Global test environment set-up.
Assertion failed: (s_configManager), function GetCachedConfigValue, file ConfigAndCredentialsCacheManager.cpp, line 178.
Process 17773 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
frame #4: 0x0000000100462348 arrow-s3fs-test`Aws::Config::GetCachedConfigValue(key="region") at ConfigAndCredentialsCacheManager.cpp:178:13
175
176 Aws::String GetCachedConfigValue(const Aws::String &key)
177 {
-> 178 assert(s_configManager);
179 return s_configManager->GetConfig(Aws::Auth::GetConfigProfileName(), key);
180 }
181
Target 0: (arrow-s3fs-test) stopped. |
|
Can you add the backtrace too @llama90? I just ran the test locally and all tests pass. |
|
Thank you for your attention. Here it is. BTW, Is it necessary to have AWS credentials to run the test? I am experiencing the same error on two different laptops. |
|
Could you try |
|
I tried as you suggested, but it's still the same. 😭 |
|
I raised this question because I wanted to resolve some S3 issues. I executed the PyArrow code, and it worked well. I am trying to resolve the issues with PyArrow through testing. import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.fs as fs
s3 = fs.S3FileSystem(region='ap-northeast-2')
data = {
'name': ['Alice', 'Bob', 'Charlie', 'David'],
'age': [24, 27, 22, 32]
}
table = pa.table(data)
bucket = 'bucket_name'
file_path = f'{bucket}/prefix/example.parquet'
with s3.open_output_stream(file_path) as f:
pq.write_table(table, f)
print(f"Data written to {file_path}")UPDATE: Why the Thank you for your attention! |
|
Googling the error you got leads me to only a couple of results, one of which is aws/aws-sdk-cpp#2605 and suggests InitAPI isn't getting called but, from your stack trace, it looks like it is. In s3fs_test, I see a call to InitAPI() guarded by an ifdef on |
|
Could you set breakpoints at |
|
I am sharing my build and debug. build logbreakpoint with CleanupConfigAndCredentialsCacheManager and InitConfigAndCredentialsCacheManagerbreakpoint with ConfigAndCredentialsCacheManager line 127 and line 130It seems that |
|
Just FYI, I was able to reproduce the issue with that CMake config ^. |
|
Could you try this? diff --git a/cpp/src/arrow/filesystem/CMakeLists.txt b/cpp/src/arrow/filesystem/CMakeLists.txt
index 0a31a64b7a..dec4bb6e3d 100644
--- a/cpp/src/arrow/filesystem/CMakeLists.txt
+++ b/cpp/src/arrow/filesystem/CMakeLists.txt
@@ -63,6 +63,23 @@ if(ARROW_AZURE)
endif()
if(ARROW_S3)
+ set(ARROW_S3_TEST_EXTRA_LINK_LIBS)
+ # arrow_shared/arrow_static is specified implicitly via
+ # arrow_testing_shared/arrow_testing_static but we specify
+ # arrow_shared/arrow_static explicitly here to ensure using libarrow
+ # before libaws* on link. If we use libaws*.a before libarrow,
+ # static variables storage of AWS SDK for C++ in libaws*.a may be
+ # mixed with one in libarrow.
+ if(ARROW_TEST_LINKAGE STREQUAL "shared")
+ list(APPEND ARROW_S3_TEST_EXTRA_LINK_LIBS arrow_shared)
+ else()
+ list(APPEND ARROW_S3_TEST_EXTRA_LINK_LIBS arrow_static)
+ endif()
+ list(APPEND
+ ARROW_S3_TEST_EXTRA_LINK_LIBS
+ ${AWSSDK_LINK_LIBRARIES}
+ Boost::filesystem
+ Boost::system)
add_arrow_test(s3fs_test
SOURCES
s3fs_test.cc
@@ -70,18 +87,17 @@ if(ARROW_S3)
EXTRA_LABELS
filesystem
EXTRA_LINK_LIBS
- ${AWSSDK_LINK_LIBRARIES}
- Boost::filesystem
- Boost::system)
+ ${ARROW_S3_TEST_EXTRA_LINK_LIBS})
if(TARGET arrow-s3fs-test)
set(ARROW_S3FS_TEST_COMPILE_DEFINITIONS)
get_target_property(AWS_CPP_SDK_S3_TYPE aws-cpp-sdk-s3 TYPE)
- # We need to initialize AWS C++ SDK for direct use (not via
+ # We need to initialize AWS SDK for C++ for direct use (not via
# arrow::fs::S3FileSystem) in arrow-s3fs-test if we use static AWS
- # C++ SDK and hide symbols of them. Because AWS C++ SDK has
- # internal static variables that aren't shared in libarrow and
+ # SDK for C++ and hide symbols of them. Because AWS SDK for C++
+ # has internal static variables that aren't shared in libarrow and
# arrow-s3fs-test. It means that arrow::fs::InitializeS3() doesn't
- # initialize AWS C++ SDK that is directly used in arrow-s3fs-test.
+ # initialize AWS SDK for C++ that is directly used in
+ # arrow-s3fs-test.
if(AWS_CPP_SDK_S3_TYPE STREQUAL "STATIC_LIBRARY"
AND CXX_LINKER_SUPPORTS_VERSION_SCRIPT)
list(APPEND ARROW_S3FS_TEST_COMPILE_DEFINITIONS "AWS_CPP_SDK_S3_PRIVATE_STATIC") |
|
That seems to fix it, the tests now run normally with this type of build. Though I'm seeing some test failures now and some escape sequence spam I wasn't seeing before but I changed my local minio setup since other things were wrong. I'll re-run the tests on a non-bundled build in a sec to compare. Edit: I actually get the original failure on my usual custom CMake preset (without this patch) and, with this patch, I get the same failures and spam mentioned above. It doesn't seem like it would be related to the fix here. I've included it below for reference: debug/arrow-s3fs-test output❯ ./debug/arrow-s3fs-test
Running main() from /Users/bryce/src/apache/arrow/cpp/build/_deps/googletest-src/googletest/src/gtest_main.cc
[==========] Running 69 tests from 7 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from S3OptionsTest
[ RUN ] S3OptionsTest.FromUri
[ OK ] S3OptionsTest.FromUri (495 ms)
[ RUN ] S3OptionsTest.FromAccessKey
[ OK ] S3OptionsTest.FromAccessKey (0 ms)
[ RUN ] S3OptionsTest.FromAssumeRole
[ OK ] S3OptionsTest.FromAssumeRole (5 ms)
[----------] 3 tests from S3OptionsTest (501 ms total)
[----------] 4 tests from S3RegionResolutionTest
[ RUN ] S3RegionResolutionTest.PublicBucket
[ OK ] S3RegionResolutionTest.PublicBucket (236 ms)
[ RUN ] S3RegionResolutionTest.RestrictedBucket
[ OK ] S3RegionResolutionTest.RestrictedBucket (240 ms)
[ RUN ] S3RegionResolutionTest.NonExistentBucket
[ OK ] S3RegionResolutionTest.NonExistentBucket (246 ms)
[ RUN ] S3RegionResolutionTest.InvalidBucketName
[ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
[----------] 4 tests from S3RegionResolutionTest (724 ms total)
[----------] 2 tests from S3FileSystemRegionTest
[ RUN ] S3FileSystemRegionTest.Default
[ OK ] S3FileSystemRegionTest.Default (2 ms)
[ RUN ] S3FileSystemRegionTest.EnvironmentVariable
[ OK ] S3FileSystemRegionTest.EnvironmentVariable (7 ms)
[----------] 2 tests from S3FileSystemRegionTest (10 ms total)
[----------] 1 test from TestMinioServer
[ RUN ] TestMinioServer.Connect
6/f7f6/f7f6^[\^[[45;1R^[[45;1R^[]11;rgb:f7f6/f7f6/f7f6^[\^[[45;1R^[]11;rgb:f7f6/f7f6/f7f6^[\^[[45;1R^[]11;rgb:f7f6/f7f6/f7f6^[\^[[45;1R^[]11;rgb:f7f6/f7f6/f7f6^[\^[[45;1R^[]11;rgb:f7f6/f7f6/f7f6^[\^[[45;1R^[]11;rgb:f7f6/f7f6/f7f6^[\^[[45;1R[ OK ] TestMinioServer.Connect (5426 ms)
[----------] 1 test from TestMinioServer (5426 ms total)
[----------] 32 tests from TestS3FS
[ RUN ] TestS3FS.GetFileInfoRoot
[ OK ] TestS3FS.GetFileInfoRoot (20 ms)
[ RUN ] TestS3FS.GetFileInfoBucket
[ OK ] TestS3FS.GetFileInfoBucket (23 ms)
[ RUN ] TestS3FS.GetFileInfoObject
[ OK ] TestS3FS.GetFileInfoObject (52 ms)
[ RUN ] TestS3FS.GetFileInfoSelector
[ OK ] TestS3FS.GetFileInfoSelector (58 ms)
[ RUN ] TestS3FS.GetFileInfoSelectorRecursive
[ OK ] TestS3FS.GetFileInfoSelectorRecursive (53 ms)
[ RUN ] TestS3FS.GetFileInfoGenerator
[ OK ] TestS3FS.GetFileInfoGenerator (57 ms)
[ RUN ] TestS3FS.GetFileInfoGeneratorStress
[ OK ] TestS3FS.GetFileInfoGeneratorStress (7573 ms)
[ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
[ OK ] TestS3FS.GetFileInfoGeneratorCancelled (20 ms)
[ RUN ] TestS3FS.CreateDir
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
Failed
Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
[ FAILED ] TestS3FS.CreateDir (50 ms)
[ RUN ] TestS3FS.DeleteFile
[ OK ] TestS3FS.DeleteFile (30 ms)
[ RUN ] TestS3FS.DeleteDir
[ OK ] TestS3FS.DeleteDir (74 ms)
[ RUN ] TestS3FS.DeleteDirContents
[ OK ] TestS3FS.DeleteDirContents (79 ms)
[ RUN ] TestS3FS.DeleteDirContentsAsync
[ OK ] TestS3FS.DeleteDirContentsAsync (61 ms)
[ RUN ] TestS3FS.CopyFile
[ OK ] TestS3FS.CopyFile (68 ms)
[ RUN ] TestS3FS.Move
[ OK ] TestS3FS.Move (77 ms)
[ RUN ] TestS3FS.OpenInputStream
[ OK ] TestS3FS.OpenInputStream (48 ms)
[ RUN ] TestS3FS.OpenInputStreamMetadata
[ OK ] TestS3FS.OpenInputStreamMetadata (43 ms)
[ RUN ] TestS3FS.OpenInputFile
[ OK ] TestS3FS.OpenInputFile (54 ms)
[ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
[ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (540 ms)
[ RUN ] TestS3FS.OpenOutputStreamSyncWrites
[ OK ] TestS3FS.OpenOutputStreamSyncWrites (496 ms)
[ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
[ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (24 ms)
[ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
[ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (25 ms)
[ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
[ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (32 ms)
[ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
[ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (46 ms)
[ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
[ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (63 ms)
[ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
[ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (73 ms)
[ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
[ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (75 ms)
[ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
[ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (73 ms)
[ RUN ] TestS3FS.OpenOutputStreamMetadata
[ OK ] TestS3FS.OpenOutputStreamMetadata (97 ms)
[ RUN ] TestS3FS.FileSystemFromUri
[ OK ] TestS3FS.FileSystemFromUri (36 ms)
[ RUN ] TestS3FS.NoCreateDeleteBucket
[ OK ] TestS3FS.NoCreateDeleteBucket (53 ms)
[ RUN ] TestS3FS.CustomRetryStrategy
[ OK ] TestS3FS.CustomRetryStrategy (42 ms)
[----------] 32 tests from TestS3FS (10129 ms total)
[----------] 26 tests from TestS3FSGeneric
[ RUN ] TestS3FSGeneric.Empty
[ OK ] TestS3FSGeneric.Empty (26 ms)
[ RUN ] TestS3FSGeneric.NormalizePath
[ OK ] TestS3FSGeneric.NormalizePath (24 ms)
[ RUN ] TestS3FSGeneric.CreateDir
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/test_util.cc:244: Failure
Failed
Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
[ FAILED ] TestS3FSGeneric.CreateDir (101 ms)
[ RUN ] TestS3FSGeneric.DeleteDir
[ OK ] TestS3FSGeneric.DeleteDir (161 ms)
[ RUN ] TestS3FSGeneric.DeleteDirContents
[ OK ] TestS3FSGeneric.DeleteDirContents (83 ms)
[ RUN ] TestS3FSGeneric.DeleteRootDirContents
[ OK ] TestS3FSGeneric.DeleteRootDirContents (37 ms)
[ RUN ] TestS3FSGeneric.DeleteFile
[ OK ] TestS3FSGeneric.DeleteFile (57 ms)
[ RUN ] TestS3FSGeneric.DeleteFiles
[ OK ] TestS3FSGeneric.DeleteFiles (107 ms)
[ RUN ] TestS3FSGeneric.MoveFile
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/test_util.cc:450: Failure
Failed
Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
[ FAILED ] TestS3FSGeneric.MoveFile (127 ms)
[ RUN ] TestS3FSGeneric.MoveDir
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
Filesystem doesn't allow moving directories
[ SKIPPED ] TestS3FSGeneric.MoveDir (20 ms)
[ RUN ] TestS3FSGeneric.CopyFile
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/test_util.cc:572: Failure
Failed
Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
[ FAILED ] TestS3FSGeneric.CopyFile (68 ms)
[ RUN ] TestS3FSGeneric.GetFileInfo
[ OK ] TestS3FSGeneric.GetFileInfo (60 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoVector
[ OK ] TestS3FSGeneric.GetFileInfoVector (68 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoSelector
[ OK ] TestS3FSGeneric.GetFileInfoSelector (91 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
[ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (91 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoAsync
[ OK ] TestS3FSGeneric.GetFileInfoAsync (54 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoGenerator
[ OK ] TestS3FSGeneric.GetFileInfoGenerator (73 ms)
[ RUN ] TestS3FSGeneric.OpenOutputStream
[ OK ] TestS3FSGeneric.OpenOutputStream (84 ms)
[ RUN ] TestS3FSGeneric.OpenAppendStream
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
Filesystem doesn't allow file appends
[ SKIPPED ] TestS3FSGeneric.OpenAppendStream (18 ms)
[ RUN ] TestS3FSGeneric.OpenInputStream
[ OK ] TestS3FSGeneric.OpenInputStream (56 ms)
[ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
[ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (70 ms)
[ RUN ] TestS3FSGeneric.OpenInputStreamAsync
[ OK ] TestS3FSGeneric.OpenInputStreamAsync (50 ms)
[ RUN ] TestS3FSGeneric.OpenInputFile
[ OK ] TestS3FSGeneric.OpenInputFile (79 ms)
[ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
[ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (68 ms)
[ RUN ] TestS3FSGeneric.OpenInputFileAsync
[ OK ] TestS3FSGeneric.OpenInputFileAsync (48 ms)
[ RUN ] TestS3FSGeneric.SpecialChars
[ OK ] TestS3FSGeneric.SpecialChars (87 ms)
[----------] 26 tests from TestS3FSGeneric (1820 ms total)
[----------] 1 test from S3GlobalOptions
[ RUN ] S3GlobalOptions.DefaultsLogLevel
[ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
[----------] 1 test from S3GlobalOptions (0 ms total)
[----------] Global test environment tear-down
[==========] 69 tests from 7 test suites ran. (18664 ms total)
[ PASSED ] 63 tests.
[ SKIPPED ] 2 tests, listed below:
[ SKIPPED ] TestS3FSGeneric.MoveDir
[ SKIPPED ] TestS3FSGeneric.OpenAppendStream
[ FAILED ] 4 tests, listed below:
[ FAILED ] TestS3FS.CreateDir
[ FAILED ] TestS3FSGeneric.CreateDir
[ FAILED ] TestS3FSGeneric.MoveFile
[ FAILED ] TestS3FSGeneric.CopyFile
4 FAILED TESTS
~/src/apache/arrow/cpp/build main* ≡ ⇣ 18s
❯ ]11;rgb:f7f6/f7f6/f7f6\[45;1R]11;rgb:f7f6/f7f6/f7f6\[45;1R]11;rgb:f7f6/f7f6/f7f6\[45;1R]11;rgb:f7f6/f7f6/f7f6\[45;1R]11;rgb:f7f6/f7f6/f7f6\[45;1R |
kou
pushed a commit
that referenced
this issue
Jul 15, 2024
### Rationale for this change To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order. ### What changes are included in this PR? - Adjusted `CMakeLists.txt` to set `ARROW_S3_TEST_EXTRA_LINK_LIBS`. - Ensured `libarrow` is linked before `libaws*` libraries. - Updated `s3fs_test` configuration to use the new link order. ### Are these changes tested? I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if `ARROW_S3` is set to OFF or if the configuration includes `exclude_tests=arrow-s3fs-test`, the test is excluded. ### Are there any user-facing changes? No. * GitHub Issue: #43164 Authored-by: Hyunseok Seo <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
|
Issue resolved by pull request 43230 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the usage question you have. Please include as many useful details as possible.
I'm encountering an error while running tests for
arrow-s3fs-test. Despite configuring AWS credentials using the `aws configure command, the following error occurs:Is there any documentation or guidance on how to execute the test properly?
Component(s)
C++
The text was updated successfully, but these errors were encountered: