Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement from_json_to_structs #2510

Open
wants to merge 67 commits into
base: branch-24.12
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
1376061
Implement `castStringsToBooleans`
ttnghia Oct 16, 2024
ff2f340
Merge branch 'branch-24.12' into convert_table
ttnghia Oct 16, 2024
c3fa10d
Implement `removeQuotes`
ttnghia Oct 16, 2024
ae2b41f
Rewrite using offsets and chars
ttnghia Oct 16, 2024
8d7ad2e
Fix empty input
ttnghia Oct 17, 2024
9e759c4
Misc
ttnghia Oct 17, 2024
2fff949
Add `nullifyIfNotQuoted` option for `removeQuotes`
ttnghia Oct 17, 2024
d09de41
Implement `castStringsToDecimals`
ttnghia Oct 18, 2024
576b65c
Implement `removeQuotesForFloats`
ttnghia Oct 18, 2024
2bd5335
Fix `removeQuotesForFloats`
ttnghia Oct 18, 2024
21c80a5
Implement `castStringsToIntegers`
ttnghia Oct 18, 2024
1a7d192
Implement non-legacy `castStringsToDates`
ttnghia Oct 18, 2024
dcb463e
WIP for `cast_strings_to_dates_legacy`
ttnghia Oct 21, 2024
f059c21
Revert "WIP for `cast_strings_to_dates_legacy`"
ttnghia Oct 21, 2024
207d6a3
Merge branch 'branch-24.12' into convert_table
ttnghia Oct 23, 2024
07b23ea
Fix compile issues
ttnghia Oct 23, 2024
de83a25
WIP: Implement `from_json_to_structs`
ttnghia Oct 24, 2024
443ca38
Merge branch 'branch-24.12' into convert_table
ttnghia Oct 24, 2024
6c2bd5e
Fix cmake
ttnghia Oct 24, 2024
904d857
Fix compile issues
ttnghia Oct 24, 2024
d84f1fe
Implement `castStringsToFloats`
ttnghia Oct 24, 2024
3024583
WIP
ttnghia Oct 24, 2024
d33d8e2
WIP: Implementing `fromJSONToStructs`
ttnghia Oct 25, 2024
295c36c
Merge branch 'branch-24.12' into convert_table
ttnghia Oct 28, 2024
1ea9cc8
Fix compile errors
ttnghia Oct 29, 2024
c1bb2d4
Cleanup
ttnghia Oct 29, 2024
f6634b4
Revert code as we still need them
ttnghia Oct 29, 2024
06b2c19
Add error check
ttnghia Oct 29, 2024
2dcdd11
Add more comments
ttnghia Oct 29, 2024
f3c391b
Cleanup
ttnghia Oct 29, 2024
52c42a6
Return as-is if the column is date/time
ttnghia Oct 29, 2024
19c64be
Update test
ttnghia Oct 30, 2024
cb9d252
Merge branch 'branch-24.12' into convert_table
ttnghia Oct 30, 2024
5d07db1
Update cudf
ttnghia Oct 30, 2024
39e3a9b
Revert "Update cudf"
ttnghia Oct 30, 2024
8628136
Merge branch 'branch-24.12' into convert_table
ttnghia Oct 30, 2024
df1428d
Update cudf
ttnghia Oct 30, 2024
0fd8d0e
Merge branch 'branch-24.12' into convert_table
ttnghia Nov 8, 2024
1d48906
Update cudf
ttnghia Nov 8, 2024
d9e1db5
Change header
ttnghia Nov 9, 2024
0f053a6
Rewrite JSONUtils.cpp
ttnghia Nov 9, 2024
8912e00
Implement a common function for converting column
ttnghia Nov 12, 2024
3614718
Rewrite `convert_data_type`
ttnghia Nov 12, 2024
6d9bbdc
Remove `cast_strings_to_dates`
ttnghia Nov 12, 2024
a832938
Implement `convert_data_type`
ttnghia Nov 13, 2024
44b885b
Fix compile errors
ttnghia Nov 13, 2024
ab45de8
Add `CUDF_FUNC_RANGE();`
ttnghia Nov 13, 2024
89e74a0
Fix schema
ttnghia Nov 13, 2024
27ef532
Complete `from_json_to_structs`
ttnghia Nov 13, 2024
5b65712
Fix null mask
ttnghia Nov 13, 2024
6788471
Write Javadoc
ttnghia Nov 13, 2024
49c78ce
Rewrite JNI
ttnghia Nov 13, 2024
9d16d43
Merge branch 'branch-24.12' into convert_table
ttnghia Nov 13, 2024
bb9029b
Remove deprecated function
ttnghia Nov 14, 2024
1243599
Revert test
ttnghia Nov 14, 2024
6f89fcd
Remove header
ttnghia Nov 14, 2024
deb3ebf
Rewrite Javadoc
ttnghia Nov 14, 2024
9dc641f
Rename variable
ttnghia Nov 14, 2024
53b121d
Rewrite docs
ttnghia Nov 14, 2024
69265b4
Revert test
ttnghia Nov 14, 2024
da4d1f6
Cleanup headers
ttnghia Nov 14, 2024
1d91e64
Cleanup
ttnghia Nov 14, 2024
d0fa2ae
Rewrite the conversion functions
ttnghia Nov 14, 2024
f375a4d
Move code
ttnghia Nov 14, 2024
034a5ec
Remove call to `make_structs_column`
ttnghia Nov 14, 2024
74d858c
Cleanup
ttnghia Nov 14, 2024
7a32b6f
Merge branch 'branch-24.12' into convert_table
ttnghia Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions src/main/cpp/src/JSONUtilsJni.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -198,4 +198,47 @@ JNIEXPORT jlong JNICALL Java_com_nvidia_spark_rapids_jni_JSONUtils_makeStructs(
}
CATCH_STD(env, 0);
}

JNIEXPORT jlong JNICALL
Java_com_nvidia_spark_rapids_jni_JSONUtils_castStringsToBooleans(JNIEnv* env, jclass, jlong j_input)
{
JNI_NULL_CHECK(env, j_input, "j_input is null", 0);

try {
cudf::jni::auto_set_device(env);
auto const input = *reinterpret_cast<cudf::column_view const*>(j_input);
return cudf::jni::ptr_as_jlong(spark_rapids_jni::cast_strings_to_booleans(input).release());
}
CATCH_STD(env, 0);
}

JNIEXPORT jlong JNICALL Java_com_nvidia_spark_rapids_jni_JSONUtils_castStringsToDecimals(
JNIEnv* env, jclass, jlong j_input, jint precision, jint scale, jboolean is_us_locale)
{
JNI_NULL_CHECK(env, j_input, "j_input is null", 0);

try {
cudf::jni::auto_set_device(env);
auto const input = *reinterpret_cast<cudf::column_view const*>(j_input);

return cudf::jni::ptr_as_jlong(
spark_rapids_jni::cast_strings_to_decimals(input, precision, scale, is_us_locale).release());
}
CATCH_STD(env, 0);
}

JNIEXPORT jlong JNICALL Java_com_nvidia_spark_rapids_jni_JSONUtils_removeQuotes(
JNIEnv* env, jclass, jlong j_input, jboolean nullify_if_not_quoted)
{
JNI_NULL_CHECK(env, j_input, "j_input is null", 0);

try {
cudf::jni::auto_set_device(env);
auto const input = *reinterpret_cast<cudf::column_view const*>(j_input);
return cudf::jni::ptr_as_jlong(
spark_rapids_jni::remove_quotes(input, nullify_if_not_quoted).release());
}
CATCH_STD(env, 0);
}

} // extern "C"
Loading
Loading