Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement from_json_to_structs #2510

Open
wants to merge 67 commits into
base: branch-24.12
Choose a base branch
from
Open

Implement from_json_to_structs #2510

wants to merge 67 commits into from

Conversation

ttnghia
Copy link
Collaborator

@ttnghia ttnghia commented Oct 17, 2024 •
edited
Loading

This implements the function from_json_to_structs, which performs parsing a strings column into a structs column with a given data schema. This is the case of running from_json SQL function in Spark with struct schema.

Closes #2468.

Contribute to NVIDIA/spark-rapids#11560.

@ttnghia ttnghia requested a review from revans2 October 17, 2024 03:07
@ttnghia ttnghia self-assigned this Oct 17, 2024
@ttnghia ttnghia mentioned this pull request Oct 17, 2024
Open
@ttnghia
Fix empty input
8d7ad2e 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Misc
9e759c4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add nullifyIfNotQuoted option for removeQuotes
2fff949 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Implement castStringsToDecimals
d09de41 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Oct 21, 2024

CC @karthikeyann.

@karthikeyann karthikeyann mentioned this pull request Oct 21, 2024
Open
@ttnghia
Rewrite JSONUtils.cpp
0f053a6
@ttnghia
Implement a common function for converting column
8912e00 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite convert_data_type
3614718 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Remove cast_strings_to_dates
6d9bbdc 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Implement convert_data_type
a832938 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix compile errors
44b885b 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add CUDF_FUNC_RANGE();
ab45de8 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix schema
89e74a0 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Complete from_json_to_structs
27ef532 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix null mask
5b65712 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Write Javadoc
6788471 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite JNI
49c78ce 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Merge branch 'branch-24.12' into convert_table
9d16d43 
# Conflicts:
#	src/main/cpp/src/json_utils.hpp
#	thirdparty/cudf
@ttnghia
Remove deprecated function
bb9029b 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia mentioned this pull request Nov 14, 2024
Open
@ttnghia ttnghia changed the title Convert strings columns output from cudf::read_json to other types Implement from_json_to_structs Nov 14, 2024
@ttnghia
Revert test
1243599 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Remove header
6f89fcd 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite Javadoc
deb3ebf 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rename variable
9dc641f 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite docs
53b121d 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Revert test
69265b4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup headers
da4d1f6 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
1d91e64 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite the conversion functions
d0fa2ae 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Move code
f375a4d 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Remove call to make_structs_column
034a5ec 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
74d858c 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia marked this pull request as ready for review November 14, 2024 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hyperbolic2346 hyperbolic2346 hyperbolic2346 left review comments

@revans2 revans2 Awaiting requested review from revans2

At least 1 approving review is required to merge this pull request.

Assignees

@ttnghia ttnghia

Labels
enhancement New feature or request feature request performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Implement a function to convert a list of strings columns into a structs column with desired schema
2 participants
@ttnghia @hyperbolic2346