Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

Merged
merged 78 commits into from
Aug 21, 2023
Merged

GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

merged 78 commits into from
Aug 21, 2023

Commits on Aug 21, 2023

  1. Initial commit

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c35b86c View commit details
    Browse the repository at this point in the history

  2. Adding PutBinaryArray

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    0eeea96 View commit details
    Browse the repository at this point in the history

  3. Add FIXED_LEN_BYTE_ARRAY

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    450b0a6 View commit details
    Browse the repository at this point in the history

  4. More FLBAType work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    ad7b35f View commit details
    Browse the repository at this point in the history

  5. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    166ecf9 View commit details
    Browse the repository at this point in the history

  6. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    d365c7d View commit details
    Browse the repository at this point in the history

  7. DeltaByteArrayDecoderImpl

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c4e2226 View commit details
    Browse the repository at this point in the history

  8. Add Python test for FLBA and boolean with RLE

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    d1fbd21 View commit details
    Browse the repository at this point in the history

  9. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    ad63efc View commit details
    Browse the repository at this point in the history

  10. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    665048f View commit details
    Browse the repository at this point in the history

  11. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    4c9b90f View commit details
    Browse the repository at this point in the history

  12. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    23d663c View commit details
    Browse the repository at this point in the history

  13. Refactoring

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c6408f5 View commit details
    Browse the repository at this point in the history

  14. Apply suggestions from code review 

    Co-authored-by: mwish <[email protected]>
    @rok @mapleFU @pitrou
    2 people authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    54053f9 View commit details
    Browse the repository at this point in the history

  15. Linting and adding a python flba test

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    e88c838 View commit details
    Browse the repository at this point in the history

  16. Update cpp/src/parquet/encoding.cc 

    Co-authored-by: Will Jones <[email protected]>
    @rok @wjones127 @pitrou
    2 people authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c74e3f2 View commit details
    Browse the repository at this point in the history

  17. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    8994bf6 View commit details
    Browse the repository at this point in the history

  18. CheckDecode 

    Co-authored-by: mwish <[email protected]>
    @rok @mapleFU @pitrou
    2 people authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    925f1f7 View commit details
    Browse the repository at this point in the history

  19. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    800c3f8 View commit details
    Browse the repository at this point in the history

  20. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    10cadb1 View commit details
    Browse the repository at this point in the history

  21. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    597f567 View commit details
    Browse the repository at this point in the history

  22. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    74fbbdc View commit details
    Browse the repository at this point in the history

  23. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    703b8b6 View commit details
    Browse the repository at this point in the history

  24. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    6949b98 View commit details
    Browse the repository at this point in the history

  25. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    ea67049 View commit details
    Browse the repository at this point in the history

  26. Change to zero length suffix

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    9c31398 View commit details
    Browse the repository at this point in the history

  27. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    0d5140c View commit details
    Browse the repository at this point in the history

  28. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    d2bfd7f View commit details
    Browse the repository at this point in the history

  29. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    4d1debf View commit details
    Browse the repository at this point in the history

  30. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    37e5436 View commit details
    Browse the repository at this point in the history

  31. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    58c89bc View commit details
    Browse the repository at this point in the history

  32. Change exception message.

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    562edd8 View commit details
    Browse the repository at this point in the history

  33. Apply suggestions from code review 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @rok @pitrou
    rok and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    01f8f94 View commit details
    Browse the repository at this point in the history

  34. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    e6cd16b View commit details
    Browse the repository at this point in the history

  35. Chunk prefix lengths

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c07c865 View commit details
    Browse the repository at this point in the history

  36. Update cpp/src/parquet/encoding.cc 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @rok @pitrou
    rok and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    ca3660d View commit details
    Browse the repository at this point in the history

  37. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    6951e03 View commit details
    Browse the repository at this point in the history

  38. Change data distribution

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    02fe560 View commit details
    Browse the repository at this point in the history

  39. Refactor Put

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    0f49067 View commit details
    Browse the repository at this point in the history

  40. Batch suffixes

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    ad90f19 View commit details
    Browse the repository at this point in the history

  41. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    9fdb3e6 View commit details
    Browse the repository at this point in the history

  42. Linting

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    d78f3a9 View commit details
    Browse the repository at this point in the history

  43. Rename length

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    7c02878 View commit details
    Browse the repository at this point in the history

  44. Update cpp/src/parquet/encoding.cc 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @rok @pitrou
    rok and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    a5be621 View commit details
    Browse the repository at this point in the history

  45. ExecuteSpaced should use alternative InitData

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    e876222 View commit details
    Browse the repository at this point in the history

  46. Random data generator

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    744520c View commit details
    Browse the repository at this point in the history

  47. Concatenation probability

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    33d5111 View commit details
    Browse the repository at this point in the history

  48. Update encoding_test.cc

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    9ba2e0e View commit details
    Browse the repository at this point in the history

  49. Change random strning generation

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    bab955b View commit details
    Browse the repository at this point in the history

  50. test

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    a3b3d0c View commit details
    Browse the repository at this point in the history

  51. Minor change

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    1867347 View commit details
    Browse the repository at this point in the history

  52. Refactor DeltaByteArrayEncodingDirectPut

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    bb54a87 View commit details
    Browse the repository at this point in the history

  53. Update cpp/src/parquet/encoding.cc 

    Co-authored-by: Gang Wu <[email protected]>
    @rok @wgtmac @pitrou
    2 people authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    73316cf View commit details
    Browse the repository at this point in the history

  54. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    ba4538b View commit details
    Browse the repository at this point in the history

  55. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    bc7fcde View commit details
    Browse the repository at this point in the history

  56. Apply suggestions from code review 

    Co-authored-by: Gang Wu <[email protected]>
    @rok @wgtmac @pitrou
    2 people authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    3dc32b3 View commit details
    Browse the repository at this point in the history

  57. Repeats

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    51ff60a View commit details
    Browse the repository at this point in the history

  58. Enable DirectPut tests

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    1d2fa5f View commit details
    Browse the repository at this point in the history

  59. GeneratePrefixedData

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    b53e84e View commit details
    Browse the repository at this point in the history

  60. Work

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    8911b5e View commit details
    Browse the repository at this point in the history

  61. Enable DeltaByteArrayEncodingDirectPut sans FLBAType

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    2538ab8 View commit details
    Browse the repository at this point in the history

  62. Refactoring DeltaByteArrayEncodingDirectPut

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c19a528 View commit details
    Browse the repository at this point in the history

  63. random_byte_array etc

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    74da1b7 View commit details
    Browse the repository at this point in the history

  64. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    92b457a View commit details
    Browse the repository at this point in the history

  65. Apply suggestions from code review 

    Co-authored-by: mwish <[email protected]>
Co-authored-by: Gang Wu <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
    @rok @mapleFU
    @wgtmac @pitrou
    4 people committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    e4b96aa View commit details
    Browse the repository at this point in the history

  66. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    45a6d51 View commit details
    Browse the repository at this point in the history

  67. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    1e10149 View commit details
    Browse the repository at this point in the history

  68. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    6c6fbde View commit details
    Browse the repository at this point in the history

  69. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    6930a79 View commit details
    Browse the repository at this point in the history

  70. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    409a6ee View commit details
    Browse the repository at this point in the history

  71. Apply suggestions from code review 

    Co-authored-by: mwish <[email protected]>
    @rok @mapleFU @pitrou
    2 people authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    7dc32e1 View commit details
    Browse the repository at this point in the history

  72. Switch to resizable buffer

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    578d7de View commit details
    Browse the repository at this point in the history

  73. Apply suggestions from code review 

    Co-authored-by: Antoine Pitrou <[email protected]>
    @rok @pitrou
    rok and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    8167771 View commit details
    Browse the repository at this point in the history

  74. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    0aaa6b4 View commit details
    Browse the repository at this point in the history

  75. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    9f0cdbb View commit details
    Browse the repository at this point in the history

  76. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    46aa303 View commit details
    Browse the repository at this point in the history

  77. Review feedback

    @rok @pitrou
    rok authored and pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    acc40ed View commit details
    Browse the repository at this point in the history

  78. Rewrite Binary/FLBA specializations to correctly account for Accumula… 

    …tor APIs
    @pitrou
    pitrou committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    5039cf9 View commit details
    Browse the repository at this point in the history