[CELEBORN-843][BUILD] `sbt` support flink-related module build/test #1764

cfmcgrady · 2023-07-27T03:56:51Z

What changes were proposed in this pull request?

This PR adds packaging and testing support for Flink-related modules using SBT based on #1757

Why are the changes needed?

improve project build speed

running flink-it tests with -Pflink-1.14

sbt:celeborn> project flink-it
sbt:flink-it> clean
sbt:flink-it> test
[success] Total time: 136 s (02:16), completed 2023-7-27 11:55:10

running flink-it tests with -Pflink-1.17

$ ./build/sbt -Pflink-1.17
sbt:celeborn> project flink-it
sbt:flink-it> clean
sbt:flink-it> test
[success] Total time: 168 s (02:48), completed 2023-7-27 11:28:35

packing and shading the flink 1.14 client

$ ./build/sbt -Pflink-1.14
sbt:celeborn> clean
sbt:celeborn> project celeborn-client-flink-1_14-shaded
sbt:celeborn-client-flink-1_14-shaded> assembly
[success] Total time: 35 s, completed 2023-7-27 11:51:54

packing and shading the flink 1.17 client

$ ./build/sbt -Pflink-1.17
sbt:celeborn> clean
sbt:celeborn> project celeborn-client-flink-1_17-shaded
sbt:celeborn-client-flink-1_17-shaded> assembly
[success] Total time: 39 s, completed 2023-7-27 11:49:20

Does this PR introduce any user-facing change?

yes

How was this patch tested?

tested locally

codecov · 2023-07-27T04:06:33Z

Codecov Report

Merging #1764 (77a72ee) into main (5f0295e) will decrease coverage by 0.00%.
Report is 4 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1764      +/-   ##
==========================================
- Coverage   46.81%   46.80%   -0.00%     
==========================================
  Files         162      162              
  Lines       10012    10012              
  Branches      923      923              
==========================================
- Hits         4686     4685       -1     
+ Misses       5019     5018       -1     
- Partials      307      309       +2

see 5 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

cfmcgrady · 2023-07-31T08:45:53Z

diff between the celeborn-client-flink-1.17-shaded_2.12-0.4.0-SNAPSHOT.jar file in sbt and maven:

diff --git a/maven-files.out b/sbt-files.out
index 8228cae..8db58f9 100644
--- a/maven-files.out
+++ b/sbt-files.out
@@ -1,19 +1,6 @@
 ./celeborn-client-flink-1.17-shaded_2.12-0.4.0-SNAPSHOT.jar
-./google/protobuf/any.proto
-./google/protobuf/api.proto
-./google/protobuf/compiler/plugin.proto
-./google/protobuf/descriptor.proto
-./google/protobuf/duration.proto
-./google/protobuf/empty.proto
-./google/protobuf/field_mask.proto
-./google/protobuf/source_context.proto
-./google/protobuf/struct.proto
-./google/protobuf/timestamp.proto
-./google/protobuf/type.proto
-./google/protobuf/wrappers.proto
-./META-INF/DEPENDENCIES
+./META-INF/INDEX.LIST
 ./META-INF/io.netty.versions.properties
-./META-INF/LICENSE
 ./META-INF/LICENSE.txt
 ./META-INF/MANIFEST.MF
 ./META-INF/maven/com.google.guava/guava/pom.properties
@@ -80,53 +67,12 @@
 ./META-INF/maven/io.netty/netty-transport-sctp/pom.xml
 ./META-INF/maven/io.netty/netty-transport-udt/pom.properties
 ./META-INF/maven/io.netty/netty-transport-udt/pom.xml
-./META-INF/maven/org.apache.celeborn/celeborn-client_2.12/pom.properties
-./META-INF/maven/org.apache.celeborn/celeborn-client_2.12/pom.xml
-./META-INF/maven/org.apache.celeborn/celeborn-client-flink-1.17_2.12/pom.properties
-./META-INF/maven/org.apache.celeborn/celeborn-client-flink-1.17_2.12/pom.xml
-./META-INF/maven/org.apache.celeborn/celeborn-client-flink-1.17-shaded_2.12/pom.properties
-./META-INF/maven/org.apache.celeborn/celeborn-client-flink-1.17-shaded_2.12/pom.xml
-./META-INF/maven/org.apache.celeborn/celeborn-client-flink-common_2.12/pom.properties
-./META-INF/maven/org.apache.celeborn/celeborn-client-flink-common_2.12/pom.xml
-./META-INF/maven/org.apache.celeborn/celeborn-common_2.12/pom.properties
-./META-INF/maven/org.apache.celeborn/celeborn-common_2.12/pom.xml
 ./META-INF/maven/org.apache.commons/commons-lang3/pom.properties
 ./META-INF/maven/org.apache.commons/commons-lang3/pom.xml
 ./META-INF/maven/org.jctools/jctools-core/pom.properties
 ./META-INF/maven/org.jctools/jctools-core/pom.xml
-./META-INF/native-image/io.netty/netty-buffer/native-image.properties
-./META-INF/native-image/io.netty/netty-codec-dns/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-haproxy/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-http2/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-http2/native-image.properties
-./META-INF/native-image/io.netty/netty-codec-http/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-http/native-image.properties
-./META-INF/native-image/io.netty/netty-codec-memcache/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-mqtt/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec/native-image.properties
-./META-INF/native-image/io.netty/netty-codec-redis/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-smtp/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-socks/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-stomp/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-codec-xml/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-common/native-image.properties
-./META-INF/native-image/io.netty/netty-handler/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-handler/native-image.properties
-./META-INF/native-image/io.netty/netty-handler-proxy/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-resolver-dns/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-resolver-dns/native-image.properties
-./META-INF/native-image/io.netty/netty-transport-classes-epoll/jni-config.json
-./META-INF/native-image/io.netty/netty-transport-classes-epoll/native-image.properties
-./META-INF/native-image/io.netty/netty-transport-classes-epoll/reflect-config.json
-./META-INF/native-image/io.netty/netty-transport-classes-epoll/resource-config.json
-./META-INF/native-image/io.netty/netty-transport/generated/handlers/reflect-config.json
-./META-INF/native-image/io.netty/netty-transport/native-image.properties
-./META-INF/native-image/io.netty/netty-transport/reflection-config.json
-./META-INF/native-image/io.netty/netty-transport-sctp/generated/handlers/reflect-config.json
 ./META-INF/native/liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so
 ./META-INF/native/liborg_apache_celeborn_shaded_netty_transport_native_epoll_x86_64.so
-./META-INF/NOTICE
 ./META-INF/NOTICE.txt
 ./META-INF/services/reactor.blockhound.integration.BlockHoundIntegration
 ./org/apache/celeborn/client/ApplicationHeartbeater$$anon$1.class
@@ -844,6 +790,8 @@
 ./org/apache/celeborn/common/write/SimplePushStrategy.class
 ./org/apache/celeborn/common/write/SlowStartPushStrategy$CongestControlContext.class
 ./org/apache/celeborn/common/write/SlowStartPushStrategy.class
+./org/apache/celeborn/package$.class
+./org/apache/celeborn/package.class
 ./org/apache/celeborn/plugin/flink/AbstractRemoteShuffleEnvironment.class
 ./org/apache/celeborn/plugin/flink/AbstractRemoteShuffleInputGateFactory.class
 ./org/apache/celeborn/plugin/flink/AbstractRemoteShuffleResultPartitionFactory.class
@@ -6664,4 +6612,3 @@
 ./org/apache/celeborn/shaded/org/roaringbitmap/RunContainer.class
 ./org/apache/celeborn/shaded/org/roaringbitmap/Util.class
 ./org/apache/celeborn/shaded/org/roaringbitmap/WordStorage.class
-./TransportMessages.proto

waitinfuture · 2023-07-31T12:17:25Z

Ping @RexXiong @FMX @zhongqiangczq could you take a look at this PR? thanks!

pan3793 · 2023-07-31T12:22:09Z

project/CelebornBuild.scala

@@ -202,7 +213,7 @@ object CelebornCommon {
        "io.dropwizard.metrics" % "metrics-core" % metricsVersion,
        "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion,
        "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion,
-        "io.netty" % "netty-all" % nettyVersion,
+        ("io.netty" % "netty-all" % nettyVersion).exclude("io.netty", "netty-handler-ssl-ocsp"),


can you remove this change? as we discussed before, do exclusion after introducing the dependency list audit mechanism.

zhongqiangczq · 2023-08-01T02:19:35Z

can sbt still use the maven local repository to avoid download dependencies again ?

cfmcgrady · 2023-08-01T03:22:09Z

can sbt still use the maven local repository to avoid download dependencies again ?

Good catch, it should be, let me raise a separate PR to support this.

zhongqiangczq

LGTM

### What changes were proposed in this pull request? As title ### Why are the changes needed? #1764 (comment) before this PR ```shell > ./build/sbt -Pspark-3.3 "clean; show celeborn-common/csrResolvers" [info] * FileRepository(local, Patterns(ivyPatterns=Vector(/Users/fchen/.ivy2/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)([branch]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), artifactPatterns=Vector(/Users/fchen/.ivy2/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)([branch]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), isMavenCompatible=false, descriptorOptional=false, skipConsistencyCheck=false), FileConfiguration(true, None)) [info] * private: file:/dev/null [info] * aliyun-maven: https://maven.aliyun.com/nexus/content/groups/public/ [info] * huawei-central: https://mirrors.huaweicloud.com/repository/maven/ [success] Total time: 0 s, completed 2023-8-1 11:40:25 ``` ```shell > ./build/sbt -Pspark-3.3 "clean; show celeborn-client-spark-3/managedClasspath" | grep spark [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-core_2.12/3.3.2/spark-core_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-sql_2.12/3.3.2/spark-sql_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-launcher_2.12/3.3.2/spark-launcher_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-kvstore_2.12/3.3.2/spark-kvstore_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-network-common_2.12/3.3.2/spark-network-common_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-network-shuffle_2.12/3.3.2/spark-network-shuffle_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-unsafe_2.12/3.3.2/spark-unsafe_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-tags_2.12/3.3.2/spark-tags_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-sketch_2.12/3.3.2/spark-sketch_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/Library/Caches/Coursier/v1/https/mirrors.huaweicloud.com/repository/maven/org/apache/spark/spark-catalyst_2.12/3.3.2/spark-catalyst_2.12-3.3.2.jar) ``` after this PR ```shell > ./build/sbt -Pspark-3.3 "clean; show celeborn-common/csrResolvers" [success] Total time: 0 s, completed 2023-8-1 11:45:02 [info] * FileRepository(local, Patterns(ivyPatterns=Vector(/Users/fchen/.ivy2/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)([branch]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), artifactPatterns=Vector(/Users/fchen/.ivy2/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)([branch]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), isMavenCompatible=false, descriptorOptional=false, skipConsistencyCheck=false), FileConfiguration(true, None)) [info] * mavenLocal: file:/Users/fchen/.m2/repository/ [info] * private: file:/dev/null [info] * aliyun-maven: https://maven.aliyun.com/nexus/content/groups/public/ [info] * huawei-central: https://mirrors.huaweicloud.com/repository/maven/ [success] Total time: 0 s, completed 2023-8-1 11:45:02 ``` ```shell > ./build/sbt -Pspark-3.3 "clean; show celeborn-client-spark-3/managedClasspath" | grep spark [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-core_2.12/3.3.2/spark-core_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-sql_2.12/3.3.2/spark-sql_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-launcher_2.12/3.3.2/spark-launcher_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-kvstore_2.12/3.3.2/spark-kvstore_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-network-common_2.12/3.3.2/spark-network-common_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-network-shuffle_2.12/3.3.2/spark-network-shuffle_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-unsafe_2.12/3.3.2/spark-unsafe_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-tags_2.12/3.3.2/spark-tags_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-sketch_2.12/3.3.2/spark-sketch_2.12-3.3.2.jar) [info] * Attributed(/Users/fchen/.m2/repository/org/apache/spark/spark-catalyst_2.12/3.3.2/spark-catalyst_2.12-3.3.2.jar) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA Closes #1780 from cfmcgrady/sbt-maven-local. Authored-by: Fu Chen <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

pan3793 · 2023-08-01T04:29:58Z

Thanks, merged to main

### What changes were proposed in this pull request? This PR adds new GitHub Actions workflows to enable Continuous Integration using SBT based on #1764 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA Closes #1771 from cfmcgrady/sbt-ci. Authored-by: Fu Chen <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

cfmcgrady force-pushed the sbt-flink branch from ee0f90f to 03d9464 Compare July 27, 2023 09:22

cfmcgrady mentioned this pull request Jul 27, 2023

[CELEBORN-850][INFRA] Add SBT CI #1771

Closed

cfmcgrady added 6 commits July 28, 2023 10:50

flink build support

ff2dd78

add flink 1.15/1.17 support

6673ef3

add flink-it/flink-client-shaded support

9c7626f

rebase branch pure-sbt

a241d54

align final jar name with maven

d1b7a21

add prefix and update project name

3fb80a0

cfmcgrady force-pushed the sbt-flink branch from 03d9464 to 3fb80a0 Compare July 28, 2023 02:50

pan3793 approved these changes Jul 28, 2023

View reviewed changes

pan3793 requested review from FMX and RexXiong July 28, 2023 02:53

nit

6234eed

cfmcgrady mentioned this pull request Jul 31, 2023

[CELEBORN-836][BUILD] Initial support sbt #1757

Closed

include RoaringBitmap

fc92874

pan3793 reviewed Jul 31, 2023

View reviewed changes

cfmcgrady added 2 commits July 31, 2023 20:32

address comment

37fa057

trigger GitHub actions

77a72ee

cfmcgrady mentioned this pull request Aug 1, 2023

[CELEBORN-867][BUILD] Add maven local repository to sbt respositories #1780

Closed

zhongqiangczq approved these changes Aug 1, 2023

View reviewed changes

pan3793 closed this in 40e416c Aug 1, 2023

cfmcgrady deleted the sbt-flink branch August 1, 2023 04:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CELEBORN-843][BUILD] `sbt` support flink-related module build/test #1764

[CELEBORN-843][BUILD] `sbt` support flink-related module build/test #1764

cfmcgrady commented Jul 27, 2023

codecov bot commented Jul 27, 2023 •

edited

Loading

cfmcgrady commented Jul 31, 2023

waitinfuture commented Jul 31, 2023

pan3793 Jul 31, 2023

cfmcgrady Jul 31, 2023

zhongqiangczq commented Aug 1, 2023

cfmcgrady commented Aug 1, 2023

zhongqiangczq left a comment

pan3793 commented Aug 1, 2023

[CELEBORN-843][BUILD] sbt support flink-related module build/test #1764

[CELEBORN-843][BUILD] sbt support flink-related module build/test #1764

Conversation

cfmcgrady commented Jul 27, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

codecov bot commented Jul 27, 2023 • edited Loading

Codecov Report

cfmcgrady commented Jul 31, 2023

waitinfuture commented Jul 31, 2023

pan3793 Jul 31, 2023

Choose a reason for hiding this comment

cfmcgrady Jul 31, 2023

Choose a reason for hiding this comment

zhongqiangczq commented Aug 1, 2023

cfmcgrady commented Aug 1, 2023

zhongqiangczq left a comment

Choose a reason for hiding this comment

pan3793 commented Aug 1, 2023

[CELEBORN-843][BUILD] `sbt` support flink-related module build/test #1764

[CELEBORN-843][BUILD] `sbt` support flink-related module build/test #1764

codecov bot commented Jul 27, 2023 •

edited

Loading