Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CELEBORN-836][BUILD] Initial support sbt #1757

Closed
wants to merge 44 commits into from
Closed

[CELEBORN-836][BUILD] Initial support sbt #1757

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
014dded
sbt
cfmcgrady Jun 26, 2023
ed700ce
pure sbt
cfmcgrady Jul 20, 2023
847404b
add spark 3 client
cfmcgrady Jul 24, 2023
5bdc2cc
CelebornBuild.scala
cfmcgrady Jul 24, 2023
42e0893
refactor to support spark 2.4
cfmcgrady Jul 24, 2023
b37570b
auto set scalaVersion
cfmcgrady Jul 24, 2023
40396eb
refactor SparkClientProjects
cfmcgrady Jul 24, 2023
dd0eb5e
set lz4/zstd version
cfmcgrady Jul 24, 2023
aa01dd8
enabled junit
cfmcgrady Jul 24, 2023
8da3e32
commonUnitTestDependencies
cfmcgrady Jul 24, 2023
0cf7dc1
ignore sbt-launch-*.jar
cfmcgrady Jul 25, 2023
8e638b2
add default repositories
cfmcgrady Jul 25, 2023
2996822
add license header
cfmcgrady Jul 25, 2023
3cea3e4
revert client-spark/common/pom.xml
cfmcgrady Jul 25, 2023
0313dd3
add license header
cfmcgrady Jul 25, 2023
19f9845
add license header
cfmcgrady Jul 25, 2023
3161a8b
extract dependencies version
cfmcgrady Jul 25, 2023
23b64e6
spark-3.0 support
cfmcgrady Jul 25, 2023
65f688d
spark-3.1 support
cfmcgrady Jul 25, 2023
b7db214
spark-3.2 support
cfmcgrady Jul 25, 2023
0dc5c83
spark-3.4 support
cfmcgrady Jul 25, 2023
bc9bf27
addresse comment
cfmcgrady Jul 25, 2023
af73dae
add worker module
cfmcgrady Jul 25, 2023
8de82fd
add spark-it module
cfmcgrady Jul 25, 2023
e6022e2
revert common/src/test/scala/org/apache/celeborn/common/CelebornConfS…
cfmcgrady Jul 25, 2023
40af990
Xmx2048m -> Xmx4g && add missing deps
cfmcgrady Jul 25, 2023
913bdd7
rat exclude repositories
cfmcgrady Jul 25, 2023
55592f3
fix spark-3.4 CI
cfmcgrady Jul 25, 2023
3b41963
fix spark-common CI for spark 3.4
cfmcgrady Jul 25, 2023
2e6852a
address comment
cfmcgrady Jul 25, 2023
9980561
order
cfmcgrady Jul 25, 2023
5a9aa09
fix project version
cfmcgrady Jul 25, 2023
cb7bec1
revert common/src/main/java/org/apache/celeborn/common/network/protoc…
cfmcgrady Jul 25, 2023
26edf32
address comment
cfmcgrady Jul 25, 2023
bc9a89e
add CN repositories
cfmcgrady Jul 26, 2023
8f16921
add repositories-cn.template
cfmcgrady Jul 26, 2023
4b2dcea
address comment
cfmcgrady Jul 26, 2023
668577e
add DEFAULT_ARTIFACT_REPOSITORY to repositories
cfmcgrady Jul 26, 2023
8f21700
address comment
cfmcgrady Jul 26, 2023
48a1aa9
set system property sbt.celeborn.default.artifact.repository
cfmcgrady Jul 26, 2023
5c8507d
correct path
cfmcgrady Jul 27, 2023
960a4d5
align final shaded jar name with maven
cfmcgrady Jul 27, 2023
bb161d7
add `celeborn-` prefix for project name
cfmcgrady Jul 27, 2023
fab0da0
address comment
cfmcgrady Jul 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .rat-excludes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,5 @@ NOTICE*
assets/**
build/apache-maven-*/**
build/scala-*/**
build/sbt-config/repositories
build/sbt-config/repositories-cn.template
build/sbt-config/**
**/benchmarks/**
1 change: 1 addition & 0 deletions build/sbt-config/repositories
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
local
local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
private: ${DEFAULT_ARTIFACT_REPOSITORY-file:///dev/null}
gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
maven-central
typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
Expand Down
10 changes: 7 additions & 3 deletions build/sbt-config/repositories-cn.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# As a Chinese developer facing network issues, you can accelerate the download speed of bootstrap/plugin/dependencies jar packages by executing the following command to configure the mirror:
#
# ```
# cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
# ```

[repositories]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the file support comments? we can add comments to guide user how to use this file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://stackoverflow.com/questions/56343343/how-do-you-comment-in-the-sbt-repositories-file

The commenting style commences with the symbol #

local
local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
private: ${DEFAULT_ARTIFACT_REPOSITORY-file:///dev/null}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the user has configured it, include the DEFAULT_ARTIFACT_REPOSITORY in the list of repositories. cc @pan3793

the variable substitution docs: https://www.scala-sbt.org/1.x/docs/Launcher-Configuration.html#Variable+Substitution

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the variable substitution required system property instead of the environment variable, updated in 48a1aa9

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for fixing it.

aliyun-maven: https://maven.aliyun.com/nexus/content/groups/public/
huawei-central: https://mirrors.huaweicloud.com/repository/maven/
maven-central