Fixed issues with split build

aiping312 · Oct 2, 2015 · 25b6b07 · 25b6b07
1 parent f04325b
commit 25b6b07
Show file tree

Hide file tree

Showing 8 changed files with 33 additions and 91 deletions.
diff --git a/DEVELOPER.md b/DEVELOPER.md
@@ -192,6 +192,7 @@ To pull in a merge request you should generally follow the command line instruct
 3. Merge the pull request into your local test branch.
 
         $ git pull <remote_repo_url> <remote_branch>
+    You can use `./dev-tools/storm-merge.py <pull-number>` to produce the above command most of the time.
 
 4.  Assuming that the pull request merges without any conflicts:
     Update the top-level `CHANGELOG.md`, and add in the JIRA ticket number (example: `STORM-1234`) and ticket
@@ -222,36 +223,10 @@ To pull in a merge request you should generally follow the command line instruct
 
 # Build the code and run the tests
 
-Maven has some issues with maven pugins, shading and multi-module projects which force Storm's build process to be somewhat non-standard.
-
-Essentially what happens is that if you build a multi-module project where one module depends on another module that is a part of the same build, maven tries to be smart and will use the pom.xml and .jar or .class directory from the module that is a dependency.
-
-So if we have a project set up like the following where we have flux depend on storm-core, and storm-core is shaded.  flux will see different versions of the code and pom.xml from storm-core depending on what maven command is run.
-
-- storm.git/
-  - pom.xml
-  - storm-core/
-    - pom.xml
-    - dependency-reduce-pom.xml
-    - target/
-      - classes/
-      - storm-core.jar (shaded)
-      - original.storm-core.jar (no-shaded)
-  - flux/
-    - pom.xml (depends on storm-core)
-    - target/
-      - classes/
-      - flux.uber.jar
-
-If I run `mvn clean compile` the classpath for compiling flux will include `storm.git/storm-core/target/classes` which are not shaded, and if flux is written to use storm-core shaded class it will fail to compile.  flux will also see the full set of non-shaded dependencies from storm-core, so if flux is creating an uber jar, it will assume that all of those dependencies will be provided by storm-core, when in reality they will not be.
-
-If I run `mvn clean package` or `mvn clean install` flux will see the shaded storm-core.jar on its classpath, but it will still see the full non-shaded set of storm-core's dependencies.
-
-The only way I found to make flux use the dependency-reduced-pom.xml from storm-core, is to first build and install storm-core by itself, and then build and install flux without storm-core.  That way storm-core will install the shaded jar and dependency-reduced-pom.xml into the local repository.  Then flux will get its dependencies from the local repo instead of trying to rebuild storm-core.
-
-Because of this we have split the build into two profiles storm-core and storm-more.  By default both build together which works OK for simple testing, but when doing a release it is important to build them in two phases or flux, storm-starter, and possibly others will not be packaged correctly.
-
 ## Prerequisites
+Firt of all you need to make sure you are using maven 3.2.5 or below.  There is a bug in later versions of maven as linked to from https://issues.apache.org/jira/browse/MSHADE-206 that
+cause shaded dependencies to not be packaged correctly.  Also please be aware that because we are shading dependencies mvn dependency:tree will not always show the dependencies correctly. 
+
 In order to build `storm` you need `python`, `ruby` and `nodejs`. In order to avoid an overful page we don't provide platform/OS specific installation instructions for those here. Please refer to you platform's/OS' documentation for support.
 
 The `ruby` package manager `rvm` and `nodejs` package manager `nvm` are for convenience and are used in the tests which run on [travis](https://travis-ci.org/apache/storm). They can be installed using `curl -L https://get.rvm.io | bash -s stable --autolibs=enabled && source ~/.profile` (see the [rvm installation instructions](https://github.com/rvm/rvm) for details) and `wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.26.1/install.sh | bash && source ~/.bashrc` (see the [nvm installation instructions](https://github.com/creationix/nvm) for details).
@@ -270,16 +245,9 @@ in order to get started as fast as possible. Users can still install a specific
 
 The following commands must be run from the top-level directory.
 
-The first step is to build/install the plugins and storm-core
-
-`mvn clean install -Pstorm-core`
+`mvn clean install`
 
-If you wish to skip the unit tests you can do this by adding `-DskipTests` to the command line. Once this
-completes successfully you may run
-
-`mvn clean install -Pstorm-more`
-
-to build and test all of the external libraries and examples.  Again if you want to include `-DskipTests` you can.
+If you wish to skip the unit tests you can do this by adding `-DskipTests` to the command line. 
 
 In case you modified `storm.thrift`, you have to regenerate thrift code as java and python code before compiling whole project.
 
@@ -302,7 +270,6 @@ You can also run tests selectively with `-Dtest=<test_name>`.  This works for bo
 
 Unfortunately you might experience failures in clojure tests which are wrapped in the `maven-clojure-plugin` and thus doesn't provide too much useful output at first sight - you might end up with a maven test failure with an error message as unhelpful as `Clojure failed.`. In this case it's recommended to look into `target/test-reports` of the failed project to see what actual tests have failed or scroll through the maven output looking for obvious issues like missing binaries.
 
-
 <a name="packaging"></a>
 
 ## Create a Storm distribution (packaging)
@@ -311,8 +278,7 @@ You can create a _distribution_ (like what you can download from Apache) as foll
 do not use the Maven release plugin because creating an official release is the task of our release manager.
 
     # First, build the code.
-    $ mvn clean install -Pstorm-core # you may skip tests with `-DskipTests=true` to save time
-    $ mvn clean install -Pstorm-more # you may skip tests with `-DskipTests=true` to save time
+    $ mvn clean install # you may skip tests with `-DskipTests=true` to save time
 
     # Create the binary distribution.
     $ cd storm-dist/binary && mvn package

diff --git a/dev-tools/travis/travis-install.sh b/dev-tools/travis/travis-install.sh
@@ -26,21 +26,12 @@ TRAVIS_SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
 
 cd ${STORM_SRC_ROOT_DIR}
 
-python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-core.txt" mvn clean install -DskipTests -Pnative -Pstorm-core --batch-mode
+python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install.txt" mvn clean install -DskipTests -Pnative --batch-mode
 BUILD_RET_VAL=$?
 
 if [[ "$BUILD_RET_VAL" != "0" ]];
 then
-  cat "install-storm-core.txt"
-  exit ${BUILD_RET_VAL}
-fi
-
-python ${TRAVIS_SCRIPT_DIR}/save-logs.py "install-storm-more.txt" mvn clean install -DskipTests -Pstorm-more
-BUILD_RET_VAL=$?
-
-if [[ "$BUILD_RET_VAL" != "0" ]];
-then
-  cat "install-storm-more.txt"
+  cat "install.txt"
 fi
 
 exit ${BUILD_RET_VAL}
diff --git a/dev-tools/travis/travis-script.sh b/dev-tools/travis/travis-script.sh
@@ -15,11 +15,12 @@ PYTHON_VERSION_TO_FILE=`python -V > /tmp/python_version 2>&1`
 PYTHON_VERSION=`cat /tmp/python_version`
 RUBY_VERSION=`ruby -v`
 NODEJS_VERSION=`node -v`
+MVN_VERSION=`mvn -v`
 
 echo "Python version : $PYTHON_VERSION"
 echo "Ruby version : $RUBY_VERSION"
 echo "NodeJs version : $NODEJS_VERSION"
-
+echo "mvn version : $MVN_VERSION"
 
 STORM_SRC_ROOT_DIR=$1
 
@@ -31,7 +32,7 @@ cd ${STORM_SRC_ROOT_DIR}
 export STORM_TEST_TIMEOUT_MS=100000
 
 # We now lean on Travis CI's implicit behavior, ```mvn clean install -DskipTests``` before running script
-mvn --batch-mode test -fae -Pnative -Pstorm-core && mvn --batch-mode test -fae -Pstorm-more
+mvn --batch-mode install -fae -Pnative
 BUILD_RET_VAL=$?
 
 for dir in `find . -type d -and -wholename \*/target/\*-reports`;

diff --git a/examples/storm-starter/pom.xml b/examples/storm-starter/pom.xml
@@ -142,7 +142,6 @@
             <artifactId>maven-shade-plugin</artifactId>
             <configuration>
                 <createDependencyReducedPom>true</createDependencyReducedPom>
-                <minimizeJar>true</minimizeJar>
             </configuration>
             <executions>
                 <execution>

diff --git a/external/flux/flux-core/pom.xml b/external/flux/flux-core/pom.xml
@@ -72,7 +72,6 @@
             <artifactId>maven-shade-plugin</artifactId>
             <configuration>
                 <createDependencyReducedPom>true</createDependencyReducedPom>
-                <minimizeJar>true</minimizeJar>
             </configuration>
             <executions>
                 <execution>

diff --git a/external/flux/flux-examples/pom.xml b/external/flux/flux-examples/pom.xml
@@ -117,7 +117,6 @@
                 <artifactId>maven-shade-plugin</artifactId>
                 <configuration>
                     <createDependencyReducedPom>true</createDependencyReducedPom>
-                    <minimizeJar>true</minimizeJar>
                 </configuration>
                 <executions>
                     <execution>

diff --git a/external/storm-eventhubs/pom.xml b/external/storm-eventhubs/pom.xml
@@ -49,7 +49,6 @@
                     </execution>
                 </executions>
                 <configuration>
-                    <minimizeJar>true</minimizeJar>
                     <transformers>
                         <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer">
                         </transformer>

diff --git a/pom.xml b/pom.xml
@@ -215,40 +215,28 @@
         <junit.version>4.11</junit.version>
     </properties>
 
+    <modules>
+        <module>storm-multilang/javascript</module>
+        <module>storm-multilang/python</module>
+        <module>storm-multilang/ruby</module>
+        <module>storm-buildtools/maven-shade-clojure-transformer</module>
+        <module>storm-buildtools/storm-maven-plugins</module>
+        <module>storm-core</module>
+        <module>external/storm-kafka</module>
+        <module>external/storm-hdfs</module>
+        <module>external/storm-hbase</module>
+        <module>external/storm-hive</module>
+        <module>external/storm-jdbc</module>
+        <module>external/storm-redis</module>
+        <module>external/storm-eventhubs</module>
+        <module>external/flux</module>
+        <module>external/storm-elasticsearch</module>
+        <module>external/storm-solr</module>
+        <module>examples/storm-starter</module>
+    </modules>
+
+
     <profiles>
-        <profile>
-            <id>storm-core</id>
-            <activation>
-                <activeByDefault>true</activeByDefault>
-            </activation>
-            <modules>
-                <module>storm-multilang/javascript</module>
-                <module>storm-multilang/python</module>
-                <module>storm-multilang/ruby</module>
-                <module>storm-buildtools/maven-shade-clojure-transformer</module>
-                <module>storm-buildtools/storm-maven-plugins</module>
-                <module>storm-core</module>
-            </modules>
-        </profile>
-        <profile>
-            <id>storm-more</id>
-            <activation>
-                <activeByDefault>true</activeByDefault>
-            </activation>
-            <modules>
-                <module>external/storm-kafka</module>
-                <module>external/storm-hdfs</module>
-                <module>external/storm-hbase</module>
-                <module>external/storm-hive</module>
-                <module>external/storm-jdbc</module>
-                <module>external/storm-redis</module>
-                <module>external/storm-eventhubs</module>
-                <module>external/flux</module>
-                <module>external/storm-elasticsearch</module>
-                <module>external/storm-solr</module>
-                <module>examples/storm-starter</module>
-            </modules>
-        </profile>
         <profile>
             <id>sign</id>
             <build>