Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run integration test with a given Java version and vendor #1749

Merged
merged 13 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
387 changes: 288 additions & 99 deletions .github/workflows/ci.yaml

Large diffs are not rendered by default.

8 changes: 3 additions & 5 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ subprojects {
// apply plugin: 'jacoco'
apply plugin: 'java-library-distribution'
apply plugin: 'com.diffplug.spotless'
apply plugin: 'com.scalar.db.jdk-configuration'

project.version = '4.0.0-SNAPSHOT'

Expand Down Expand Up @@ -54,10 +55,9 @@ subprojects {
log4jVersion = '2.23.1'
stefanbirknerSystemLambdaVersion = '1.2.1'
spotbugsPluginVersion = '5.2.5'
errorpronePluginVersion = '2.0.2'
errorpronePluginVersion = '3.1.0'
protobufPluginVersion = '0.8.19'
shadowPluginVersion = '7.1.2'
dockerPluginVersion = '0.34.0'
// Make JDK11+ use the same version as JDK8 uses
googleJavaFormatVersion = '1.7'
}
Expand All @@ -71,9 +71,6 @@ subprojects {
testLogging.showStandardStreams = true
}

sourceCompatibility = 1.8
targetCompatibility = 1.8
Comment on lines -74 to -75
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't have this anymore since the com.scalar.db.jdk-configuration plugin sets the JDK for all Java tasks and we need it to be configurable.


group = "com.scalar-labs"

java {
Expand All @@ -99,3 +96,4 @@ subprojects {
duplicatesStrategy DuplicatesStrategy.EXCLUDE
}
}

26 changes: 26 additions & 0 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
plugins {
id 'java-gradle-plugin'
id 'com.diffplug.spotless' version '6.13.0'
}

repositories {
mavenCentral()
}

gradlePlugin {
plugins {
jdkConfiguration {
id = 'com.scalar.db.jdk-configuration'
implementationClass = 'com.scalar.db.JdkConfigurationPlugin'
}
}
}

spotless {
java {
target 'src/*/java/**/*.java'
importOrder()
removeUnusedImports()
googleJavaFormat("1.7")
}
}
152 changes: 152 additions & 0 deletions buildSrc/src/main/java/com/scalar/db/JdkConfigurationPlugin.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
package com.scalar.db;

import java.util.Objects;
import java.util.function.Consumer;
import javax.annotation.Nullable;
import org.gradle.api.Plugin;
import org.gradle.api.Project;
import org.gradle.api.plugins.JavaPluginExtension;
import org.gradle.api.tasks.testing.Test;
import org.gradle.jvm.toolchain.JavaLanguageVersion;
import org.gradle.jvm.toolchain.JavaToolchainService;
import org.gradle.jvm.toolchain.JvmVendorSpec;
import org.jetbrains.annotations.NotNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* Plugin configuring all tasks and integration test tasks to use a given JDK defined by the
* following Gradle properties:
*
* <ul>
* <li>`javaVersion` : configure all Java related tasks to use the given JDK version. The default
* java version 8.
* <li>`javaVendor` : configure all Java related tasks to use the given JDK vendor. The default
* java vendor is Adoptium (also known as Temurin).
* <li>`integrationTestJavaRuntimeVersion` : configure all test tasks name starting with
* "integrationTest" to run with the given JDK version. The default Java runtime version is 8.
* <li>`integrationTestJavaRuntimeVendor` : configure all test tasks name starting with
* "integrationTest" to run with the given JDK vendor. The default Java runtime vendor is
* Adoptium (also known as Temurin).
* </ul>
*
* <p>Usage example using the CLI:
*
* <p>1. To use JDK 11 (amazon) for all Java tasks including integration tests
*
* <pre><code>
* gradle integrationTestJdbc -PjavaVersion=11 -PjavaVendor=amazon
* </code></pre>
*
* 2. To use JDK 11 (amazon) for all Java tasks while having integration test use JDK 17 (microsoft)
*
* <pre><code>
* gradle integrationTestJdbc -PjavaVersion=11 -PjavaVendor=amazon -PintegrationTestJavaRuntimeVersion=17 -PintegrationTestJavaRuntimeVendor=microsoft
* </code></pre>
*/
public class JdkConfigurationPlugin implements Plugin<Project> {
Copy link
Contributor Author

@Torch3333 Torch3333 May 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a Gradle plugin used to configure the compilation and integration test tasks to use a given SDK configured through Gradle properties.

Instead of writing a plugin, I could have added logic directly in build.gradle files but it was easier to develop that way and our build.gradle files are already complicated enough.

Comment on lines +17 to +47
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a way to configure the SDK used for all Java tasks (compilation, unit tests, integration tests, etc...).

gradle integrationTestJdbc -PjavaVersion=11 -PjavaVendor=amazon

This differs from what we decided earlier because if we could only configure the compiler JDK, we could run into an issue where the test tasks fail to run because it uses a JDK version anterior to the compiler JDK version. So, for simplicity, we configure all Java tasks to use a common JDK, but we can still set a specific JDK to run the integration test with:

gradle integrationTestJdbc -PintegrationTestJavaRuntimeVersion=17 -PintegrationTestJavaRuntimeVendor=microsoft


private static final Logger logger = LoggerFactory.getLogger(JdkConfigurationPlugin.class);
// JDK 8 (temurin) is used as default for all tasks (compilation, tests, etc.)
private static final JavaLanguageVersion DEFAULT_JAVA_VERSION = JavaLanguageVersion.of(8);
private static final JvmVendorSpec DEFAULT_JAVA_VENDOR = JvmVendorSpec.ADOPTIUM;

private static final String JAVA_VERSION_PROP = "javaVersion";
private static final String JAVA_VENDOR_PROP = "javaVendor";
private static final String INTEGRATION_TEST_JAVA_RUNTIME_VERSION_PROP =
"integrationTestJavaRuntimeVersion";
private static final String INTEGRATION_TEST_JAVA_RUNTIME_VENDOR_PROP =
"integrationTestJavaRuntimeVendor";

private JavaLanguageVersion javaVersion = DEFAULT_JAVA_VERSION;
private JvmVendorSpec javaVendor = DEFAULT_JAVA_VENDOR;
@Nullable private JavaLanguageVersion integrationTestJavaVersion;
@Nullable private JvmVendorSpec integrationTestJavaVendor;

@Override
public void apply(@NotNull Project project) {
parseIntegrationTestInputProperties(project);
configureJdkForAllJavaTasks(project);
configureJdkForIntegrationTestTasks(project);
}

private void configureJdkForAllJavaTasks(Project project) {
JavaPluginExtension javaPlugin = project.getExtensions().getByType(JavaPluginExtension.class);
javaPlugin.getToolchain().getLanguageVersion().set(javaVersion);
javaPlugin.getToolchain().getVendor().set(javaVendor);
logger.debug("Configure JDK {} ({}) for Java tasks", javaVersion, javaVendor);
}

private void configureJdkForIntegrationTestTasks(Project project) {
if (integrationTestJavaVersion == null) {
return;
}
project
.getTasks()
.withType(Test.class)
.matching(testTask -> testTask.getName().startsWith("integrationTest"))
.configureEach(
integrationTestTask ->
integrationTestTask
.getJavaLauncher()
.set(
getJavaToolchainService(project)
.launcherFor(
config -> {
config.getLanguageVersion().set(integrationTestJavaVersion);
logger.debug(
"Configure task '{}' to use JDK version {}",
integrationTestTask.getName(),
integrationTestJavaVersion);
if (integrationTestJavaVendor != null) {
config.getVendor().set(integrationTestJavaVendor);
logger.debug(
"Configure task '{}' to use {} JDK vendor",
integrationTestTask.getName(),
integrationTestJavaVendor);
}
})));
}

private void parseIntegrationTestInputProperties(Project project) {
parseVersionInputProperty(project, JAVA_VERSION_PROP, (version) -> javaVersion = version);
parseVendorInputProperty(project, JAVA_VENDOR_PROP, (vendor) -> javaVendor = vendor);
parseVersionInputProperty(
project,
INTEGRATION_TEST_JAVA_RUNTIME_VERSION_PROP,
(version) -> integrationTestJavaVersion = version);
parseVendorInputProperty(
project,
INTEGRATION_TEST_JAVA_RUNTIME_VENDOR_PROP,
(vendor) -> integrationTestJavaVendor = vendor);
}

private void parseVersionInputProperty(
Project project, String property, Consumer<JavaLanguageVersion> attributeSetter) {
if (project.hasProperty(property)) {
attributeSetter.accept(JavaLanguageVersion.of(Objects.toString(project.property(property))));
}
}

private void parseVendorInputProperty(
Project project, String property, Consumer<JvmVendorSpec> attributeSetter) {
if (!project.getRootProject().hasProperty(property)) {
return;
}
String propertyValue = Objects.toString(project.property(property));
switch (propertyValue) {
case "corretto":
attributeSetter.accept(JvmVendorSpec.AMAZON);
break;
case "temurin":
attributeSetter.accept(JvmVendorSpec.ADOPTIUM);
break;
default:
attributeSetter.accept(JvmVendorSpec.matching(propertyValue));
}
}

private @NotNull JavaToolchainService getJavaToolchainService(Project project) {
return project.getExtensions().getByType(JavaToolchainService.class);
}
}
16 changes: 9 additions & 7 deletions core/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ plugins {
id 'net.ltgt.errorprone' version "${errorpronePluginVersion}"
id 'maven-publish'
id 'signing'
id 'base'
}

sourceSets {
Expand Down Expand Up @@ -128,15 +129,14 @@ javadoc {
title = "ScalarDB"
}


task integrationTestCassandra(type: Test) {
description = 'Runs the integration tests for Cassandra.'
group = 'verification'
testClassesDirs = sourceSets.integrationTestCassandra.output.classesDirs
classpath = sourceSets.integrationTestCassandra.runtimeClasspath
outputs.upToDateWhen { false } // ensures integration tests are run every time when called
options {
systemProperties(System.getProperties())
systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When running the integration test with Java 9 but using Java 8 to run IDEA, some Java 8 environment properties were unwillingly added by IDEA that caused the test to crash, so I restricted only to passing properties starting with scalardb

}
}

Expand All @@ -147,7 +147,7 @@ task integrationTestCosmos(type: Test) {
classpath = sourceSets.integrationTestCosmos.runtimeClasspath
outputs.upToDateWhen { false } // ensures integration tests are run every time when called
options {
systemProperties(System.getProperties())
systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
}
maxParallelForks = 3
jvmArgs '-XX:MaxDirectMemorySize=4g', '-Xmx4g'
Expand All @@ -160,7 +160,7 @@ task integrationTestDynamo(type: Test) {
classpath = sourceSets.integrationTestDynamo.runtimeClasspath
outputs.upToDateWhen { false } // ensures integration tests are run every time when called
options {
systemProperties(System.getProperties())
systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
}
maxParallelForks = 10
}
Expand All @@ -172,7 +172,7 @@ task integrationTestJdbc(type: Test) {
classpath = sourceSets.integrationTestJdbc.runtimeClasspath
outputs.upToDateWhen { false } // ensures integration tests are run every time when called
options {
systemProperties(System.getProperties())
systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
}
}

Expand All @@ -183,7 +183,7 @@ task integrationTestMultiStorage(type: Test) {
classpath = sourceSets.integrationTestMultiStorage.runtimeClasspath
outputs.upToDateWhen { false } // ensures integration tests are run every time when called
options {
systemProperties(System.getProperties())
systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
}
}

Expand Down Expand Up @@ -229,7 +229,9 @@ check.dependsOn += spotbugsMain
check.dependsOn += spotbugsTest
check.dependsOn += spotbugsIntegrationTest

archivesBaseName = "scalardb"
base {
archivesName = "scalardb"
}

// for archiving and uploading to maven central
publishing {
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ repositories {
mavenCentral()
}

mainClassName = "sample.ElectronicMoneyMain"
mainClass = "sample.ElectronicMoneyMain"

dependencies {
implementation 'com.scalar-labs:scalardb:3.12.2'
Expand Down
Binary file modified gradle/wrapper/gradle-wrapper.jar
Binary file not shown.
2 changes: 1 addition & 1 deletion gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.0.2-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-8.7-bin.zip
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upgraded Gradlew to the latest stable version 8.7

zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
2 changes: 2 additions & 0 deletions gradlew
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ esac

CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar


# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
Expand Down Expand Up @@ -129,6 +130,7 @@ fi
if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
APP_HOME=`cygpath --path --mixed "$APP_HOME"`
CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`

JAVACMD=`cygpath --unix "$JAVACMD"`

# We build the pattern for arguments to be converted via cygpath
Expand Down
25 changes: 7 additions & 18 deletions gradlew.bat
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ if "%DIRNAME%" == "" set DIRNAME=.
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%

@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi

@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"

Expand All @@ -37,7 +40,7 @@ if defined JAVA_HOME goto findJavaFromJavaHome

set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if "%ERRORLEVEL%" == "0" goto init
if "%ERRORLEVEL%" == "0" goto execute

echo.
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Expand All @@ -51,7 +54,7 @@ goto fail
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe

if exist "%JAVA_EXE%" goto init
if exist "%JAVA_EXE%" goto execute

echo.
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
Expand All @@ -61,28 +64,14 @@ echo location of your Java installation.

goto fail

:init
@rem Get command-line arguments, handling Windows variants

if not "%OS%" == "Windows_NT" goto win9xME_args

:win9xME_args
@rem Slurp the command line arguments.
set CMD_LINE_ARGS=
set _SKIP=2

:win9xME_args_slurp
if "x%~1" == "x" goto execute

set CMD_LINE_ARGS=%*

:execute
@rem Setup the command line

set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar


@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*

:end
@rem End local scope for the variables with windows NT shell
Expand Down
5 changes: 4 additions & 1 deletion integration-test/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ plugins {
id 'net.ltgt.errorprone' version "${errorpronePluginVersion}"
id 'maven-publish'
id 'signing'
id 'base'
}

dependencies {
Expand Down Expand Up @@ -52,7 +53,9 @@ spotbugsTest.reports {
}
spotbugsTest.excludeFilter = file("${project.rootDir}/gradle/spotbugs-exclude.xml")

archivesBaseName = "scalardb-integration-test"
base {
archivesName = "scalardb-integration-test"
}

// for archiving and uploading to maven central
publishing {
Expand Down
Loading
Loading