Update to Spring Boot 3, new h2, use graalvm, GH actions (#816)

* Handle database recreation errors better * Handle different db usernames * Improve shutdown * Don't run startup problem detection in main thread * Fix stats calculation with h2 2 * Fix unit tests * Disable some tests not yet fixed (if ever) * Use SCRIPT TO to backup database as it verifies integrity * Only delete backups if a newer successful one exists * Fix error while analysing log file for OOM errors * Fix CSRF * Log Spring stuff on INFO * Add features for v5 to changelog * Don't try to remove windows tray icon cause it results in a hang * Fix notification history pagination * Fix shutdown behavior, prevent misleading error messages * Remove javax.annotation and .activation * Fix baseConfig.yml * First step to native build Compile runs, executable can be started, but has errors. More work needed. * First step to native build RUns under windows, HtmlUnit error, great progress! * Instantiate indexers and downloaders directly to make it work with native BeanFactory.createBean and .autowireBean do not work * Disable javascript in OpenPortChecker to fix charset error in native * Test Github Actions * Test Github Actions * Test Github Actions * Test Github Actions * Adapt wrappers to support native images * Update wrapper build to use docker containers clean folder structure Use v2 binaries for native support * Add build scripts, remove circleCi * Native build github actions * Don't write yaml, use in-memory db in native build mode * Upload native image artifact * Test self-hosted runner * Build on all three OSes * Build on all three OSes * Build on all three OSes * Build macos * Build windows * Build windows and macos * user proper paths for uploading artifacts * user proper paths for uploading artifacts * user proper paths for uploading artifacts * All three native images are built and uploaded But linux native not runnable under WSL, trying ubuntu 20.04 * Build static binary for linux * Make wrapper and update manager support both 4.x and 5.x, native and generic * Try building static image again * Commit wrapper binaries to LFS * Move files back to include folders * Adapt build-and-release.cmd to shell script * Prepare release via actions, add generic release * Prepare release via actions, add generic release * Update other package to 3.0.0 * Delete old docker files * Support dry run in release plugin and github action * Always use github token in release if available * Fix PhantomJS compile error, disable actions on push * release needs build * Show file structure on runner * Build on self-hosted windows runner * Disable graalvm watchdog so hopefully the windows build completes * Build on push, update dependencies * Build on windows * Determine release type from existance of files instead of json * Check version of executables before releasing * Scripts for test docker containers * Show system tray via wrapper, make main process headless * Native hints * Migrate JUnit 4 to 5 * Fix auth error when no internal api key was provided * Move integration tests * First step to system tests * Run system tests on github actions * Run system tests on github actions * Delete tests using old mockwebserver, use mockserver in system test * Delete tests using old mockwebserver, use mockserver in system test * Disable native image build to get to working systemtest faster * Specifically download coreLinux * Upload and download docker images * Save images in separate step * Fix docker port mapping syntax, keep docker images for one day only * Fix docker daemon syntax * Check out repo before running tests * Move config validation to separate classes * Start moving config classes to separate module * Finish moving config classes to separate module * Initialize test instance with sensible data * Push and use docker images using ghcr * Use custom health check for hydra docker container * Disable CSRF, use debug level for log file * Tests for debug infos * Use ${{ github.workspace }} * Use temp folder for hydradocker data * Write simple downloader web tests * Use /tmp as data folder * Test download of zipped NZBs * Build shared module before running tests * Load joptsimple resource bundle * Improve error page * Log spring security on debug level * Build new native image * Use new image * USe new reflect-config.json. Do I have to do this every time? * Reenable native build * Add reflection marker for native hints * Ensure all relevant classes have a reflection marker * Call TMDB api directly instead of using library * Use correct URL in mocked newznab results when in pipeline * Fix session error in IndexerUniquenessScoreSaver * Clean up code after migration to Java 17 * Backup tests * Remove spring profile from core dockerfile so that it can hopefully be loaded from env * GOOD STATE: Build native before doing tests Image is built, Two tests fail, but most succeed, yay * Test configuration of external tools * Run unit tests, upload test reports * Fix test being OS dependent (oops) * Fix test being OS dependent (oops) * Fix another OS dependent test, try other test report action * Create system test report * Try controlling native build by using "skipnativebuild" in the commit message * Controlling native build by using "skipnativebuild" in the commit message works, yay * Prepare entity TOs * Merge shared config and mapping modules * Merge shared entities and mapping modules * First simple test for history * Clean install for native image build * Commit missing files * More tests for search history * Disable repo.spring.io because central repos should be preferred * Test download history * Remove repo.spring.io repo * Upload settings.xml for debugging * Try custom settings.xml to prevent using repo.spring.io skipnativebuild * Remove settings.xml stuff, try running *arr dockers directly skipnativebuild * Fix sonar host * Don't skip prepareArtifacts as other jobs depend on it * Run all containers manually skipnativebuild * Set spring profiles in test env again skipnativebuild * SEMI_GOOD_STEP USe custom docker network in system test Communication between containers works, still bytecode error and BackupData#getBackupFile not working skipnativebuild * Remove lazy loading in SearchResultEntity, use TransactionTemplate in IndexerUniquenessScoreSaver.java * Remove lazy loading in IndexerApiAccessEntity.java * Make entities final to perhaps prevent lazy loading? * Remove ToStringSerializer for entity IDs * Support TVMAZE ID in search history * Remove names from docker run commands * Try using docker host network * Try systemtest network again skipnativebuild * Print build version and timestamp at startup * GOOD_STEP Run mockserver first so core can access it Only two failing tests remain * Print version and timestamp even if in docker * Use mapped volume for black hole * YESSSS GREEN PIPELINE Use a category for *arr which is supported by a configured indexer * Add stats test, is kinda flakey * Remove integration tests * Install java in docker container for db migration * First step to migration test * Remove migration module from mvn call * Generate custom assertions for mapping module * Add caps check test * Add media info test * Add news test * Add notifications test * Reenable migration test, perhaps it works now? * Add missing assertion files * Try different docker sock, add unmarshaller resource bundle * Don't trim stacktraces in maven, log unparsable indexer output * Use hopefully correct migration tmp folder, create it before using * Use different md to html renderer * Make caps inherit XML * Fix news test, create /tmp/data/v1MigrationDataFolder in action * Try different temp folder skipnativebuild * Print version earlier * Run migration docker directly * Print version differently * Wait for healthy containers * Print version and timestamp from properties * Wait for healthy containers * Use latest docker image when skipping nativebuild skipnativebuild * Give containers a bit of time skipnativebuild * Use docker-compose skipnativebuild * Try java from path if executable not found * Try to get more current artifact * ALL GREEN YAAAAAAY Build image in first job if configured, then just use latest in system test * Precheck that wrapper executables are newer than source files * Run all for core and migration, respectively * Remove discord bot, publish via maven plugin when releasing * Use container name in external hydra url * Move database migration to spring component * Use SLF4J in system tests skipnativebuild * Use 5076 on container skipnativebuild * ONE LEFT Wait a bit between file mod date changes Only blackhole test not working anymore * Use name of test component (core, v1Migration) in black hole folder * Log properties and docker mounts skipnativebuild * ALL GREEN WITH two test runs Mount blackhole folder for core skipnativebuild * Check changes files to decide if native image is built * Check changes files to decide if mock server is built * Send build decision messages to notice * Use changelog.yaml instead of .json * Fix startup of wrapper without internal api key * Use new wrapperHashes2.json * Write messages for discord and wiki * Ensure database integrity before creating backup * Show warning to user if automatic backup has failed * Show better error message when user triggered backup creation failed * Shoe proper error message when debug infos creation / download failed * Update wrapper hashes * Add loadtest, cache OKHTTP clients * Improve performance of search result parsing by getting rid of regex * Compress native image binary * Update release script * Update release script * Check for environment variables in release script * Check for untracked files or uncommitted changes in release script * git ignore token files * Read tokens from files * Color output for release script * Remove call from release script (only needed on windows) * Call maven in batch mode * Build shared module first * Change compiled modules in release script * Ask me to run mvn in windows * Try building on self hosted runner again * Run on all self-hosted * Run on all self-hosted * Run on all self-hosted * Run on all self-hosted * Translate build script to powershell * Make maven quiet * Escape -d options in maven * Check exit code differently * Ignore afile.txt * Escape -D * Allow "wet run" without release * Add missing changelog.yaml * Prepare for release * Don't build shared in buildCore.cmd * Wait for linux version after check of others * Disable commit for test * Build core from main folder * Copy windows build artifacts to include folder * Make wrapper open browser if main process can't * Delete release shell script * Fix build in system-test.yml * Update changelog * Update some libraries, ignore snakeyaml exploit in snyk * Remove system test data * Update to Spring Boot 3.0.2 * Differentiate versions of spring boot and devtools * Set githubReleasesUrl directly
theotherp · Jan 20, 2023 · 279b119 · 279b119
1 parent dc1cc46
commit 279b119
Show file tree

Hide file tree

Showing 1,016 changed files with 102,202 additions and 12,004 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
diff --git a/.github/workflows/buildNative.yml b/.github/workflows/buildNative.yml
@@ -0,0 +1,112 @@
+name: Native Build
+
+on:
+    workflow_dispatch:
+    workflow_call:
+
+jobs:
+    build:
+        strategy:
+            matrix:
+                os: [ ubuntu-20.04 ]
+            fail-fast: false
+        env:
+            HYDRA_NATIVE_BUILD: true
+        runs-on: ${{ matrix.os }}
+        steps:
+            -   uses: actions/checkout@v3
+                with:
+                    # Check out last 15 commits
+                    fetch-depth: 15
+
+            -   name: Set up JDK 17
+                uses: actions/setup-java@v3
+                with:
+                    java-version: '17'
+                    distribution: 'adopt'
+                    cache: 'maven'
+
+            -   name: "Get changed files in core module"
+                id: changed-files-specific
+                uses: tj-actions/changed-files@v35
+                with:
+                    since_last_remote_commit: true
+                    files: |
+                        core/**
+
+            -   if: steps.changed-files-specific.outputs.any_changed == 'true'
+                run: |
+                    echo "::notice::Will build new native image / docker container. Changed files in core since last push:"
+                    echo "${{ steps.changed-files-specific.outputs.all_changed_files }}"
+
+            -   if: steps.changed-files-specific.outputs.any_changed == 'false'
+                run: |
+                    echo "::notice::Will skip build of native image / docker container. No changed files in core since last push."
+
+            -   name: NativeImage
+                uses: graalvm/setup-graalvm@v1
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                with:
+                    java-version: '17'
+                    version: 'latest'
+                    components: 'native-image'
+                    cache: 'maven'
+                    github-token: ${{ secrets.GITHUB_TOKEN }}
+
+            -   name: "Install all with Maven"
+
+                run: mvn --batch-mode clean install -DskipTests -T 1C
+            -   name: "Run unit tests"
+                run: mvn --batch-mode test -T 1C -pl !org.nzbhydra:tests,!org.nzbhydra.tests:system --fail-at-end
+
+            -   name: "Create test Report"
+                uses: dorny/test-reporter@v1
+                if: always()
+                continue-on-error: true
+                with:
+                    name: Unit test report
+                    path: "**/surefire-reports/*.xml"
+                    reporter: java-junit
+
+            -   name: "Build native image"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                working-directory: ./core
+                run: |
+                    mvn --batch-mode -Pnative clean native:compile -DskipTests
+
+            -   name: "UPX linux artifact"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                uses: crazy-max/ghaction-upx@v2
+                with:
+                    files: core/target/core
+                    args: -q
+
+            -   name: "Upload linux artifact"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                uses: actions/upload-artifact@v3
+                with:
+                    name: coreLinux
+                    path: core/target/core
+
+            -   name: "Copy artifact to include folder"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                run: |
+                    mv core/target/core ./docker/nativeTest/
+                    chmod +x ./docker/nativeTest/core
+
+            -   name: "Login to GitHub Container Registry"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                uses: docker/login-action@v2
+                with:
+                    registry: ghcr.io
+                    username: ${{ github.actor }}
+                    password: ${{ secrets.GITHUB_TOKEN }}
+
+            -   name: "Build core and push container"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                run: |
+                    cp other/wrapper/nzbhydra2wrapperPy3.py ./docker/nativeTest/
+                    cd ./docker/nativeTest/
+                    docker build -t hydradocker .
+                    docker tag hydradocker:latest ghcr.io/theotherp/hydradocker:latest
+                    docker push ghcr.io/theotherp/hydradocker:latest
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -0,0 +1,57 @@
+name: Release
+
+on:
+    workflow_dispatch:
+        inputs:
+            releaseVersion:
+                required: true
+                type: string
+                description: "Version to be released, like 1.2.3"
+            nextVersion:
+                required: true
+                type: string
+                description: "Version to be set afterwards, like 1.2.4-SNAPSHOT"
+            dryRun:
+                required: true
+                default: true
+                type: boolean
+                description: "Uncheck to actually execute the release"
+            selfHostedRunner:
+                required: true
+                default: false
+                type: boolean
+                description: "Has no effect, just as a reminder that the self-hosted windows runner must be running"
+
+jobs:
+    build:
+        uses: ./.github/workflows/buildNative.yml
+    release:
+        needs: [build]
+        runs-on: ubuntu-latest
+        env:
+            githubReleasesUrl: https://api.github.com/repos/{{github.repository}}/releases
+        steps:
+            -   uses: actions/checkout@v3
+                name: "Check out source"
+            -   name: "Display structure of working directory"
+                run: ls .
+            -   uses: actions/download-artifact@v3
+                name: "Download native artifacts"
+                with:
+                    path: ~/artifacts
+            -   name: "Display structure of artifacts folder"
+                run: ls -R ~/artifacts
+            -   name: "Copy artifacts to include folders"
+                run: |
+                    mv ~/artifacts/coreLinux/* ./releases/linux-release/include/
+                    chmod +x ./releases/linux-release/include/nzbhydra2
+                    mv ~/artifacts/coreWindows/* ./releases/windows-release/include/
+            -   name: Set up JDK 17
+                uses: actions/setup-java@v3
+                with:
+                    java-version: '17'
+                    distribution: 'adopt'
+                    cache: 'maven'
+            -   name: "Run release script"
+                run: |
+                    misc/build-and-release.sh ${{ github.event.inputs.releaseVersion }} ${{ github.event.inputs.nextVersion }} ${{ github.event.inputs.dryRun }}
diff --git a/.github/workflows/system-test.yml b/.github/workflows/system-test.yml
@@ -0,0 +1,130 @@
+name: system-test
+
+on:
+    push:
+    workflow_dispatch:
+
+jobs:
+    waitForNative:
+        uses: ./.github/workflows/buildNative.yml
+    buildMockserver:
+        runs-on: ubuntu-latest
+        steps:
+            -   uses: actions/checkout@v3
+                name: "Check out source"
+                with:
+                    # Check out last 15 commits
+                    fetch-depth: 15
+
+            -   name: "Get changed files in core module"
+                id: changed-files-specific
+                uses: tj-actions/changed-files@v35
+                with:
+                    since_last_remote_commit: true
+                    files: |
+                        other/mockserver/**
+
+            -   if: steps.changed-files-specific.outputs.any_changed == 'true'
+                run: |
+                    echo "::notice::Will build new mock server container. Changed files since last push:"
+                    echo "${{ steps.changed-files-specific.outputs.all_changed_files }}"
+
+            -   if: steps.changed-files-specific.outputs.any_changed == 'false'
+                run: |
+                    echo "::notice::Will skip build of new mock server container. No changed files since last push."
+
+            -   name: Set up JDK 17
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                uses: actions/setup-java@v3
+                with:
+                    java-version: '17'
+                    distribution: 'adopt'
+                    cache: 'maven'
+
+            -   name: "Login to GitHub Container Registry"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                uses: docker/login-action@v2
+                with:
+                    registry: ghcr.io
+                    username: ${{ github.actor }}
+                    password: ${{ secrets.GITHUB_TOKEN }}
+
+            -   name: "Build and push mockserver container"
+                if: steps.changed-files-specific.outputs.any_changed == 'true'
+                run: |
+                    mvn --batch-mode install -DskipTests -T 1C
+                    cd other/mockserver/
+                    mvn --batch-mode spring-boot:build-image
+                    docker tag mockserver:3.0.0 ghcr.io/theotherp/mockserver:3.0.0
+                    docker push ghcr.io/theotherp/mockserver:3.0.0
+
+    runSystemTests:
+        needs: [ waitForNative, buildMockserver ]
+        runs-on: ubuntu-latest
+        strategy:
+            matrix:
+                test: [ { port: 5076, name: core }, { port: 5077, name: v1Migration } ]
+        env:
+            spring_profiles_active: build,systemtest
+            nzbhydra_port: ${{ matrix.test.port }}
+            nzbhydra_name: ${{ matrix.test.name }}
+            nzbhydra_host_external: http://${{ matrix.test.name }}:5076
+            nzbhydra.host.external: http://${{ matrix.test.name }}:5076
+        steps:
+            -   run: echo Running test ${{ matrix.test.name }} with port ${{ matrix.test.port }}
+            -   uses: actions/checkout@v3
+                name: "Check out source"
+
+            -   name: Set up JDK 17
+                uses: actions/setup-java@v3
+                with:
+                    java-version: '17'
+                    distribution: 'adopt'
+                    cache: 'maven'
+
+            -   name: "Install"
+                run: mvn --batch-mode clean install -DskipTests -pl org.nzbhydra:nzbhydra2,org.nzbhydra:shared,org.nzbhydra:mapping,org.nzbhydra:assertions
+
+            -   name: "Create docker network"
+                run: docker network create systemtest
+
+            -   name: "Copy v1Migration docker data"
+                run: |
+                    mkdir -p /tmp/hydra/v1MigrationDataFolder
+                    cp -R tests/system/instanceData/v1Migration/* /tmp/hydra/v1MigrationDataFolder/
+
+            -   name: "Run docker compose"
+                run: |
+                    cd docker
+                    docker-compose up -d
+
+            -   name: "Wait for healthy containers"
+                run: |
+                    docker ps
+                    sleep 10
+                    docker ps
+                    sleep 10
+                    docker ps
+                    echo "Core container mounts:"
+                    docker container inspect -f '{{ .Mounts}}' core
+                    echo "v1Migration container mounts:"
+                    docker container inspect -f '{{ .Mounts}}' v1Migration
+
+            -   name: "Run tests"
+                run: mvn --batch-mode test -pl org.nzbhydra.tests:system -DtrimStackTrace=false
+
+            -   name: "Upload data folder artifact"
+                uses: actions/upload-artifact@v3
+                if: always()
+                with:
+                    name: data
+                    path: /tmp/hydra
+
+            -   name: "Create test Report"
+                uses: dorny/test-reporter@v1
+                if: always()
+                continue-on-error: true
+                with:
+                    name: System test report ${{ matrix.test.name }}
+                    path: "**/surefire-reports/*.xml"
+                    reporter: java-junit
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -0,0 +1,19 @@
+name: Java CI
+
+on: [ workflow_dispatch ]
+
+
+jobs:
+    build:
+        runs-on: ubuntu-latest
+
+        steps:
+            -   uses: actions/checkout@v3
+            -   name: Set up JDK 17
+                uses: actions/setup-java@v3
+                with:
+                    java-version: '17'
+                    distribution: 'adopt'
+                    cache: 'maven'
+            -   name: Test with Maven
+                run: mvn --batch-mode --update-snapshots verify
diff --git a/.gitignore b/.gitignore
@@ -19,9 +19,11 @@ pom.xml.versionsBackup
 javacore*
 heapdump*
 /ui
+*token*
 
 !/.idea/compiler.xml
 !/.idea/vcs.xml
 !/.idea/misc.xml
 misc/rsyncToServers.sh
 /nzbhydra.yml
+/results