Add new regression data download method and update github actions ubu…

…ntu version (#699)

* Added new regression data download method

* Updated ubuntu workflows to ubuntu-20.04

* Fiddling with directory structures

* Updated test-suite to cached poetry install

* Removed cached deps

* Attempt two: https://jacobian.org/til/github-actions-poetry/

* Added poetry run to all python calls

* More poetry fiddling

* More poetry fiddling

* Updated download method

* Remove python 3.10 support

* Updated test data url in docs

* Updated changelog

* Restrict python to <3.10

* Remove outdated line from download.sh

.github/workflows/lint.yml

@@ -4,7 +4,7 @@ on: [pull_request]
 
 jobs:
   lint:
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
 
     steps:
     - uses: actions/checkout@v2

.github/workflows/test-suite.yml

@@ -8,11 +8,11 @@ env:
 
 jobs:
   test:
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
     strategy:
       fail-fast: false
       matrix:
-        python_version: ["3.8.12", "3.9.10", "3.10.2"]
+        python_version: ["3.8.12", "3.9.10"]
 
     steps:
       - uses: actions/checkout@v2
@@ -30,16 +30,41 @@ jobs:
         with:
           python-version: ${{ matrix.python_version }}
 
+      - name: Install LLVM and Clang
+        uses: KyleMayes/install-llvm-action@v1
+        with:
+          version: "10.0"
+      - name: cache poetry install
+        uses: actions/cache@v2
+        with:
+          path: ~/.local
+          key: poetry-1.5.1-0
+
+      - uses: snok/install-poetry@v1
+        with:
+          version: 1.5.1
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+      - name: cache deps
+        id: cache-deps
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: pydeps-${{ hashFiles('**/poetry.lock') }}
+
+      - run: poetry install --no-interaction --no-root
+        if: steps.cache-deps.outputs.cache-hit != 'true'
+
+      - run: poetry install --no-interaction
+
       - uses: actions/setup-node@v2
         with:
           node-version: "12"
 
-      - name: Set up frontend assets and install python dependencies
+      - name: Set up frontend assets
         run: |
           npm ci
           npm start
-          pip install wheel
-          pip install .
 
       - name: Set DATABASE_URL (using ansible)
         run: >
@@ -54,7 +79,7 @@ jobs:
 
       - name: Create database user and database name
         run: >
-          python ./init-tools/init-db.py
+          poetry run python ./init-tools/init-db.py
           localhost ${{ env.db_port }}
           postgres ${{ env.db_password }}
           vast vastpsw vastdb
@@ -67,4 +92,4 @@ jobs:
         working-directory: ./vast_pipeline/tests/regression-data
 
       - name: Run tests
-        run: ./manage.py test
+        run: poetry run python manage.py test

CHANGELOG.md

@@ -41,6 +41,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Changed
 
+- Updated path to test data in github actions and docs [#699](https://github.com/askap-vast/vast-pipeline/pull/699)
+- Changed GitHub actions test suite to install pipeline via poetry [#699](https://github.com/askap-vast/vast-pipeline/pull/699).
+- Updated GitHub actions ubuntu version to 20.04 [#699](https://github.com/askap-vast/vast-pipeline/pull/699).
+- Removed python 3.10 testing from GitHub actions [#699](https://github.com/askap-vast/vast-pipeline/pull/699).
 - Updated GitHub actions Gr1N/setup-poetry to v7 [#665](https://github.com/askap-vast/vast-pipeline/pull/665).
 - Changed lightcurve plot cursor hit-test mode from "vline" to "mouse" to avoid a regression in Bokeh [#652](https://github.com/askap-vast/vast-pipeline/pull/652).
 - Updated Bokeh from v2.3.3 to v2.4.2 [#652](https://github.com/askap-vast/vast-pipeline/pull/652).
@@ -106,6 +110,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### List of PRs
 
+- [#699](https://github.com/askap-vast/vast-pipeline/pull/699): docs, feat: Add new regression data download URL and updates to Github Actions
 - [#697](https://github.com/askap-vast/vast-pipeline/pull/697/): feat: Added links to Data Central DAS and the Fink Broker to the source page 
 - [#685](https://github.com/askap-vast/vast-pipeline/pull/685): docs: Updated `runpipeline` section on CLI docs.
 - [#676](https://github.com/askap-vast/vast-pipeline/pull/676): Removed home counts and new source count.

docs/developing/tests.md

@@ -34,15 +34,15 @@ to run the tests located in [`test_webserver.py`](https://github.com/askap-vast/
 
 ## Regression Tests
 
-Regression tests located in [`test_regression`](https://github.com/askap-vast/vast-pipeline/blob/master/vast_pipeline/tests/test_regression/){:target="\_blank"} require the use of the _VAST\_2118-06A_ field test dataset which is not a part of the repository. This data is downloadable from [cloudstor](https://cloudstor.aarnet.edu.au/plus/s/xjh0aRr1EGY6Bt3){:target="_blank"}. You can use the script located in [tests/regression-data/](https://github.com/askap-vast/vast-pipeline/blob/master/vast_pipeline/tests/regression-data/){:target="_blank"}:
+Regression tests located in [`test_regression`](https://github.com/askap-vast/vast-pipeline/blob/master/vast_pipeline/tests/test_regression/){:target="\_blank"} require the use of the _VAST\_2118-06A_ field test dataset which is not a part of the repository. This data is downloadable from [the USyd machines](http://www.physics.usyd.edu.au/~ddob1600/vast-pipeline/pipeline-test-data.zip){:target="_blank"}. You can use the script located in [tests/regression-data/](https://github.com/askap-vast/vast-pipeline/blob/master/vast_pipeline/tests/regression-data/){:target="_blank"}:
 
 ```bash
 cd vast_pipeline/tests/regression-data/ && ./download.sh
 ```
 
 to download the _VAST\_2118-06A_ field test dataset into the [`regression-data`](https://github.com/askap-vast/vast-pipeline/blob/master/vast_pipeline/tests/regression-data/){:target="_blank"} folder. Or manually by clicking the button below:
 
-[Download data for test :material-cloud-download-outline:](https://cloudstor.aarnet.edu.au/plus/s/xjh0aRr1EGY6Bt3/download){: .md-button target="_blank"}
+[Download data for test :material-cloud-download-outline:](http://www.physics.usyd.edu.au/~ddob1600/vast-pipeline/pipeline-test-data.zip){: .md-button target="_blank"}
 
 and place the _VAST\_2118-06A_ field test dataset into the [`regression-data`](https://github.com/askap-vast/vast-pipeline/blob/master/vast_pipeline/tests/regression-data/){:target="_blank"} folder. These regression tests are skipped if the dataset is not present.
 

pyproject.toml

@@ -6,7 +6,7 @@ authors = ["The VAST Development Team"]
 license = "MIT"
 
 [tool.poetry.dependencies]
-python = ">=3.8.0,<3.11"
+python = ">=3.8.0,<3.10"
 astropy = "^5.0"
 astroquery = "^0.4.4"
 bokeh = "2.4.2" # must align with @bokeh/bokehjs version in package.json

vast_pipeline/tests/regression-data/download.sh

@@ -10,13 +10,13 @@ echo
 
 TMPFILE=`mktemp`
 TMPDIR=`mktemp -d`
-wget "https://cloudstor.aarnet.edu.au/plus/s/xjh0aRr1EGY6Bt3/download" -O ${TMPFILE}
+wget "http://www.physics.usyd.edu.au/~ddob1600/vast-pipeline/pipeline-test-data.zip" -O ${TMPFILE}
 
 echo "Unzipping archive to ${TMPDIR} ..."
 unzip -q -d ${TMPDIR} ${TMPFILE}
 
 echo "Moving data to relevant directory..."
-mv ${TMPDIR}/VAST_PIPELINE_TEST_DATASET/* $(pwd)/.
+mv ${TMPDIR}/pipeline-test-data/* $(pwd)/.
 
 echo "Deleting temporary files..."
 rm -r ${TMPFILE} ${TMPDIR}