GDAL 3 Environment variables not being set

[dcdenu4]

Describe the bug

Using setup-conda to install gdal 3+ on Ubuntu 18.04 and noticed GDAL_DATA and PROJ_LIB environment variables are not being set. This was noticed by getting the following error in one of our tests for a Python project when trying to use the gdal library:
ERROR 1: PROJ: proj_create_from_database: Open of /usr/share/miniconda/share/proj failed

When setting GDAL_DATA and PROJ_LIB environment variables in a yml workflow manually, calling printenv will show the paths to those variables, however, from the test in Python, printing os.environ shows that these variables are still not present or set (see screeshots).

To Reproduce
Steps to reproduce the behavior:

1. Create a Github Actions workflow with OS as Ubuntu 18.04
2. Use setup-conda and install gdal3+ from a requirements file
3. Run a Python file that imports gdal and osr to load a EPSG Code
4. See error

From dcdenu4/gdal3-conda-setup, .github/workflow/pythonpackage.yml :

name: Test PyGeoprocessing

on: [push, pull_request]

jobs:
  Test:
    runs-on: ${{ matrix.os }}
    env:
        PACKAGES: "pytest flake8"
    strategy:
      fail-fast: false
      max-parallel: 4
      matrix:
        python-version: [3.7]
        os: [ubuntu-18.04]

    steps:
    - uses: actions/checkout@v2
      with:
          # Fetch all history so that setuptool_scm can build the correct version string.
          fetch-depth: 0

    - name: setup-conda
      uses: s-weigand/[email protected]
      with:
        update-conda: false
        python-version: ${{ matrix.python-version }}
        activate-conda: true
        conda-channels: anaconda, conda-forge
    - name: Install dependencies
      shell: bash
      run: conda upgrade -y pip setuptools

    - name: Install GDAL3 Test (Linux)
      env: 
          GDAL_DATA: /usr/share/miniconda/share/gdal/
          PROJ_LIB: /usr/share/miniconda/share/proj/
      run: |
          conda install --file requirements.txt
          conda install $PACKAGES
          printenv
          find /usr -name proj.db 
          ls -l /usr/share/miniconda/share/proj/

    - name: Test with pytest
      run: pytest -s

Expected behavior
I would expect GDAL_DATA and PROJ_LIB to be set as environment variables and to be available to Python to run the GDAL Library.

Screenshots
Environment variables printed out using printenv after install gdal and manually setting environment variables. If not manually set, GDAL_DATA and PROJ_LIB do not appear here.

A Python test using os.environ to show environment variables. GDAL_DATA and PROJ_LIB not present and error occurs.

Desktop (please complete the following information):

• OS: Ubuntu 18.04 via Github Actions VM
• Browser [N / A]
• Version [N / A]

Additional context
I created a dummy repo here: github.com/dcdenu4/gdal-conda-setup to help demonstrate this issue. Ultimately this problem has arisen from trying to go from GDAL 2 to GDAL 3 in a larger project, where using GDAL 3 our tests fail using this kind of workflow.

[s-weigand]

I see that tests work now in your dummy repo.
So is this still an issue?
Also, I kinda miss the point how this is an issue with this gh-action.
Are the env vars set by conda, if you install GDAL locally?
Personally, I know similar problems, when installing graphviz with conda, wich isn't installed in the <conda>/bin folder (which is added to the PATH), but <conda>/bin/graphviz .
Maybe adding an additional option to add folders, relative to the conda root, to the PATH would be usefull in those cases.

Additionally:
export <ENV_VAR_NAME>=<ENV_VAR_VALUE>
Should set the env var for the whole workflow, not only a single step.

[dcdenu4]

@s-weigand sorry for the delay.

This is still an issue, however, using setup-miniconda seems to properly set the environment variables. The easiest way to see this is by looking at the two latest Actions for that dummy repository. The test is shown as passing, but in our actual code base when we transform across coordinate systems using GDAL, this error will cause our tests to fail. Sorry for the lack of clarity with the example test.

Using setup-miniconda. GDAL_DATA, PROJ_LIB ENVs are set.
https://github.com/dcdenu4/gdal3-conda-setup/runs/491287284?check_suite_focus=true

Using setup-conda. GDAL_DATA, PROJ_LIB are NOT set and error can be seen.
https://github.com/dcdenu4/gdal3-conda-setup/actions/runs/51021337

The only big difference I can see from the two approaches is that setup-miniconda seems to use an environment that is not the base environment. Maybe that activation and installing helps those environment variables get and stay set from the GDAL and PROJ libraries...

[s-weigand]

After some testing I guess I found the culpit.
When calling source activate base again, after installing the dependencies in the base env, the env vars get set properly.
Would you be so kinda on confirm this on your demo repo? 😄
This should be fixed when I implemented the option requirement-file and/or only-activate-env (see #6 (comment)).

The way setup-miniconda appears to manage this, is by adding the env activation to all possible shell config files.
https://github.com/goanpeca/setup-miniconda/blob/aa03dee7ae253136d4fae5fe8ba5c4d9ad25ae25/src/setup-conda.ts#L433

[dcdenu4]

@s-weigand, I can confirm that adding source activate base does indeed patch the issue. Both GDAL_DATA and PROJ_LIB are present.

[s-weigand]

@dcdenu4 Would you be so kind to try it again with the old config?
If this works now we could close this issue. 😄

[dcdenu4]

@s-weigand it does indeed work! Sorry for the lack of follow up.

[s-weigand]

@dcdenu4 Just to make sure, we both talk about the same.
My question was regarding, a workflow w/o the source activate hack.

[dcdenu4]

Yes, we have removed the source activate hack from our workflow and GDAL 3+ is building just fine!

[s-weigand]

Awesome 😄

[s-weigand]

@all-contributors please add @dcdenu4 for bug

[allcontributors]

@s-weigand

I've put up a pull request to add @dcdenu4! 🎉

[s-weigand]

@all-contributors please add @dcdenu4 for bug

[allcontributors]

@s-weigand

I've put up a pull request to add @dcdenu4! 🎉