Add Python MapScript 3.8 support

[geographika]

There are currently 2 blockers to adding Python 3.8 support (on Windows):

1. Python 3.8 uses sqlite version 3.28.0

Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.28.0'

The SDK from GISInternals uses sqlite-3.24.0
https://www.gisinternals.com/packageinfo.php?file=release-1911-x64-gdal-mapserver.zip

This causes the following error when attempting to import mapscript:

ImportError: DLL load failed while importing _mapscript: The specified procedure could not be found.

2. mapserver.dll and its dependencies (proj.dll, geos.dll, gdal300.dll etc.) are currently loaded by adding the /bin folder to the Windows PATH variable. From version 3.8 this is no longer allowed for security reasons. From https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api

Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library.

To fix this I propose adding a new MAPSERVER_HOME variable to the Python MapScript library and adding this folder to the allowed DLLs folders using os.add_dll_directory on startup. All MapServer DLLs should be copied to this location prior to running tests.

On the command line the variable can be set:

SET MAPSERVER_HOME=D:\Temp\DLLs

And the mapscript init.py file will have the following added:

import sys
import platform

if platform.system() == 'Windows' and (3, 8) <= sys.version_info:
    mapserver_home = os.getenv('MAPSERVER_HOME', '')
    if os.path.isdir(mapserver_home):
        with os.add_dll_directory(mapserver_home):
            import mapscript._mapscript

See Toblerity/Fiona#851 which has the same problem.

[jmckenna]

+1 on leveraging a MAPSERVER_HOME environment variable (and add_dll_directory), this will be very useful for Windows environments.

[dmorissette]

Should it be called MAPSERVER_HOME, or instead MAPSERVER_PATH or MAPSERVER_DLL_PATH, or even MAPSCRIPT_PATH if it's specific to MapScript?

The name MAPSERVER_HOME implies that it would apply to more than just DLLs and could potientally be the root of a complete MapServer tree where mapfiles and other support files would also be located in sub-directories, which could potentially be a future enhancement (TBD??) but is not the case here if I understood correctly.

If we want to start using MAPSERVER_HOME, then I'd suggest that the DLLs be placed in a sub-directory called MAPSERVER_HOME/lib and it's this MAPSERVER_HOME/lib directory that would be added to the dll path.

My 0.02$

[jmckenna]

Good points @dmorissette. Maybe MAPSCRIPT_PATH is better.

[geographika]

@dmorissette @dmorissette thanks for the comments.

My preference would be for MAPSERVER_DLL_PATH.

MAPSCRIPT_PATH could make users think they need to point to the mapscript installation which would be a completely different location. Adding DLL makes it clearer it is a Windows only setting and it needs to point to a folder containing *.dlls.

Possible alternative would be MAPSERVER_LIB to match PROJ_LIB.

Looking at https://github.com/Toblerity/Fiona/pull/945/files Fiona might be checking all folders on the Windows PATH for a "gdal*.dll" file and adding that folder automatically. It could be an option to search for "mapserver*.dll" with the same approach - I'll look more at this approach as it wouldn't require the user to make any changes.

[jmckenna]

MAPSERVER_DLL_PATH works for me.