support windows install through Conda environment

Pipfile

@@ -17,7 +17,4 @@ alembic = "*"
 pgrastertime = {path = ".",editable = true}
 "psycopg2" = "*"
 requests = "*"
-gdal = "*"
-
-[requires]
-python_version = "3.5"
+gdal = "*"

README.md

@@ -9,6 +9,8 @@ will be used to query raster value at a specific time for analysis.
 
 # Install on Ubuntu
 
+On Ubuntu, pgRastertime run in a Python virtual environment (pipenv)
+
 ## Python virtual environment
 ```
 sudo apt install python3 pip python-pip
@@ -18,29 +20,18 @@ cp development.ini local.ini
 ```
 Edit the local.ini to fit your installation
 
-## Map Algebra 
-This tool need to perform map algebra operations on raster.  We use gdal_calc.py script for this.
-
-```
-wget -O gdal_calc.py https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/scripts/gdal_calc.py
-```
 
 ## Update dependencies
+
 ```
 pipenv shell
 pipenv sync
-```
-*NOTE:* If GDAL Python Library fail to install do this:
-
-```
-export CPLUS_INCLUDE_PATH=/usr/include/gdal
-export C_INCLUDE_PATH=/usr/include/gdal
-pipenv run pip install "GDAL<=$(gdal-config --version)"
+python pgrastertime.py
 ```
 
 ## Database
 
- - Find your Postgresql version and install
+Find your Postgresql and Postgis to install for your Ubuntu LTS version
 
 ```
 sudo apt-cache search postgresql
@@ -50,44 +41,63 @@ sudo apt-cache search postgis
 sudo apt-get install postgresql-9.3-postgis-2.1 postgresql-9.3-postgis-2.1-scripts
 ```
 
+# Install on Windows
 
- - Init your database
+This installation has been tested on Windows 10
 
-```
-sudo su postgres
-psql
-CREATE USER loader WITH PASSWORD 'ChangeMe';
-CREATE DATABASE pgraster WITH OWNER loader ENCODING 'UTF8';
-\q
-psql -d pgraster -U loader -c "CREATE EXTENSION postgis;"
-cat  ./sql/dfo_add_conformance_band.sql ./sql/dfo_add_shoal_geom.sql ./sql/dfo_calculate_tile_extents.sql ./sql/dfo_calculate_tile_geoms.sql ./sql/dfo_delete_empty_tiles.sql ./sql/dfo_merge_bands.sql ./sql/dfo_metadata.sql ./sql/dfo_most_recent.sql ./sql/init_exta.sql  | psql pgraster -U loader -1 -f -
+1- Install Postgresql (version 10.9 tested)
+
+ * Download and install PostgreSQL from EnterpriseDB.
+
+https://www.enterprisedb.com/thank-you-downloading-postgresql?anid=1256722
 
-#to allow the deploy step.
-cat ./sql/dfoTables/parent.sql ./sql/dfoTables/25cm.sql ./sql/dfoTables/50cm.sql ./sql/dfoTables/1m.sql ./sql/dfoTables/2m.sql ./sql/dfoTables/4m.sql ./sql/dfoTables/8m.sql ./sql/dfoTables/16m.sql | psql pgraster -U loader -1 -f -
+NOTE: choose your admin password and your database port
+
+ * Run the **StackBuilder** utility and install the PostGIS add-on.
+
+NOTE: Chose PostGIS bundle and clic "YES" to all installer question
+
+3- Install Conda environment
+
+The best way to pgRastertim on Windows is to build a dedicated silo where all needed package will be installed.  For this need we use cross platform Conda packaging management solution.
+
+ * Install Miniconda
+
+Install the latest available Miniconda python 3 version from here :  https://conda.io/en/latest/miniconda.html
+
+ * Create a "pgratertime" conda environment
 
 ```
+conda create -n pgrastertime python=3.7
+conda activate pgrastertime
+conda install -c conda-forge --file conda-package.lst
+```
 
-Update database connection info in `local.ini` file
+# Init your database
 
+Basic postgresql database init
 
- - Load table:
-
 ```
-alembic -c local.ini upgrade head
+psql -h localhost -p 5432 -U postgres -W
+CREATE USER loader WITH PASSWORD 'ChangeMe';
+CREATE DATABASE pgraster WITH OWNER loader ENCODING 'UTF8';
+\q
+psql -h localhost -p 5432 -d pgraster -U loader -c "CREATE EXTENSION postgis;"
+psql -h localhost -p 5432 -U loader -W -d pgraster -U loader -f ./sql/init_exta.sql
 ```
 
-### To add new fields in the model
+Optional sql files to load to perform wis custom operations.
 
 ```
-alembic --config local.ini revision --autogenerate -m "Add sys_period"
-alembic -c local.ini upgrade head
+psql -h localhost -p 5432 -U loader -W -d pgraster -U loader -f ./sql/wis/dfo_functions.sql
+psql -h localhost -p 5432 -U loader -W -d pgraster -U loader -f ./sql/wis/dfo_all_tables.sql
 ```
 
-## Running pgrastertime
+
+# Running pgrastertime
 
 ```
-pipenv shell
-pgrastertime -h
+python pgrastertime.py -h
 usage: pgrastertime [-h] [--config config_file] --tablename TABLENAME
                     [--sqlfiles SQLFILES] [--dataset DATASET]
                     [--reader READER]
@@ -133,39 +143,39 @@ The `local.ini` is the default configuration file.  You can have multiple config
 can use `-c` flag to use a different one.
 
 ```
-pgrastertime -c myconf_dev.ini -r ./data/ -p xml
+python pgrastertime.py -c myconf_dev.ini -r ./data/ -p xml
 ```
 
 *NOTE:* When use GDAL with path, add this environment variable
 ```
 export GDAL_DATA=/usr/share/gdal/2.2/
 ```
 
-## Example
+## Examples
 
 First iteration of pgRastertime was designed to import your raster data in a postgresql database.  You need to 
 edit your local.ini file to change your postgresql connection info, local path and postprocess file. 
 
 ```
-pgrastertime -t testtable -r ./data/18g063330911_0250.object.xml -p load
+python pgrastertime.py -t testtable -r ./data/18g063330911_0250.object.xml -p load
 ```
 
 A specific driver was added for a specific raster format define by an XML file. You can create your own 
 driver in `process` folder.  As example, `xml_import.py` driver alows to import files link to a specific XML object file.
 
 ```
-pgrastertime -t testtable -r ./data/18g063330911_0250.object.xml -p xml
+python pgrastertime.py -t testtable -r ./data/18g063330911_0250.object.xml -p xml
 ```
 
 You can add post process SQL script(s) to the command line (can be multiple script separated by commas).  
 Postprocess script (-s option) are execute after each raster updated in table.  Use `pgrastertime` template
 name and the pgrastertime script will find and replace them with your target table name of `-t` flag. 
 
 ```
-pgrastertime -s  ./sql/basePostProcess.sql -t testtable -f -r ./data/ -p xml
+python pgrastertime.py -s  ./sql/basePostProcess.sql -t testtable -f -r ./data/ -p xml
 
 if secteur_sondage (dfo) is loaded in db we can use postprocess.
-pgrastertime -s  ./sql/postprocess.sql -t testtable -f -r ./data/ -p xml
+python pgrastertime.py -s  ./sql/postprocess.sql -t testtable -f -r ./data/ -p xml
 
 To validate postprocess
 select metadata_id,resolution , st_scalex(raster),st_area(tile_geom) ,filename ,st_numbands(raster) 
@@ -174,13 +184,12 @@ from soundingue  ;
 ```
 
  * The force `-f` optional flag is used to force overwrite the target table.  When force is not use and `-r` is a directory, all validation is made to import ONLY raster that is not already processed.  This check is made through the metadata target raster table.
- * w
 
 You can `deploy` your pgrastertable table ( `-t` flag) to your production table through `./sql/deploy.sql` script (edit this
 script for your needed).  
 
 ```
-pgrastertime -p deploy -t testtable
+python pgrastertime.py -p deploy -t testtable
 
 sql validation
 select count(*) from  soundings_4m; should be greater than 0
@@ -190,19 +199,19 @@ select count(*) from  soundings_error; should be 0
 Validation script can be use and updated for your need.
 
 ```
-pgrastertime -p validate -t datatest
+python pgrastertime.py -p validate -t datatest
 ```
 
 This custom sedimentation process need multiple input value add with `-m` flags.  This example output the table processed with a cutome postgresql function to a tif file
 
 ```
-pgrastertime -t soundings_4m -m time_start='2017-12-31' -m time_end='2018-10-22' -m resolution=4 -d ./datatest/secteur.shp -o ./datatest/sm1.tif -of gtiff -v -p sedimentation
+python pgrastertime.py -t soundings_4m -m time_start='2017-12-31' -m time_end='2018-10-22' -m resolution=4 -d ./datatest/secteur.shp -o ./datatest/sm1.tif -of gtiff -v -p sedimentation
 ```
 
 In this example, the output is a Postgresql table `my_table`
 
 ```
-pgrastertime -t soundings_4m -m time_start='2017-12-31' -m time_end='2018-10-22' -m resolution=4 -d ./datatest/secteur.shp -o my_table -of pg -v -p sedimentation
+python pgrastertime.py -t soundings_4m -m time_start='2017-12-31' -m time_end='2018-10-22' -m resolution=4 -d ./datatest/secteur.shp -o my_table -of pg -v -p sedimentation
 ```
 
 ## Todo list

conda-package.lst

@@ -0,0 +1,6 @@
+gdal
+sqlalchemy
+geoalchemy2
+psycopg2
+numpy
+alembic

pgrastertime/commandline.py

@@ -61,7 +61,7 @@ def parse_arguments():
         help='Output format Geotiff or PostGIS table'
     )
     parser.add_argument(
-        '--param', '-m', nargs='?', action='append',
+        '--param', '-m', nargs='?',default='', action='append',
         help='Option(s) input'
     )
     parser.add_argument(
@@ -198,7 +198,8 @@ def main():
                              args.force,
                              args.sqlfiles,
                              args.verbose,
-                             args.dry_run).importRasters() != "SUCCESS"):
+                             args.dry_run,
+                             args.param).importRasters() != "SUCCESS"):
                         er += 1
                         error_list.append(os.path.join(args.reader, file))
                     else:
@@ -245,7 +246,8 @@ def main():
                              args.force,
                              args.sqlfiles,
                              args.verbose,
-                             args.dry_run).importRasters()                 
+                             args.dry_run,
+                             args.param).importRasters()                 
 
             print("\n\n==== Parameters\n")
             print("Param : Target table -> %s" % args.tablename)

pgrastertime/processes/load_raster.py

@@ -31,9 +31,10 @@ def run(self):
                 pg_dbname = CONFIG['app:main'].get('sqlalchemy.url').split('/')[3]
                 pg_pw = CONFIG['app:main'].get('sqlalchemy.url').split('/')[2].split('@')[0].split(':')[1]
                 pg_user = CONFIG['app:main'].get('sqlalchemy.url').split('/')[2].split('@')[0].split(':')[0]
-                cmd = "PGPASSWORD=" + pg_pw + " psql -q -p " + pg_port + " -h " + pg_host + \
+                cmd = "psql -q -p " + pg_port + " -h " + pg_host + \
                       " -U " + pg_user + " -d " +  pg_dbname + \
                       " -f " + filename + ".sql"
+                os.environ["PGPASSWORD"] = pg_pw    
                 if subprocess.call(cmd, shell=True) == 0:
 
                     ## if raster file upload w/ success, we update some metadata in DFO model

pgrastertime/processes/raster2pgsql.py

@@ -37,8 +37,8 @@ def run(self):
 
         # if raster2pgsql run w/ success, we import SQL file in database
         user_param = self.getConParam()
-        cmd_psql = "PGPASSWORD=%s psql -q -p %s -h %s -U %s -d %s -f %s.sql" % (
-                            user_param['pg_pw'],
+        os.environ["PGPASSWORD"] = user_param['pg_pw']
+        cmd_psql = "psql -q -p %s -h %s -U %s -d %s -f %s.sql" % (
                             user_param['pg_port'], 
                             user_param['pg_host'],
                             user_param['pg_user'],

pgrastertime/processes/xml_import.py

@@ -37,9 +37,9 @@ def insertXML(self, xml_filename):
            print("Fail to convert xml to sql...")
            return False
 
-        con_pg = self.getConParam() 
-        cmd = "PGPASSWORD=%s psql -q -h %s -p %s -U %s -d %s -f ins.sql" % (
-                     con_pg['pg_pw'],
+        con_pg = self.getConParam()
+        os.environ["PGPASSWORD"] = con_pg['pg_pw']   
+        cmd = "psql -q -h %s -p %s -U %s -d %s -f ins.sql" % (
                      con_pg['pg_host'],
                      con_pg['pg_port'],
                      con_pg['pg_user'],

pgrastertime/processes/xml_resampling.py

@@ -9,17 +9,27 @@
 from pgrastertime.processes.spinner import Spinner
 from pgrastertime import CONFIG
 import subprocess
-import sys, os, ntpath, tempfile
+import sys, os, ntpath, tempfile, platform
 
 class XML2RastersResampling:
 
-    def __init__(self, xml_filename, tablename, force, sqlfiles, verbose, dry_run):
+    def __init__(self, xml_filename, tablename, force, sqlfiles, verbose, dry_run,userparam=''):
         self.xml_filename = xml_filename
         self.tablename = tablename
         self.force = force
         self.sqlfiles = sqlfiles
         self.verbose = verbose
         self.dry_run = dry_run
+        self.userparam = userparam
+
+    def getParams(self,param):
+        # this process need gdal_path parametre
+        p = ''
+        for i in self.userparam:
+            if i.split("=")[0].lower() == param:
+                p = i.split("=")[1]                                      
+        return p
+
 
     def getConParam(self):
         conDic = {}
@@ -38,14 +48,16 @@ def insertXML(self, xml_filename):
            print("Fail to convert xml to sql...")
            return False
 
-        con_pg = self.getConParam() 
-        cmd = "PGPASSWORD=%s psql -q -h %s -p %s -U %s -d %s -f ins.sql" % (
-                     con_pg['pg_pw'],
+        con_pg = self.getConParam()
+
+
+        os.environ["PGPASSWORD"] = con_pg['pg_pw']
+        cmd = "psql -q -h %s -p %s -U %s -d %s -f ins.sql" % (
                      con_pg['pg_host'],
                      con_pg['pg_port'],
                      con_pg['pg_user'],
                      con_pg['pg_dbname'])
-                     
+
         if subprocess.call(cmd, shell=True) != 0:
              print("Fail to insert sql in database...")
              return False
@@ -179,7 +191,9 @@ def ImportXmlObject(self, raster_prefix):
         resolutions = CONFIG['app:main'].get('output.resolutions').split(',')
         raster_dict = self.initRasterFileDict()
 
-
+        # Check gdal_path param
+        gdal_path =  self.getParams('gdal_path')
+
         #  loop in all raster type
         nb_of_raster = 0
         for raster_type in ['depth', 'density', 'mean', 'stddev']:
@@ -194,7 +208,7 @@ def ImportXmlObject(self, raster_prefix):
                                      False, 
                                      self.force, 
                                      True, 
-                                     '/home/srvlocadm/gdal-2.4.0/apps/',
+                                     gdal_path,
                                      raster_type)
                # start at the  nearest resolution                
                if float(resolution) >= float(reader.resolution):

sql/dfoTables/16m.sql → sql/wis/16m.sql

@@ -28,9 +28,6 @@ WITH (
 )
 TABLESPACE pg_default;
 
-ALTER TABLE soundings_16m
-    OWNER to stecyr;
-
 -- Index: soundings_16m_lower_sysperiod
 
 -- DROP INDEX soundings_16m_lower_sysperiod;

sql/dfoTables/1m.sql → sql/wis/1m.sql

@@ -28,9 +28,6 @@ WITH (
 )
 TABLESPACE pg_default;
 
-ALTER TABLE soundings_1m
-    OWNER to stecyr;
-
 -- Index: soundings_1m_lower_sysperiod
 
 -- DROP INDEX soundings_1m_lower_sysperiod;

sql/dfoTables/2m.sql → sql/wis/2m.sql

@@ -114,9 +114,6 @@ WITH (
 )
 TABLESPACE pg_default;
 
-ALTER TABLE soundings_vnsl_2m
-    OWNER to stecyr;
-
 -- Index: soundings_vnsl_2m_lower_sysperiod
 
 -- DROP INDEX soundings_vnsl_2m_lower_sysperiod;