Skip to content

Commit

Permalink
add doc and support schema
Browse files Browse the repository at this point in the history
  • Loading branch information
@smercier
smercier committed Apr 24, 2020
1 parent 8267c3f commit da25f32
Show file tree
Hide file tree
Showing 10 changed files with 3,553 additions and 230 deletions.
139 changes: 139 additions & 0 deletions add_csar_file.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
h1. Add csar Files in WIS database

This HowTo show how to load a file in a wis database from a Linux VM in the Lab environment.

h2. Get files

File to load comes from the PROD environment. To access this network we can transfer a file through ssh protocol via qdmzenavweb01 server ( wis.dev.ccg-gcc.gc.ca or 10.209.0.246)

<pre>
$ ssh [email protected] ls -l ./*.object.xml
[email protected]'s password:
-rw-rw-r-- 1 sftpwis sftpwis 3174 Apr 17 21:32 ./brasdor_c_20190507_0150.object.xml
-rw-rw-r-- 1 sftpwis sftpwis 3258 Apr 17 21:32 ./dalhousie_20190710_1m_0100.object.xml
-rw-rw-r-- 1 sftpwis sftpwis 3311 Apr 17 21:33 ./fortune_20190925_05m_0050.object.xml
-rw-rw-r-- 1 sftpwis sftpwis 3263 Apr 17 21:33 ./liver20191009_1-5x1-5_grid_0150.object.xml
-rw-rw-r-- 1 sftpwis sftpwis 3260 Apr 17 21:33 ./longpond_20190926_05m_0050.object.xml
-rw-rw-r-- 1 sftpwis sftpwis 3260 Apr 17 21:33 ./lunenburg_20191009_1-5m_0150.object.xml
-rw-rw-r-- 1 sftpwis sftpwis 3409 Apr 17 21:33 ./shippagan_20190409-0619_4x4_0400.object.xml
</pre>

Now copy files:

<pre>
$ scp [email protected]:./brasdor_c_20190507_0150* ./data
[email protected]'s password:
brasdor_c_20190507_0150.csar 100% 192KB 2.9MB/s 00:00
brasdor_c_20190507_0150.csar0 100% 6992KB 29.4MB/s 00:00
brasdor_c_20190507_0150_density.tiff 100% 190KB 56.8MB/s 00:00
brasdor_c_20190507_0150_depth.tiff 100% 1140KB 8.0MB/s 00:00
brasdor_c_20190507_0150_mean.tiff 100% 1089KB 15.5MB/s 00:00
brasdor_c_20190507_0150.object.xml 100% 3174 2.2MB/s 00:00

</pre>

h2. Start pgrastertime pipenv

We presume you've already have pgrastertime installed on your home (if not follow installation instruction here: https://github.com/mapgears/pgrastertime)

<pre>
~/pgrastertime$ pipenv shell
Loading .env environment variables…
Launching subshell in virtual environment…
(pgrastertime) smercier@qbccenenavwis01:~/pgrastertime$
</pre>


NOTE: on 10.209.0.182 connect with generic user `srvlocadm` and move in `./pgrastertime` directory

Be sure to point on the desired database. Update `local.ini` file for that

h2. Process files with pgrastertime

h4. Step 1

Load data in database

<pre>
$ python3 pgrastertime.py -s ./sql/postprocess.sql -t brasdor_c_20190507_0150 -p xml -f -r ../data/brasdor_c_20190507_0150.object.xml -m gdal_path=../gdal-2.4.0/

...

...

==== pgRastertime log file
--- 2020-04-21 19:56:19 : brasdor_c_20190507_0150.object.xml (1 of 1)

==== Parameters
Param : Target table -> brasdor_c_20190507_0150
Param : Source directory -> ../data/brasdor_c_20190507_0150.object.xml
Param : Force -> True
Param : Post process -> ./sql/postprocess.sql
==== Result
Import Started : 2020-04-21 19:56:19
Import Ended : 2020-04-21 19:58:17
Number of XML file to process : 1
Number of invalid XML file or failed processes : 0
Execution took 118 seconds to process

==== No Invalid files
Execution took 2.0 minutes to process

</pre>

Now we have a new `brasdor_c_20190507_0150` table in the database. We can double check with `validate` option

h4. Step 2

validate ...

<pre>
$ python3 pgrastertime.py -t brasdor_c_20190507_0150 -p validate

Validate pgrastertime table brasdor_c_20190507_0150 ...
('Space disk use 8960 kB',)
('The brasdor_c_20190507_0150 table containe 57 rows',)
('The brasdor_c_20190507_0150 table containe 1 objnam(xml files) in it.',)
('0 rows deploy in soundings_1m',)
('0 rows deploy in soundings_2m',)
('0 rows deploy in soundings_4m',)
('0 rows deploy in soundings_8m',)
('0 rows deploy in soundings_16m',)
('0 rows invalidate in soundings_1m',)
('0 rows invalidate in soundings_2m',)
('0 rows invalidate in soundings_4m',)
('0 rows invalidate in soundings_8m',)
('0 rows invalidate in soundings_16m',)

Post process run successfully!

</pre>

NOTE: any query can be added in `./sql/validate.sql` file

h4. Step 3

Two ways are available to deploy: with pgrastertime or plsql:

<pre>
$ python3 pgrastertime.py -t brasdor_c_20190507_0150 -p deploy
</pre>

<pre>
pgrastertime=# SELECT dfo_deploy('brasdor_c_20190507_0150');
pgrastertime=# SELECT dfo_invalidate( 'brasdor_c_20190507_0150','TRUE' );
pgrastertime=# SELECT dfo_update_most_recent_tables('brasdor_c_20190507_0150');
</pre>

h4. Step 4

Finally, validate ...

<pre>
$ python3 pgrastertime.py -t brasdor_c_20190507_0150 -p validate


</pre>



78 changes: 78 additions & 0 deletions create_new_wis_schema.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
h1. Create new wis Schema

This how-to allow you to create your new WIS _ad hoc_ database.

h2. 1 Create schema

The first step will be to create a schema. We assume you create a schema in a Postgresql database that already has all dfo wis functions. (see [[Database_overview]] overview for more info)

<pre>
CREATE SCHEMA z_test_sm;
</pre>

h2. 2 Create tables structure

Use this SQL script to create an empty wis database structure. Before run the script, be sure to _find and replace_ the `wis.` schema keyword for your new schema name.

*NOTE:* the script is attache to this page but can change over time. Look into the github pgrastertime for the last version of the script.
https://github.com/mapgears/pgrastertime/tree/master/sql/wis

<pre>
cp dfo_wis_soundings_1_tables_structure.sql tmp_create_schema.sql
sed -i "s/wis./z_test_sm./g" tmp_create_schema.sql
</pre>

h2. 3 Load empty tables

Run the script to create all tables in your new schema.

<pre>
psql -h 10.208.34.187 -p 5432 -U pgrastertime -W -d enav -f wis_create_schema.sql
</pre>

h2. 4 Load data

Now you can load new raster data into the database with the `pgrastertime` tool set. See the [[Add_csar]] page for more info about this topic.

h2. 5 Database process

Run those functions to add the new loaded raster into the WIS tables (soundings_x). The deploy function will append data into `soundinds` and will be partitioned in subtables table `soundinds_x` based on the `partition_insert_trigger` trigger.

<pre>
SELECT dfo_deploy('z_test_sm','brasdor_c_20190507_0150');
SELECT dfo_invalidate( 'z_test_sm','brasdor_c_20190507_0150','TRUE' );
</pre>

h2. 6 Add Raster Constraints

After you insert data into your empty table structure, you can add Raster Contraints to your tables.

NOTE: AddRasterConstraints() function work properly after you put some data in your wis structure. Adding that constraint will help PostgreSQL to queried data and will guarantee that data is valid for pgraster extension.

<pre>
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_16m'::name, 'rast'::name);
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_8m'::name, 'rast'::name);
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_4m'::name, 'rast'::name);
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_2m'::name, 'rast'::name);
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_1m'::name, 'rast'::name);
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_50cmm'::name, 'rast'::name);
SELECT AddRasterConstraints('z_test_sm'::name,'soundings_25cmm'::name, 'rast'::name);
</pre>

h2. 7 Create most_recent tables

After the most recent table is created, you have to use this function

<pre>
SELECT dfo_create_most_recent_tables('z_test_sm','soundings_16m');
SELECT dfo_create_most_recent_tables('z_test_sm','soundings_8m');
SELECT dfo_create_most_recent_tables('z_test_sm','soundings_4m');
SELECT dfo_create_most_recent_tables('z_test_sm','soundings_2m');
SELECT dfo_create_most_recent_tables('z_test_sm','soundings_1m');
</pre>

Then use update function to update most_recent tables with your new loaded data

<pre>
SELECT dfo_update_most_recent_tables('z_test_sm','[new_data_table]');
</pre>
Loading

0 comments on commit da25f32

Please sign in to comment.