installing QGIS server

[cccs-ip]

Per Kartoza's application design, CCCS will need to install QGIS Server on our web server. We'd appreciate your help in so doing.

PLEASE NOTE: One challenge for the installation process for CCCS is that nearly all the installation instructions available are predicated on using Apache. We use nginx. Please help us to configure QGIS server accordingly.

As discussed, kindly also help us to document the installation steps. To facilitate this process, I started a wiki page:

https://github.com/cccs-web/soc-maps/wiki/QGIS-Server-installation

You'll note that we have included already some examples we found about configuring 'trac', among which ws an an example nginx config. Kindly help us to revise (and delete!) as necessary.

[cccs-ip]

Paul, kindly take a crack at getting the QGIS server configured if you could, please. As I mentioned in previous conversations, I'm not very capable in regard to nginx configuration.

You'll need a directory to set this up. There should be some 'example' map that comes with the download. Let's try to configure eitiher an s3 data store or a GeoGig directory on the GitLabs server. We'll be using a cccs-maps for public materials and will maintain separate client project directories (e.g. esms-maps). Please help us maintain an appropriate structures and naming conventions.

[timlinux]

Hi

So the idea is that we will use docker for deploying services such as QGIS server. You can run nginx in front of the docker container no problem. Internally it uses apache. Is that ok for you?

We will give you automated scripts to get these things deployed so please don't spend too much time on the sysadmin side of things at the moment.

[timlinux]

See also https://github.com/kartoza/docker-qgis-server for basic instructions.

[cccs-ip]

Thanks, Tim, but I am not sure we need to or want to go that route, unless it is needed. I think Paul will be able to manage the nginx configuration, unless there are lots of known issues? To me, it sounds as if adding docker is simply introducing another service that may not be needed.

I suppose the advantage of docker is that it also introduces another layer of separation on the QGIS server (e.g., ringfencing any potential hackers)?

I would like to present these maps within a django-server, html framed webpage. Are there any different advantages / limitations to consider to either approach in that regard?

[pwhipp]

Docker is a cool and appealing idea that I've not used. I am worried about its performance implications, duplication and bloat. Running Nginx over Apache is an obvious example of this. My feeling about Docker is that it is a flavour of the month - it is quite possibly a step on from the likes of Vagrant in solving the aptly named 'deployment hell' but it is not a long term solution because containers don't play nice with each other. Sadly, that means Docker will eventually be seen be yet another case of 927.

It looks like the Apache bit could be a red herring because it punts straight to FCGI. I could need an hour or two of experimentation but I should be able to run it directly as an FCGI app under Nginx. Does that sound reasonable?

[cccs-ip]

Thanks, Paul. Let's see if we can figure out the nginx configs. I would like to know this in general, and it's a current gap in ready-to-find internet knowlege, so I think the investment is well worth it. Thank you.

[cccs-ip]

Please note that this is the thread and repo where I would like to continue this discussion.

I have created the necessary Nginx configuration and launched a fastcgi process that runs the QGIS server.

I've located the map projects here and I've documented the installation process here. I will need to add an upstart job to relaunch the fastcgi server once this is more than a test. If the server is restarted the fcgi server will need manual restarting for the time being.

The service is currently protected by a username/password (see Aaron for access).

The GetCapability test works and shows map information relating to the map.

Perhaps someone with more knowledge of QGIS can identify the necessary parameters to the call in order to display the map (I think the data is in the GetCapabilities list above but I've bashed my head against it for half an hour and got nowhere).

[cccs-ip]

What is needed to move forward? Can someone help us to fix this without being on our server? Is this just an nginx configuration issue, or are the 'calls' you are speaking of related to some other aspect of the process? Does Tim's docker configuration (including the Apache settings in it) offer any insights?

[timlinux]

Hi

As mentioned in our previous discussions, it is not a pre-requisite to use docker (though I prefer it). @pwhipp great stuff for getting QGIS Server configured under nginx. If you want to verify the service is working completely you can test it using QGIS (I know that is a bit recursive :-) ). With QGIS open, click on the add WMS layer icon:

Then create a new connection:

In the connection details, specify the url to the resource

And use your HTTPAuth user/password. Say ok and then click connect in the main WMS dialog

You should see the layers from your project listed there. Select any layers you want and add them to the map.

[timlinux]

Generally we would use a client like leaflet to do in-browser display of a map. One caveat I should mention is that most browser based mapping toolkits will expect the http source for the mapping service to be open. We will need to do some tests to see if we can make e.g. leaflet work nicely against authenticated HTTP. See for example this thread:

http://gis.stackexchange.com/questions/49245/how-to-access-an-wms-layer-that-requires-authorization-via-openlayers

[gubuntu]

Or ahocevar's reponse here using basic http auth: http://osgeo-org.1560.x6.nabble.com/Load-GeoServer-WMS-secured-layers-with-OpenLayers-td3919938.html and this thread has some interesting approaches: https://www.mail-archive.com/[email protected]/msg13083.html.

It depends how important authentication is and whether clear text credentials are acceptable. I would say authentication is essential for the private layers.

[cccs-ip]

Thanks for this, Tim and Gavin.

With regard to Docker: I am happy to give it a try, and it is working well locally. In terms of installatino on the CCCS website, Paul has done a lot of work to build our site in such a way that it is contained within a stable virtual environment, and scripted so that we can run a simple pip install to get all the components set up. In this context, docker represents a bit of redundancy--introducing an additional bit of software that is not strictly necessary.

This is not to say that I do not see a place for Docker -- either for the map application, or for the website overall. Since docker allows for modular deployment of a working QGIS server environment, it might be handy for us to maintain one or several different docker environments for sharing with different project experts at different points in our work (sort of like packaging maps--including both QGIS files and the database--into a "deliverable" package for a specific objective). We could also maintain our "public" application as a docker application. Your work already takes us most of the way to this goal. CCCS might also decide eventually to wrap our entire web framework into a docker app for sharing with developers and designers, who sometime get hung up on the website deployment.

For the moment, I have asked Paul to move forward to get QGIS server installed directly to the site. I have also asked him to look at the database work that you did and to advise if we need to change or modify anything to get the mapping databases to play nicely with our web databases.

Paul, did Tim's and Gavin's comments help you to determine how to configure QGIS server to present a qgis file as a map (rather than HTML output)?

[pwhipp]

Yes. It should all be working. It does seem very clunky though and may still need some attention - I think it is fixated on my example although my intention was that you could place the relevant project files here and then reference them using

http://qgis.crossculturalconsult.com/<project_filename.qgis>?REQUEST

For example:

http://qgis.crossculturalconsult.com/alaska_lakes.qgis?REQUEST=GetCapabilities

It seems to be fixated on alaska_lakes (the first one I set up) - I'm not sure why and this suggest that the URL might not be right. If I switch and use:

http://qgis.crossculturalconsult.com/foo.qgis?REQUEST=GetCapabilities

It still reports alaska_lakes as a queryable layer and does not complain about the non-existence of foo.

I'll carry out the suggested test later this week so I've left this assigned to me but as it is reporting capabilities, it is likely to benefit most from someone who undersands QGIS having a play.

Is the server 'locked' to a single project somehow?

@timlinux and @gubuntu : Note that the authentication is just a basic layer to stop prying eyes. This is not the authentication we'd use on the Abadi site.

[cccs-ip]

Thanks, Paul - and yes: our client websites are password protected and do not utilize the authentication layer we use on staging... which is mostly about keeping out bots.

[timlinux]

Hi @pwhipp

It seems to be fixated on alaska_lakes (the first one I set up) - I'm not sure why and this suggest that the URL might not be right. If I switch and use:

QGIS server will publish all the maps in the directory or at the absolute path you specify e.g. by passing the map parameter it should publish the specific map you intend.

MAP=/foo/bar.qgs

Additionally you can use the LAYER=foo,bar option to specify a subset of the layers in the project and also the order in which those layers should be rendered. If you specify the project name itself as a parameter for layer (case sensitive, excluding the .qgs extention) then you will get a render identical to that of your project layer order etc.

I am busy building a small django app (and @ismailsunni will be helping with it next week) that will basically let you specify an endpoint (url) and a layers list and then provide views / template tags that will contain a leaflet page (or snippet in the case of tag) for that project. My idea was that you would use a simple django admin page to then define the maps, and then refence the view where needed or embed the map directly into your content using a template tag.

[timlinux]

@cccs-ip wrote:

With regard to Docker: I am happy to give it a try, and it is working well locally. In terms of installatino on the CCCS website, Paul has done a lot of work to build our site in such a way that it is contained within a stable virtual environment, and scripted so that we can run a simple pip install to get all the components set up. In this context, docker represents a bit of redundancy--introducing an additional bit of software that is not strictly necessary.

Ok as I mentioned I love docker but I have no problem if you don't see a fit for in your workflows. Great that you are trying it out - I created a new docker image now so that you can easily deploy the public shared data files to a host like this:

docker run --name "public.cccs" \
    --restart=always \
    --hostname="public.btsync" \
    -e SECRET=123456 \
    -e DEVICE=public.btsync \
    -v /home/blah/yourshare:/btsync \
    -d -t kartoza/btsync

That will then create a single purpose btsync share for the shapefiles (I've ommitted specific details in the above example).

This is not to say that I do not see a place for Docker -- either for the map application, or for the website overall. Since docker allows for modular deployment of a working QGIS server environment, it might be handy for us to maintain one or several different docker environments for sharing with different project experts at different points in our work (sort of like packaging maps--including both QGIS files and the database--into a "deliverable" package for a specific objective).

Yes this is exactly the kind of use case it is good for. Also each application element is sandboxed, so a vulnerability in say PostGIS will generally be isolated to that container.

We could also maintain our "public" application as a docker application. Your work already takes us most of the way to this goal. CCCS might also decide eventually to wrap our entire web framework into a docker app for sharing with developers and designers, who sometime get hung up on the website deployment.

Yes all good uses cases should you wish to delve into docker further.

For the moment, I have asked Paul to move forward to get QGIS server installed directly to the site. I have also asked him to look at the database work that you did and to advise if we need to change or modify anything to get the mapping databases to play nicely with our web databases.

Yes I had a look at his nginx work - well done @pwhipp - I've only ever set it up on Apache so nice to see it deployed under nginx.

[cccs-ip]

Thanks, Tim. I hope we're not getting too far ahead of ourselves by creating these many docker images though (for public, for client sites, etc). At the moment, it seems that we still haven't finished building a model to show the QGIS server in our Django application?

[cccs-ip]

@timlinux and @pwhipp

Paul noted that we were having problems where we were not able to easily point to different qgis files to show different maps. Has anyone figured out what the issue is? Are we yet able to show different qgis files? Can we link these into our websites in any way?

[timlinux]

Hi cccs-ip

Apologies - my comment above (#5 (comment)) didnt come out formatted properly so you may have missed the latter part. In that comment I explain how you normally set the qgis map file using the map parameter.

@ismailsunni and I will have the wms client django app working for you this week with a simple admin interface that will let you point to a map and its layers and get a leaflet map control back. If you are wanting to get a map embedded into your site more urgently, I can provide you with a snippet that will create a leaflet map against a QGIS project file (via QGIS server).

Regards

Tim

[pwhipp]

thanks @timlinux,

Rather than mess around with the QGIS server which I see from your comment is bound to a single project so we'd need a new server for each new project. This is not practical.

How do I pick up your app and use it?

[pwhipp]

Thanks @timlunux, is your app now in a state that I can use it?

How do I set about integrating it as a test/demo?

[cccs-ip]

Hi Team,

Thank you for continuing to work on this. I am trying to get my head around what are issues and hang-ups are. Please allow me to summarize where I think we are at:

APP "INSTALLATION" AND DEPLOYMENT AS A DOCKER APPLICATION

The README file prepared by Tim and the Kartoza team documents hwo to initialize our specific data sets (currently differentiated as 'public' for CCCS' public site and 'private' for a particular CCCS client project). The source materials from which the scripts provided on our site are all currently hosted in a single, private Git repo designated for one CCCS client project (a repo that currenly also holds public data we will make available on the CCCS public site).

The kartoza scripts load data into a docker virtual environment as a complete application stack. As I have discussed with each team, while I appreciate how the docker container is helpful for compartmentalizing map data and servers (which has a clear security advantage), my preference remains to implement QGIS server directly to our site (without docker) so that he whole application can be pip installed as part of the full 'cccs/web-core` site deployment. I have therefore asked Paul to look over the kartoza install scripts, and to get the qgis application stack working independently of docker.

Paul's question suggests that once cannot deploy more than one QGIS server within a web application virtual environment, and that a single QGIS server can server only a single QGIS project file. I agree that this sounds a bit impractical, and I suspect it's just our lack of familiarity with QGIS server. @kartoza: Can't we directly to the server and still be able to host multple qgis project files?

For the CCCS website, we will want to host different qgis project files on different web pages (e.g. different maps showing areas of CCCS experience, locations of lands claimed by IP groups, demos of how we link statistical data to village shape files, etc.). For our client sites (generally speaking), it is, however, fine to have just one QGIS project file that is structured to allow users to click on and off different layers.

LOADING DATA INTO CCCS' DATABASE

To me, the first step seems simple to be evaluating the SQL script to get our spatial data into the our postgis database.

It is important to me to keep all CCCS and CCCS client data together in a single, unified, and logically-organzied database. Keeping the data in a single place (and forcing ourselves to confront hard questions about standards for data organization and curation) is my main priority (hence trying to avoid docker for the sole purpose of sharing a stack of QGIS-related apps). How we organize data becomes even more critical as we eventually link qgis files to, for example, all the socio-economic data tables i've been processing (shared with you via dropbox.. files we want to better organize and make available for statistical analysis).

As for docker, it might be that Tim prefers it b/c it allows one to have many individual qgis servers running. Also, I appreciate the idea of trying to make the application immediately deploy-able (independently of our django app). From the standpoint of our web application, however, I also feel that this redundancy is probably unnecessary.

PRESENTING QGIS FILES IN DJANGO

I am still trying to get clarity about our current hang-ups are with regard to presenting a qgis file within our django web application. I gathered from Tim that we need some sort of "plug-in" or "widget" to connect the QGIS server into Django so that one can embed the server within a web page.

I am also un-familar with QGIS as a web server, and I am further at a loss for not really understanding web apps or django (or databases).

It seems to me that the unresolved question is how to present a *.qgis project file, which manipulates and connects those data. In this regard, I have found the following as potentially informative resources:

• http://plugins.qgis.org/
• http://hub.qgis.org/projects/qgis-web-client
• https://github.com/qgis/qgis-web-client
• http://hub.qgis.org/projects/quantum-gis/wiki/QGIS_Server_Tutorial

At this point, i would like to ensure that we have all the data loaded into the database and that we have the qgis files in place; we need to look at the database loading procedure b/c if there is a need to rename the schemas at all to be meaningful in our system, this will affect how the data links into the qgis project file. Until the data is loaded in our database, there seems to be limited value in continuing to work too much with the qgis presentation file, since all the paths will have to be adjusted (so adding more layers means having to fix many data layer links).

[cccs-ip]

The following is from @pwhipp

Here is my current thinking following my review of the technology and work to date:

CCCS needs to present many complex annotated maps. These maps require interactive features akin to google maps where layers are shown/hidden, features are highlighted and spatial analysis may be performed.

QGIS is a tool that creates interactive map projects. The maps are presented and analysed within the QGIS tool.

QGIS Server presents a single QGIS project as an image that can be presented on a web page. For CCCS purposes, this is a red herring.

OpenLayers (OL) is a javascript appliction that presents data from map services. It can present complex interactive maps in layers. CCCS web maps will be presented using OL3.

QGIS and OL are not related but they process similar data. Both deal with aggregations of maps (layers) and their associated styling. There are three layer aspects of interest:

Features: Vector layer features and attributes
These need to be handled using a database so that the features draw on a common CCCS set and projects may extend and modify that set over time. For this we need a single online CCCS POT for features. This POT will be a PostGIS database service.
Styling: Symbology
The coloring, associated graphics and annotation of a layer. If developed in QGIS, this information needs to be represented within the PostGIS database service so that it can be transformed for appropriate presentation using OL.
Layer Attributes
Layer visibility, extent, precedence (for ordering). If developed in QGIS, this information needs to be represented within the PostGIS database service so that it can be transformed for appropriate presentation using OL.
To effectively present project data as interactive maps in browsers, CCCS needs to operate at least one map service which will supply the data to browsers running javascript that extends OL.

To prepare the data for presentation, tools such as QGIS will remain very useful. However a pipeline needs to be established to migrate the QGIS features, styling and layer attributes into the form needed to serve via the map service.

Legacy QGIS projects have created shapefiles. These will need to be individually handled and imported into PostGIS. New QGIS projects should be able to create PostGIS layers via a CCCS PostGIS service. This PostGIS service will be related to the Map Service making the features immediately available over the map service.

I am not yet sure how/whether it can also make the styling and layer attributes available. This is a possibility which is likely to involve some PyQGIS plugin work.

[cccs-ip]

More from @pwhipp

Further review... The QGIS web client project is a suite of openlayers stuff that works with the QGIS server to present a QGIS project as a nice interactive map.

Sounds good.

The xxx.ggs file is an XML file which encapsulates the presentation and layer information pretty much as you thought. It references the data sources for each layer. For shape files these are generally relative path references. These need to be present if the QGIS server is going to have any chance of serving their content.

I've added these in my local branch and verified that I can get the nginx QGIS server working over them. The nginx hosted QGIS server is looking good.

I've installed the QGIS web client. For some reason, loading maps is really really slow (as in terminally slow). I'm working on that. I believe there may be some version issues with QGIS causing the problem. It may also be because the javascript is making some protocol assumptions that are not the correct for my QGIS server.

I may have to copy in the web client javascript files and modify them appropriately (there is no setup or easy way to use their project that I can see).

The good news is that I think I'm close to being able to give you a projects folder where you can dump your QGIS project (plus all data sources if they are shapeflies) and then be able to access it on a web page on the site. I am optimistic that I'll have this up and running before COB tomorrow.

[pwhipp]

Digging deeper into the QGIS web client, it is built upon sencha's EXT JS framework. This framework is not open source. It costs $3,225 to license for up to five developers.

I have the qgis server serving up different projects via Nginx which I can verify through its GetCapabilities etc.

I tried reviewing the client to port it into a CCCS site page but the javascript is rather opaque and the Ext JS stuff is obscure without me having to do quite a lot of reading.

As an expediency, I set up a vanilla QGIS client and tried hooking that up to the server. This gets quite far:

As you can see, it has picked up on the layers but fails because wmsLoader.projectSettings.service is undefined.

I found this link on google. However, the version of WebgisInit.js I have (b22d92c2), looks like it has been worked on quite a bit since that post.

I'm continuing to see if I can work through this but any help would be much appreciated.

[cccs-ip]

Thanks for continuing this investigation, Team. I am not fully aware of the status of the various points and issues raised in this discussion thread. @pwhipp has been making good progress to get our application up and running. Here's a recent update:

---

You can go here to see three examples. The page automatically updates so if you add additional projects using the instructions, they will appear in the list.

helloworld is working but seems to have rather a close minimum zoom

mtb_data presentation breaks with a javascript error in the qgis_web_client code.

naturalearth presentation is also having some issues but seems to be mostly working.

They are all somewhat slow although this may be because of the errors or the fact that they are all being routed through SSL.

All of the issues appear to relate to the QGIS web client itself. Unfortunately the web client is written using Ext JS version 3.4. It is a commercial product now on version five.

It should be possible to modify the server to handle multiple maps - not doing so seems insane. This will allow one service to serve all of the qgis projects.

It should also be possible to get them presenting more robustly. Implementing the presentation from scratch using OL3 would take weeks (as would re-implementing the qgis server to handle the projects more robustly).

---

I would like help to continue to improve our capacity to present multiple map files. I would also like any guidance you can provide to help fix the CCCS qgis file that we're using to test our 'public' presentation. Please note that our test file is not the one created by Kartoza, as we continue to try to identify a solution that is not dependent on docker. That test file is located at `/home/sync/cccs-maps/public/bps_indonesia/MTB_public/MTB\ data.qgs/

Thanks for your continued help and support.

[cccs-ip]

@timlinux sent this material by email (and I think also included it in response to a different post). It looks relevant and I wanted to make sure it didn't get lost:

QGIS Server presents a single QGIS project as an image that can be presented on a web page. For CCCS purposes, this is a red herring.

​Correction - QGIS server presents a series of QGIS projects using the standard web services : Web Mapping Service (WMS) and Web Feature Service (WFS). These services can be consumed by various clients including OpenLayers, Leaflet. Within the WFS spec you can ask for available layers, and selectively display all or a subset of layers.​

QGIS and OL are not related but they process similar data. Both deal with aggregations of maps (layers) and their associated styling.

​In terms of our architecture, QGIS Server provides a backend service, OL/Leaflet provides a client side front end to that service. They both 'speak' WMS as their interoperability layer.​

Styling: Symbology
The coloring, associated graphics and annotation of a layer. If developed in QGIS, this information needs to be represented within the PostGIS database service so that it can be transformed for appropriate presentation using OL.

​> The styling can be store in the database by QGIS, but typically also resides in the .qgs project file. When using the WMS service, the styling defined on the server side is rendered into map images on the server and sent pre-styled to the client. When using the WFS service, the client (e.g. OL3) would be completely responsible for styling. Typically any more complex styling activities would be done server side.​

Layer Attributes
Layer visibility, extent, precedence (for ordering). If developed in QGIS, this information needs to be represented within the PostGIS database service so that it can be transformed for appropriate presentation using OL.

​> No - these are purely artifacts of the QGIS project file and or the OL/Leaflet client. For example a QGIS project may define layers A, B, C but the WMS client may request them in order A, C, B and they will be rendered as such. Its best to think of PostGIS as a 'smart data store' and it is not it's responsibility to do any styling of data. QGIS does have the ability to store styling information in the database, but PostGIS isn't intrinsically aware of this (it considers it to be just a stored xml document).​

To prepare the data for presentation, tools such as QGIS will remain very useful. However a pipeline needs to be established to migrate the QGIS features, styling and layer attributes into the form needed to serve via the map service.

​> ​QGIS Server does this for us. It also distributes attribute data via the 'getFeatreInfo' WMS API.​

[cccs-ip]

We must really be missing something simple. Looks like the default behavior should allow multiple maps:
http://anitagraser.com/2012/03/30/qgis-server-on-ubuntu-step-by-step/

[cccs-ip]

moving this to issue #17