This repository contains the OSM2Mapaction toolbox, for acquiring, transferring, and converting OpenStreetMap data within MapAction.
The OSM2Mapaction toolbox is divided into two parts or toolsets.
The first toolset is the OSM Change Toolset. This provides tools for acquiring OSM data, and enabling a field team to update their copy to reflect the latest OSM data without the need for them to download the whole OSM data file again.
The second toolset is the OSM to MA DNC Toolset. This provides a tool to translate the OSM data into ESRI shapefiles in the Mapaction Data Naming Convention, each containing the relevant features.
All of the tools are implemented as Python scripts, which can be run at the command line if you are comfortable with that. However, for each tool an ArcToolbox interface is also provided to enable you to set up and run each tool using a familiar interface. For most users, using this toolbox is the recommended approach.
All of the tools should work on the standard MapAction laptop environment, but because they rely on various non-standard libraries such as GDAL, you'd need to make a few configuration changes which aren't documented here to run them on other computers.
The scripts (and corresponding ArcToolbox tools to run them) in this repository are as follows:
-
Tool 0 - Get OSM files for country. This tool simply enables the MapAction support base to download OSM data from the geofabrik server (as a master PBF file). The script called by this tool is for doing this is
0_get_OSM_files.py. This doesn't really do anything that you can't do by downloading the file yourself with a web browser. -
Tool 1 - Create Difference File. This tool is intended to be run by Support Base when a field team is deployed and they have limited internet bandwidth. It takes as input two OSM .PBF format files, one of which should be the OSM data that the field team deployed with and the other should be the latest version of the OSM data. The tool will create a file that represents only the difference between those two input files, which should hopefully be much smaller than either of the actual input files themselves and which can therefore be more easily transferred to the Field Team. The Python script called by this tool is 1_create_difference_SB.py.
-
Tool 2 - Append Difference File. This is effectively the reverse of Tool 1 and is intended to be run by the Field Team. It will take the original OSM .PBF file from the start of the deployment, and the difference file provided by Support Base. It will output a new .PBF file that reflects all the changes and is therefore up-to-date. The python script called by this tool is 2_append_differnce_FT.py.
- OSM2MapActionRunner. This tool runs the main OSM2MA data conversion script (which is osm2ma/osm2mapaction.py). This program reads an OSM .PBF file and converts it into a new set of ESRI Shapefile in the MapAction DNC. (Note - it doest not update existing shapefiles!). The mapping between OSM tags / values and MapAction DNC shapefile names is derived from an Excel file which is stored in the installation directory of the tools - keeping this file up to date to reflect changes in OSM over time will of course be required.
The conversion process can be quite lengthy, but you will see progress in a window that opens when you run the tool.
All attributes are stored in OSM data as text. The conversion tool will attempt to determine what type and width each field in the shapefile should be, but this calculation also takes some time.
This behaviour, and some other aspects of how the tool works, can be controlled if you run the tool at the command line (rather than via ArcToolbox). To run the tool at the command line you should open an OSGEO4W command promppt and find the tool in the OSM2Mapaction installation directory. The main script is osm2ma/osm2mapaction.py. You can run the script with a -h to see what the options/arguments are. Otherwise, you can run the tool from the ArcToolbox dialog and it will run with the default settings which should be fine for most situations.
The tools rely on various non-ArcGIS tools which are installed in known locations on the MapAction laptops. You won't need to worry about these for running on the laptops, but for other uses you will need to know about them.
For the change toolset, you will need java installed and osmosis:
[http://wiki.openstreetmap.org/wiki/Osmosis#Latest_stable_version]
On Windows this should be placed at C:\osmosis.
On *nix this can be anywhere, and either added to your PATH environment
variable or passed on the CLI with -o.
For the OSM2Mapaction Toolset you will need OGR / GDAL installed, with the python bindings. The easiest way to do this on Windows is to install OSGEO4W, and use the OSGEO4W command prompt rather than the normal Windows one to run the tool.
You will also need two configuration files from the OSM2Mapaction code. The osmconf.ini file tells OGR how to read the OSM data in a way that the conversion code will understand. The Excel configuration file maps all the relevant tags and values from OSM to the corresponding Mapaction DNC shapefile names.