autorino is a tool for Assisted Unloading, Treatment & Organisation of RINex Observations 🛰️ 🌐 🦏
Version 0.1.0 / 2024-05-29, README Revision: 2024-05-29
Authors & Contributors:
- Pierre Sakic (IPGP-OVS, Paris, France)
- Patrice Boissier (OVPF-IPGP, La Réunion, France)
- Jean-Marie Saurel (IPGP-OVS, Paris, France)
- Cyprien Griot (OVPF-IPGP, La Réunion, France)
- Diane Pacaud (OVPF-IPGP, La Réunion, France)
- Aurélie Panetier (IPGP, Paris, France)
Contact e-mail: [email protected]
Licence: GNU GPL v3 (see attached license file)
The autorino package (for Assisted Unloading, Treatment & Organisation of RINex Observations) is designed for automated download and conversion of GNSS raw data from the main manufacturers’ receivers (Leica, Septentrio, Topcon, Trimble, and BINEX) based on their respective official conversion utilities. A special focus is put on conversion to RINEX3/4 and near real-time capability (download frequency up to 5 min).
autorino aims to perfom the four following tasks:
- Download GNSS raw data from remote GNSS receivers
- Conversion of GNSS raw data to RINEX3/4 format using official conversion utilities from the main GNSS manufacturers.
- Handle RINEX files (e.g., merging, splitting, splicing, etc.) using dedicated utilities.
- Metadata edition of RINEX files (e.g., modifying header metadata, renaming filenames, editing comments, etc.) based on autorino's spinoff tool rinexmod.
For the time being, the package is not available on PyPi, so you need to install it by adding the path of the autorino
package to your $PYTHONPATH
in your .bashrc.
PYTHONPATH=$PYTHONPATH:/home/user/path_to/autorino
If you want to use autorino in CLI mode, you must also add the path to the autorino script to your $PATH
in your .bashrc.
$PATH=$PATH:/home/user/path_to/autorino/autorino
Note that it is autorino/autorino
two times, since the CLI programs are inside the autorino's module.
autorino relies on several external dependencies. Be sure to have them installed on your system using
pip install -r /home/user/path_to/autorino/requirements.txt
You need to set up the environment variable $AUTORINO_ENV
to point to the autorino's configuration file.
In your .bashrc or .bash_profile file, add the following lines:
export AUTORINO_ENV="/home/user/path_to/autorino/configfiles/env/autorino_env.yml"
This configuration file is a YAML file that contains the paths to the different GNSS raw data converters, described in the next section below.
If $AUTORINO_ENV
is not set, autorino will use the default configuration file located in the package's configfiles/env/
folder.
Per defaults values assume that the converter executables are known by your system and (e.g. set in your $PATH
).
To properly use the autorino package, you need to install the official GNSS raw data converters from the different GNSS manufacturers websites. You can find the official converters here:
converter here: mdb2rinex
Go in: Products & Services > Downloads > GNSS Products > GRxx receiver > Tools > MDB to RINEX Converter for LINUX.
see IGSMAIL-8341 for more details.
converter here: sbf2rin
converter here: tps2rin
autorino will emulate it with wine. Be sure to have wine
installed on your computer. Detailled precedure will be added soon.
converter here: convbin
convbin is part of the RTKLIB package. You can install it from the RTKLIB (explorer version) github repository.
Detailled procedure will be added soon.
Ask Trimble support for the official Linux converter t0xConverter.
converter here: trm2rinex-docker
This docker image is a wrapper around Trimble's official converter trm2rinex which is not available for Linux.
A dedicated README file trm2rinex_readme.md
details the installation and usage of this docker image.
It relies on Trimble's official converter for Windows ConvertToRinex
available
here & there.
for legacy RINEX2 conversion with teqc converter here: runpkr00
for legacy RINEX2 conversion with the well-known but discontinued UNAVCO's teqc software converter here: teqc
You might also need RINEX handeling software:
NB: GFZRNX usage is not allowed in routine mode without a proper commercial license. Be sure to comply with it.
Once the converters are installed, you need to set the converter paths in the autorino's env
configuration file.
You must have set the $AUTORINO_ENV
environment variable to point to the env
configuration file.
(see dedicated section above)
To configure the external utilities, in the you can:
- set the full executable's paths to the in the
env
configuration file - set the paths in your
$PATH
environment variable, and then simply set the executable's names in theenv
configuration file.
The most basic and common operation that autorino can perform is to convert some RAW files to RINEX.
import autorino.common as arocmn
import glob
### Find all BINEX files in a folder
p = "/home/user/where_your/raw_data/are_stored/"
l = glob.glob(p,"*BNX")
### Define the output folder
out_dir = "/home/user/where_your/rinex_data/will_be_saved/"
tmp_dir = out_dir
### Call the conversion function
arocmn.convert_rnx(l,out_dir,tmp_dir)
python3 autorino_convert_rnx.py --force --metadata /home/user/path/of/your/sitelogs --out_dir_structure '<SITE_ID4>/%Y' --list_file_input /home/user/where_your/raw_data/are_stored/raw_data_list.txt /home/user/where_your/rinex_data/will_be_saved/```
usage: autorino_convert_rnx.py [-h] [-l] [-s OUT_DIR_STRUCTURE] [-tmp TMP_DIR]
[-log LOG_DIR] [-rnmo RINEXMOD_OPTIONS]
[-m METADATA] [-f]
raws_inp [raws_inp ...] out_dir
Convert RAW files to RINEX.
positional arguments:
raws_inp The input RAW files to be convertedPossible inputs
are: * one single RAW file path * a list of RAW path *
a text file containing a list of RAW paths (then
--list_file_input must be activated) * a directory
containing RAW files
out_dir The output directory where the converted files will be
stored
options:
-h, --help show this help message and exit
-l, --list_file_input
If set to True, the input RAW files are provided as a
list in a text file
-s OUT_DIR_STRUCTURE, --out_dir_structure OUT_DIR_STRUCTURE
The structure of the output directory.If provided, the
converted files will be stored in a subdirectory of
out_dir following this structure.See README.md for
more information.Typical values are '<SITE_ID4>/%Y/'
or '%Y/%j/
-tmp TMP_DIR, --tmp_dir TMP_DIR
The temporary directory used during the conversion
process
-log LOG_DIR, --log_dir LOG_DIR
The directory where logs will be stored. If not
provided, it defaults to tmp_dir
-rnmo RINEXMOD_OPTIONS, --rinexmod_options RINEXMOD_OPTIONS
The options for modifying the RINEX files during the
conversion
-m METADATA, --metadata METADATA
The metadata to be included in the converted RINEX
files. Possible inputs are: * list of string (sitelog
file paths), * single string (single sitelog file
path), * single string (directory containing the
sitelogs), * list of MetaData objects, * single
MetaData object
-f, --force Force the conversion even if the output files already
exist
autorino relies on YAML configuration files to perform its workflow (i.e. set of steps) operations.
Described in the Setting up the environment and Setting up external utilities sections above.
It contains the site-specific information and configuration under the main station
block.
Then, it is organized in different sub-blocks:
site
: site's name, its coordinates, and metadatadevice
: manually define the device's (receiver and antenna) characteristics if metadata are not provided insite
blockaccess
: connexion protocols, login and password etc...sessions
: define the record sessions characteristics. Each session is in a sub-block with its own parameters.session_<sessionname>
: define one record session characteristics. The session block name is, by convention,session_<sessionname>
, where<sessionname>
is also the first session'sgeneral
sub-block attribute'sname
.general
: name, data freqency, temporary and log directories of the sessionepoch_range
: the epoch range of the step i.e. a start epoch (epoch1
), an end epoch (epoch2
), andperiod
(step) of the session to download the data.epoch1
andepoch2
can be relative epochs in human readable sentences. (interpretation done with the dateparser package).steps
: list of the steps constituting the workflow. Each step is a sub-block with its own parameters.
The different possible steps are:
download
: download the dataconvert
: convert the datasplice
: splice (concatenate) the datasplit
: split the data
A step has the following generic structure:
active
: a boolean (True
orFalse
) to activate or deactivate the stepinp_dir_parent
: the parent directory of the input filesinp_structure
: the filename structure of the input filesout_dir_parent
: the parent directory of the output filesout_structure
: the filename structure of the output filesepoch_range
: the epoch range of the step. (see above)options
: a dictionary of options specific to the step
It contains the "standard" configuration for an autorino workflow,
i.e. all the parameters, paths, options ... which are the same for all stations.
main values can be called and used in the sites configuration files using the alias FROM_MAIN
(see below).
To use generic or variables values in the configuration files, you can use aliases.
Aliases take the form of <aliasname>
or <ALIASNAME>
with < >
. Alias are case-sensitive:
the first version is lower-case and the latter is upper-case.
The following aliases are managed:
<site_id4>
or<SITE_ID4>
: the 4-characters site name<site_id9>
or<SITE_ID9>
: the 9-characters site name
Time aliases can also be used. They follow the date
format convention,
e.g. %Y
for the year, %H
for the hour, %j
for the day of year, etc...
The environment variables can also be used as aliases. They follow the <$ENVVAR>
convention,
using $
and between <
& >
, e.g. <$HOME>
for the home directory.
Per default values can be called and used in the configuration files using the alias:
FROM_MAIN
. Then, the value is taken from the main configuration file, see aboveFROM_SESSION
. Then, the value is taken from thesession
block
autorino is based on a main parent class: StepGnss
.
It performs generic actions on input files, saving them in an output folder.
StepGnss
has three daughter classes:
DownloadGnss
: for downloading a RAW file to the local serverConvertGnss
: for RAW > RINEX conversionHandleGnss
: to perform decimation, spliting or splicing operations on a RINEX. It has two daughter classes:SplitGnss
: to split a RINEX fileSpliceGnss
: to splice (concatenate) RINEX files
The central attribute of a StepGnss
object is its table (step_gnss.table
).
This is a pandas' DataFrame that lists, among other things, the input files, and, where applicable, the output files if the operation has been successful.