Support Python 3.7

[SamuelDudley]

Python 3.7 brings with it both memory and cpu performance improvements. We can install it without destroying the distro level install and use it for running maverick code (such as maverick-api)

Example commands:

apt install libffi-dev libbz2-dev liblzma-dev libsqlite3-dev libncurses5-dev libgdbm-dev zlib1g-dev libreadline-dev libssl-dev tk-dev build-essential libncursesw5-dev libc6-dev openssl git
mkdir /srv/maverick/software/python
cd /srv/maverick/software/python
curl -sL https://github.com/python/cpython/archive/v3.7.2.tar.gz | tar xz
cd cpython-3.7*
# Note: the next step takes ~3 minutes on the pi
./configure --prefix=/srv/maverick/software/python --enable-optimizations --enable-shared
# Note: the next step takes ~30 minutes on the pi
make -j -l 4
make install

[fnoop]

Install from git. Add base::python::python_version param to control which git branch to use.
https://github.com/python/cpython/

[fnoop]

Also base::python::maverick_python param, set to true by default. false will skip building/installing.

[fnoop]

Custom optimised python 3.7.2 installed now by default into ~/software/python

[fnoop]

Make default python, install sensible default PIPs, ensure other packages (opencv2) use/support this version of python

[fnoop]

Also need to refactor install_python_module wrapper to deal with multiple pip_provide, or else since stankovich-python module has been deprecated in favour of puppet-python, perhaps the new provider doesn't need the wrapper any more.

[fnoop]

Need to move various python services to python3.7:

• -api
• vision_landing
• visiond
  etc

[fnoop]

In maverick, -api gets started through the maverick-api@ systemd service (the @ means it supports multiple instances). This service then calls /srv/maverick/bin/api.sh, which in turn calls:
/srv/maverick/code/maverick-api/maverick_api/maverick_api.py --config-file=/srv/maverick/config/web/maverick-api.$1.conf
So the startup relies on the hashbang in maverick_api.py to determine the python interpreter, which is currently set to /usr/bin/python3. However, changing this will change it in the -api repo and for any consumer, which is probably a bad idea.

So instead, we should probably change the api.sh wrapper to specify the interpreter path, which should override the hashbang.

[fnoop]

Looks like we have to setup PYTHONPATH to reliably config either python2 or python3, it's not going to work with both. The alternative is to create a python3 wrapper that resets PYTHONPATH and then calls the actual python3, but that's not a very nice solution.
Because we control the maverick environment, we can simply declare our custom python3.7 as the single sanctioned python version and set everything up as best we can for that.

[fnoop]

gstreamer python bindings proving challenging, they always pick up system python 2.7 no matter what. homebrew formula shows a workaround:
https://github.com/Homebrew/homebrew-core/blob/master/Formula/gst-python.rb

[fnoop]

gstreamer fixed. other components in ~/software:

• realsense-sdk2: Done
• jsbsim: update to upstream repo and no python any longer
• ros: Working with python2 libs
• -api: Done
• maverick-visiond: Raised Migrate maverick-visiond to python3 #821
• vision_landing
• maverick-mavlogd: Done
• maverick-uploader: Done
• webvision: Done

[fnoop]

v4l2 module doesn't work in python3, and seems to be a dead project:
https://bugs.launchpad.net/python-v4l2/+bug/1664158

Need to find an alternative before we can migrate visiond to python3

[fnoop]

Opened separate issue #821 for visiond

[fnoop]

Raised goodrobots/vision_landing#114 to deal with migrating vision_landing to python 3.

[fnoop]

Python 3.7 now installed into /srv/maverick/software/python and made default python3 within Maverick environment. All known services migrated to python 3.7.

[SamuelDudley]

It would be great to automatically update pip after installing python by running:
python3 -m pip install --upgrade pip

[fnoop]

Added action after install to do one-off upgrade of pip

[fnoop]

Custom python3 missing bz2 support: #832

[fnoop]

Probably also want sqlite3 support:

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_sqlite3
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

[fnoop]

Python3 dependency fail on raspbian:

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libgdbm-compat-dev' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libgdbm-compat-dev
Error: /Stage[main]/Base::Python/Package[libgdbm-compat-dev]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libgdbm-compat-dev' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libgdbm-compat-dev

[fnoop]

Building python with PGO takes massively long time, particularly on slower platforms - up to 30x longer than a non-profiled build. At least try to parallelise the build and profililing:
https://stackoverflow.com/questions/49793880/compiling-python-from-source-multiple-threads-for-tests

                environment => ["LDFLAGS=-Wl,-rpath,/srv/maverick/software/python/lib", "PROFILE_TASK=-m test.regrtest --pgo -j${::processorcount}"],
                command     => "/usr/bin/make -j${::processorcount} >/srv/maverick/var/log/build/python.make.out 2>&1",

Seems to work.