Skip to content

Base components for Windows build

Jump to bottom
Robert Oeffner edited this page May 8, 2018 · 13 revisions

Bundling base components as zip files for the Windows build

Introduction

This document details how to create zip file bundles of python, the HDF5 and the LibTiff libraries used for the bootstrap build on Windows of CCTBX as described here and other software relying on CCTBX such as Phenix or Dials. Unless you want to replace the Python, the HDF5 or the LibTiff bundle described here with your own versions you do not need to read this document. The bootstrap build will just download existing bundles from cci.lbl.gov.

The base components are bundled as zip files to be downloaded when building for Windows. This is possible because binary compatibility of executables with Windows has been retained over successive versions of Windows apart from the distinction between the 32 and 64 bit platforms. Therefore these base components are necessary to build one time only. Zipped up bundles of these builds are then stored on http://cci.lbl.gov/cctbx_dependencies from where bootstrap.py will download them during the base stage of the build. This happens around lines 136 - 171 in cctbx_project\libtbx\auto_build\install_base_packages.py, as of 12 April 2017, git SHA-1: 03e6a64c... The bootstrap build will uncompress the python bundle into base\bin\python and the HDF5 bundle into base\HDF5-1.8.16 relative to where the bootstrap build is done. Unlike the directory structure on Unix the respective library and header files of these two bundles are stored as sub directories within these directories and not in common base\include and base\lib directories.

Python 2.7

The first base component bundled is Python 2.7.15 with installed modules such as matplotlib, numpy, biopython, wxPython, etc. Version 2.7.9 is the minimum version that provides tools like "pip install" which makes it much easier to enhance python with 3rd party modules from https://pypi.python.org.

Creating the Python bundle

Download and install python2.7.15 to your PC. Unfortunately the Python for Windows installer is too clever for its own good. It will try to overwrite any existing Python installation on your PC as well as the registry key that specifies the existing installation, HKEY_CURRENT_USER\Software\Python\PythonCore\2.7\InstallPath

To deal with this the easiest thing may be to either uninstall the existing python or to make a backup of the registry key

HKEY_CURRENT_USER\Software\Python

and then delete it. After that, install Python into a folder containing no blank spaces in its name.

Warning

Entering wrong values in the registry or accidentally deleting values may render Windows bricked. Always ensure you have a recent "System Restore" point in your Windows backup. This will allow you to boot Windows to "Last known good configuration" if the registry gets corrupted.

Next we need to install wxPython package for the GUI. This is available as a binary installer from http://www.wxpython.org. Get version 3.0.2 and install it into your python installation. It should default to live in <your python folder>\\Lib\\site-packages.

The list of additional 3rd party Python modules and their versions necessary for the Python for Windows distributed with CCTBX is:

alabaster==0.7.10
apipkg==1.4
attrs==17.4.0
Babel==2.5.3
backports.functools-lru-cache==1.5
biopython==1.71
certifi==2018.4.16
chardet==3.0.4
colorama==0.3.9
comtypes==1.1.4
cycler==0.10.0
Cython==0.28.2
docutils==0.14
execnet==1.5.0
freeze==1.0.10
funcsigs==1.0.2
future==0.16.0
h5py==2.7.1
idna==2.6
imagesize==1.0.0
Jinja2==2.10
kiwisolver==1.0.1
libsvm==3.22
MarkupSafe==1.0
matplotlib==2.0.0
mock==2.0.0
more-itertools==4.1.0
numpy==1.14.2
olefile==0.45.1
orderedset==2.0.1
packaging==17.1
pbr==4.0.2
PIL==2.0+dummy
Pillow==5.1.0
pluggy==0.6.0
procrunner==0.6.1
psutil==5.4.5
py==1.5.3
Pygments==2.2.0
PyOpenGL==3.1.0
pyparsing==2.2.0
PyRTF==0.45
pytest==3.5.0
python-dateutil==2.7.2
pytz==2018.4
reportlab==3.4.0
requests==2.18.4
scikit-learn==0.19.1
scipy==1.0.1
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.7.2
sphinxcontrib-websupport==1.0.1
tabulate==0.8.2
typing==3.6.4
urllib3==1.22

These can be added to your Python installation as follows: In a command prompt change directory to the toplevel directory of your python installation and type:

python -m pip install alabaster==0.7.10 apipkg==1.4 attrs==17.4.0 Babel==2.5.3 backports.functools-lru-cache==1.5 biopython==1.71 certifi==2018.4.16 chardet==3.0.4 colorama==0.3.9 comtypes==1.1.4 cycler==0.10.0 Cython==0.28.2 docutils==0.14 execnet==1.5.0 freeze==1.0.10 funcsigs==1.0.2 future==0.16.0 h5py==2.7.1 idna==2.6 imagesize==1.0.0 Jinja2==2.10 kiwisolver==1.0.1 MarkupSafe==1.0 matplotlib==2.0.0 mock==2.0.0 more-itertools==4.1.0 numpy==1.14.2 olefile==0.45.1 orderedset==2.0.1 packaging==17.1 pbr==4.0.2 Pillow==5.1.0 pluggy==0.6.0 procrunner==0.6.1 psutil==5.4.5 py==1.5.3 Pygments==2.2.0 PyOpenGL==3.1.0 pyparsing==2.2.0 pytest==3.5.0 python-dateutil==2.7.2 pytz==2018.4 reportlab==3.4.0 requests==2.18.4 scikit-learn==0.19.1 scipy==1.0.1 six==1.11.0 snowballstemmer==1.2.1 Sphinx==1.7.2 sphinxcontrib-websupport==1.0.1 tabulate==0.8.2 typing==3.6.4 urllib3==1.22

In the above command line the modules PIL-2.0+dummy-py2.py3-none-any.whl, PyRTF-0.45-py2-none-any.whl, libsvm-3.22-cp27-cp27m-win_amd64.whl have been omitted as they do not exists on pypi.python.org. Get them from Christoph Gohlke's site. Make sure that the python version and compiler version of the modules matches 2.7 and Visual Studio 2008 respectively. Then install them likewise as:

python -m pip install ..\3rdPartyModules\PIL-2.0+dummy-py2.py3-none-any.whl
python -m pip install ..\3rdPartyModules\PyRTF-0.45-py2-none-any.whl
python -m pip install ..\3rdPartyModules\libsvm-3.22-cp27-cp27m-win_amd64.whl

Having added all these modules to your python we now delete all .pyc files to reduce file size of the of the compressed bundle. Do this from a command prompt like:

del /S /Q *.pyc

Now your customised python is ready to be bundled into a zip file. From a command prompt change directory to just above the python directory. Then execute the command:

powershell Compress-Archive -Path Python -DestinationPath Python2.7.15_x86_64_plus_relocatable.zip

which creates the zip file, python2.7.15_x86_64_plus_relocatable.zip with the python folder. If you want to use make this zip bundle available to others upload it to http://cci.lbl.gov/cctbx_dependencies and edit the WIN64PYTHON_PKG variable in cctbx_project\libtbx\auto_build\package_defs.py accordingly. When a boostrap build is started bootstrap.py will download it during the base stage and uncompress it in the base subdirectory of your build location.

HDF5 library

The second base component bundled is the HDF5 library which is necesary for Dials and Xia2. On Windows the build tool CMake from https://cmake.org/ must be installed first.

On this occassion it seems to be necessary to have the full Visual Studio 2008 installed. The free version of the compiler in the Windows 7 SDK seems to confuse CMake if other and newer Visual Studio compilers are present. This leads to error messages when attempting to build the library.

HDF5 can be downloaded from http://www.hdfgroup.org/ftp/HDF5/current/src/ such as a zipped file like CMake-hdf5-1.8.16.zip. Uncompress it on your PC.

Adapting build scripts to Visual Studio 2008

In the folder CMake-hdf5-1.8.16 there should be cmd scripts such as build-VS2012-32.bat, build-VS2013-32.bat etc. but none targetting Visual Studio 2008, i.e. with "VS2008" as part of the filename. We need to adapt HDF5 to build with Visual Studio 2008. To do this open one of the existing cmd scripts such as build-VS2012-32.bat and replace the line

ctest -S HDF518config.cmake,BUILD_GENERATOR=VS2012 -C Release -V -O hdf518.log

with

ctest -S HDF518config.cmake,BUILD_GENERATOR=VS2008 -C Release -V -O hdf518.log

Save this file with the name "build-VS2008-32.bat". Now open the file "HDF518config.cmake". Locate the section:

# build generator must be defined
if(NOT DEFINED BUILD_GENERATOR)
  message(FATAL_ERROR "BUILD_GENERATOR must be defined - Unix, VS200864, VS2013, VS201364, VS2012, or VS201264")
else()
  if(${BUILD_GENERATOR} STREQUAL "Unix")
    set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201564")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015 Win64")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013 Win64")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012 Win64")
  else()
    message(FATAL_ERROR "Invalid BUILD_GENERATOR must be - Unix, VS2013, VS201364, VS2012, or VS201264")
  endif()
endif()

and add entries for VS2008 like:

# build generator must be defined
if(NOT DEFINED BUILD_GENERATOR)
  message(FATAL_ERROR "BUILD_GENERATOR must be defined - Unix, VS2008, VS200864, VS2013, VS201364, VS2012, or VS201264")
else()
  if(${BUILD_GENERATOR} STREQUAL "Unix")
    set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201564")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015 Win64")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013 Win64")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012 Win64")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2008")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 9 2008")
  elseif(${BUILD_GENERATOR} STREQUAL "VS200864")
    set(CTEST_CMAKE_GENERATOR "Visual Studio 9 2008 Win64")
  else()
    message(FATAL_ERROR "Invalid BUILD_GENERATOR must be - Unix, VS2013, VS201364, VS2012, or VS201264")
  endif()
endif()

Now locate the section:

#########       Following describes compiler           ############
if(WIN32)
  set(SITE_OS_NAME "Windows")
  set(SITE_OS_VERSION "WIN7")
  if(${BUILD_GENERATOR} STREQUAL "VS201564")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2015")
    set(SITE_COMPILER_VERSION "14")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2015")
    set(SITE_COMPILER_VERSION "14")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2013")
    set(SITE_COMPILER_VERSION "12")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2013")
    set(SITE_COMPILER_VERSION "12")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2012")
    set(SITE_COMPILER_VERSION "11")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2012")
    set(SITE_COMPILER_VERSION "11")
  endif()
##  Set the following to unique id your computer  ##

and add entries for the VS2008 compiler like:

#########       Following describes compiler           ############
if(WIN32)
  set(SITE_OS_NAME "Windows")
  set(SITE_OS_VERSION "WIN7")
  if(${BUILD_GENERATOR} STREQUAL "VS201564")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2015")
    set(SITE_COMPILER_VERSION "14")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2015")
    set(SITE_COMPILER_VERSION "14")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2013")
    set(SITE_COMPILER_VERSION "12")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2013")
    set(SITE_COMPILER_VERSION "12")
  elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2012")
    set(SITE_COMPILER_VERSION "11")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2012")
    set(SITE_COMPILER_VERSION "11")
  elseif(${BUILD_GENERATOR} STREQUAL "VS200864")
    set(SITE_OS_BITS "64")
    set(SITE_COMPILER_NAME "vs2008")
    set(SITE_COMPILER_VERSION "9")
  elseif(${BUILD_GENERATOR} STREQUAL "VS2008")
    set(SITE_OS_BITS "32")
    set(SITE_COMPILER_NAME "vs2008")
    set(SITE_COMPILER_VERSION "9")
  endif()
##  Set the following to unique id your computer  ##

Finally due to licensing terms of the optionally compiled SZIP library as detailed here: https://www.hdfgroup.org/doc_resource/SZIP/ this needs to be excluded from builds. To do that find the lines:

### disable ext libs building
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")

and uncomment them:

### disable ext libs building
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")

Doing the actual build

The configuration is now good to build. Open a command prompt and change directory to the CMake-hdf5-1.8.16 folder with the amended build script files. If according to your platform XX is either 32 or 64 initialise the compilers environment variables by running:

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvarsXX.bat

CMake installed earlier may not be in the executable path in which case you can add it temporarily to your command prompt by entering:

set PATH=%PATH%;C:\Program Files\CMake\bin

Then run the build script by entering:

build-VS2008-XX.bat

This should build the library, bundle it into a compressed zip file located in CMake-hdf5-1.8.16\build and named HDF5-1.8.16-win32.zip or HDF5-1.8.16-win64.zip depending on the platform. As with the Python bundles these files should be uploaded to http://cci.lbl.gov/cctbx_dependencies so that bootstrap.py may fetch it during the base stage of the build.

LibTiff library

Libtiff from http://www.simplesystems.org/libtiff/ needs to be build and zipped up. Get the source file tiff-4.0.6.zip from the download site and unpack it. Assuming the compiler environment has been set with C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvarsXX.bat where XX is either 32 or 64. The library can be compiled with the command nmake /f makefile.vc. The source directory tiff-4.0.6\libtiff will then be populated with object and lib files. Only the files with the extension .h, .lib, .dll are needed. Copy these over to an empty directory named libtiff. Compress this directory to the name libtiff4.0.6xXX.zip. This file should be uploaded to http://cci.lbl.gov/cctbx_dependencies so that bootstrap.py may fetch it during the base stage of the build.

Clone this wiki locally