BSD support

[flit]

Add BSD support to libusb-package. I'd like to make sure that libusb-package can install and work successfully from the source distribution on BSD systems, since prebuilt wheels for BSD are highly unlikely.

@cederom I don't have a BSD environment to test with. Could you help here?

One question is: what does platform.system() return for BSD, and what is the file extension for shared libraries? This will allow an entry to be added to the library extension mapping dict:

libusb-package/src/libusb_package/__init__.py

Lines 40 to 45 in 7b3a847

	_LIBRARY_MAP_EXT = {
	'Darwin': '.dylib',
	'Linux': '.so',
	'Windows': '.dll',
	}
	_LIBRARY_EXT = _LIBRARY_MAP_EXT.get(platform.system(), ".so")

[cederom]

Hello @flit :-)

Here you go :-) I am working on FreeBSD 13-STABLE with GENERIC kernel configuration (some customizations for my hardware added):

uname -a
FreeBSD 0xCFMX4 13.0-STABLE FreeBSD 13.0-STABLE #0 stable/13-n247979-3e322ded35f: Thu Nov  4 12:28:58 CET 2021     root@0xCFMX4:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

There are some dependencies that needs to be installed prior build / install (only setuptools and setuptools_scm?). I have provided initial requirements.txt file in #2. Is there any other / new way to detect and install python dependencies?

Without dependencies I got:

(venv38zephyr) python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    import setuptools_scm  # noqa: F401
ModuleNotFoundError: No module named 'setuptools_scm'

With dependencies installed:

(venv38zephyr) python setup.py package
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'package'


(venv38zephyr) python setup.py develop
running develop
running egg_info
creating src/libusb_package.egg-info
writing src/libusb_package.egg-info/PKG-INFO
writing dependency_links to src/libusb_package.egg-info/dependency_links.txt
writing requirements to src/libusb_package.egg-info/requires.txt
writing top-level names to src/libusb_package.egg-info/top_level.txt
writing manifest file 'src/libusb_package.egg-info/SOURCES.txt'
warning: the 'license_file' option is deprecated, use 'license_files' instead
adding license file 'LICENSE' (matched pattern 'LICENSE')
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no directories found matching 'src/libusb'
writing manifest file 'src/libusb_package.egg-info/SOURCES.txt'
running build_ext
build_temp = /XXX/libusb-package.git/src/libusb
build_lib = /XXX/libusb-package.git/src
bash /XXX/libusb-package.git/src/libusb/bootstrap.sh
bash: /XXX/libusb-package.git/src/libusb/bootstrap.sh: No such file or directory
Error while building libusb: command 'bash' failed with exit status 127
Ignoring build failure and creating system-only libusb-package
Creating /XXX/libusb-package.egg-link (link to src)
Adding libusb-package 1.0.24.1 to easy-install.pth file

Installed /XXX/libusb-package.git/src
Processing dependencies for libusb-package==1.0.24.1
Finished processing dependencies for libusb-package==1.0.24.1

(venv38zephyr) python
Python 3.8.12 (default, Oct 14 2021, 01:14:41)
[Clang 11.0.1 ([email protected]:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import libusb-package
  File "<stdin>", line 1
    import libusb-package
                 ^
SyntaxError: invalid syntax

>>> import libusb_package
>>> dir(libusb_package)
['Any', 'Optional', 'TYPE_CHECKING', '_LIBRARY_EXT', '_LIBRARY_MAP_EXT', '_LIBRARY_NAME', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version', 'atexit', 'ctypes', 'find', 'find_library', 'functools', 'get_library_path', 'get_libusb1_backend', 'platform', 'resources', 'sys']

There seems no python setup.py package target.

This error is a bit worrying during python setup.py develop (development debug capable install):

bash: /XXX/libusb-package.git/src/libusb/bootstrap.sh: No such file or directory
Error while building libusb: command 'bash' failed with exit status 127
Ignoring build failure and creating system-only libusb-package

Is there any test I could run to verify created package operations?

Please note that FreeBSD that I work on provides its own LibUSB in the base OS install (0.x and 1.x compatible).

On NetBSD and OpenBSD there is a dedicated package for LibUSB.

Also please note that there is already pyusb package that wraps LibUSB for Python. Here is BSD related thread: pyusb/pyusb#362

On BSD systems static libraries use *.a and dynamic libraries use *.so file extensions, see built-in libusb as example:

ls /usr/lib/libusb
libusb_p.a      libusb.a        libusb.so@      libusb.so.3     libusbhid_p.a   libusbhid.a     libusbhid.so@   libusbhid.so.4

Please let me know if you need any other information / feedback / testing :-)

[cederom]

I have read the project description:

This Python package functions as an installation vehicle for libusb shared libraries, to simplify installation of tools that require libusb. The main use case is so that users don't have to install libusb manually for projects that use pyusb.

On FreeBSD there is a BSD licensed LibUSB (created by @hselasky) already pre-installed with the base OS, so no need to install from package, its here already with the OS.

Maybe on OpenBSD and NetBSD this package needs to be added by hand. But these are system provided ports / packages.. and these must only be installed by root.

Anyways on BSD local user packages should not touch anything system wide, this is not allowed and unacceptable. Any local python package should be installed in local user created virtual environment. Any custom dynamic library should be placed in local application bundle. Local means nothing outside /usr/home/user / /home/user usually that is /home/user/.local or application directory.

On BSD only root can install system packages and/or modify system. Anything else is not allowed. I am not sure if this package is feasible here.

Do I miss anything? :-)

[cederom]

(venv38zephyr) python
Python 3.8.12 (default, Oct 14 2021, 01:14:41)
[Clang 11.0.1 ([email protected]:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.system()
'FreeBSD'

Please note that there may be different flavors of BSD out there, mostly OpenBSD and NetBSD that are different OS, others are FreeBSD derivatives in most cases :-)

[cederom]

Even though there was some build error your package seems to work on FreeBSD:

(venv38zephyr) python
Python 3.8.12 (default, Oct 14 2021, 01:14:41)
[Clang 11.0.1 ([email protected]:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import libusb_package
>>>
>>> for dev in libusb_package.find(find_all=True):
...     print(dev)
...

DEVICE ID 1050:0111 on Bus 000 Address 015 =================
...
DEVICE ID 0bda:0411 on Bus 000 Address 014 =================
...
...


>>> import libusb_package
>>> import usb.core
>>> import usb.backend.libusb1
>>>
>>> libusb1_backend = usb.backend.libusb1.get_backend(find_library=libusb_package.find_library)
>>> # -> calls usb.libloader.load_locate_library(
>>> #                ('usb-1.0', 'libusb-1.0', 'usb'),
>>> #                'cygusb-1.0.dll', 'Libusb 1',
>>> #                win_cls=win_cls,
>>> #                find_library=find_library, check_symbols=('libusb_init',))
>>> #
>>> # -> calls find_library(candidate) with candidate in ('usb-1.0', 'libusb-1.0', 'usb')
>>> #   returns lib name or path (as appropriate for OS) if matching lib is found
>>>
>>> # It would also be possible to pass the output of libusb_package.get_libsusb1_backend()
>>> # to the backend parameter here. In fact, that function is simply a shorthand for the line
>>> # above.
>>> print(list(usb.core.find(find_all=True, backend=libusb1_backend)))
[<DEVICE ID 1050:0111 on Bus 000 Address 015>, <DEVICE ID 0bda:0411 on Bus 000 Address 014>, <DEVICE ID 0bda:0411 on Bus 000 Address 013>, <DEVICE ID 04dd:9761 on Bus 000 Address 012>, <DEVICE ID 1199:9041 on Bus 000 Address 011>, <DEVICE ID 5986:054c on Bus 000 Address 010>, <DEVICE ID 8087:0a2a on Bus 000 Address 009>, <DEVICE ID 04b5:0680 on Bus 000 Address 008>, <DEVICE ID 0bda:5411 on Bus 000 Address 007>, <DEVICE ID 046d:c52b on Bus 000 Address 006>, <DEVICE ID 041e:3040 on Bus 000 Address 005>, <DEVICE ID 8087:8001 on Bus 000 Address 004>, <DEVICE ID 0bda:5411 on Bus 000 Address 003>, <DEVICE ID 0000:0000 on Bus 000 Address 002>, <DEVICE ID 0000:0000 on Bus 000 Address 001>, <DEVICE ID 0000:0000 on Bus 000 Address 001>]

[flit]

Hi @cederom, apologies it took me so long to get back to this!

Thanks so much for testing! 🙏🏽

The build error (src/libusb/bootstrap.sh not found) is almost certainly caused by src/libusb being a git submodule. If you check out submodules (git submodule update --init) and run pip install . it should get past this error. (I'll add a note to the readme.)

The reason your test worked even though the build failed is that if the build if libusb fails when installing libusb-package, the install of libusb-package will still succeed. It just won't have a libusb shared library, and will always fall back to the system-provided libusb (if there is one).

System libusb and the point of libusb-package

The use case for libusb-package is that when a project like pyocd uses pyusb, there is normally no way to guarantee a known good version of libusb on all OSes. Users have to take an extra step of installing libusb before they can use pyocd.

Like the BSDs, Linux also normally comes with libusb installed. However, the system-provided version of libusb may be older and will likely change depending on the OS version. Using the libusb that comes with libusb-package, you know which version of libusb you'll be getting, and thus the limitations and bugs.

pyusb is the libusb wrapper that libusb-package is primarily designed to work with. libusb-package doesn't itself contain a Python wrapper around libusb—it's just the shared library and a few functions to conveniently get its path or create a pyusb IBackend instance.

The libusb shared library is contained within the Python package, and is not installed anywhere system-wide. So it's only used by Python tools that explicitly want to use it.

Installing and building

Running setup.py isn't supported any more for Python packages that use the (relatively) new PEP 517 style packaging (pyproject.toml). The only supported method is using a standards-compliant tool like pip, e.g. pip install . or pip install -e .. Running setup.py directly isn't guaranteed to work any more. When you install with pip, the build dependencies such as setuptools_scm are installed automatically (they are listed in pyproject.toml).

Similarly, the way to build either source distributions or wheels has changed now. You have to use the build package (install with pip). For example python -m build will build an sdist from the current directory, then build a wheel from the sdist, all done in an isolated virtual env.

Testing

The test.py script in the repo root can be run after libusb-package is installed in