Source-less distribution support

[BubuOT]

Currently pep517 installs fail when running from a source-less python distribution because pep517 tries to run a .py file which doesn't exist in the source-less install.

There has been some recent discussion around the support status of this with the reiterated conclusion that this is an officially supported (but somewhat discouraged) method of running cpython.

There's been an attempt to resolve this here but that hasn't gotten any feedback: #130 Would that be a sensible way to solve the problem or should another approach be taken?

The python distribution in question here is provided by the embedded buildroot distribution. (Though openwrt seems to run into the same problem as seen in the PR above),

[pfmoore]

There's been an attempt to resolve this here but that hasn't gotten any feedback: #130 Would that be a sensible way to solve the problem or should another approach be taken?

Apart from the merge conflict which would need to be addressed, this PR doesn't deal with the __pycache__ directory (and its layout) correctly.

I don't see why this is necessary, though. The in_process.py script is a special case - it's not a module to be imported, but rather a script to be run. I think that "source-less" distributions should not be trying to force scripts like this to be run from a .pyc, but should just keep the .py file in this one case.

[BubuOT]

I think that "source-less" distributions should not be trying to force scripts like this to be run from a .pyc, but should just keep the .py file in this one case.

That sounds sensible but what we have struggled with in buildroot is to discover what .py files are scripts, not to be byte-compiled. There was a discussion if those scripts should be installed to /usr/bin instead of /usr/lib/python but I don't think that would make a terrible amount of sense for the in_process.py script.

The thread for that discussion is http://lists.busybox.net/pipermail/buildroot/2022-August/648686.html, if anyone is interested,

Do you have any other ideas how to handle this in a more generic way than special casing this one file for both pep517 and pip (and any other build tool that decides to vendor pep517)?

(Interestingly a somewhat related issue has just been opened for pip pypa/pip#11406)

[pfmoore]

Do you have any other ideas how to handle this in a more generic way than special casing this one file for both pep517 and pip (and any other build tool that decides to vendor pep517)?

Nope, I think special-casing is the right thing to do here.

[takluyver]

I'm happy enough to have the special case inside pep517, as #130 implements, if that's the easiest way to do it. I just hadn't got round to looking at it (there's way more stuff to do than time in the day to do it).

[vfazio]

it does run as a module though...

root@rpi-a4fa30:~# python3 -m pip._vendor.pep517.in_process._in_process -h
Needs args: hook_name, control_dir

It's not immediately clear to me why _in_proc_script_path gets imported when it will just return to the path of the script that could otherwise be invoked via -m which, at least on the face of it, is the same as the import path

root@rpi-a4fa30:~# python3 -c "from pip._vendor.pep517.in_process._in_process import main; main();"
Needs args: hook_name, control_dir

So, either the module import path could be hardcoded and exec'd via _subprocess_runner in pep517.wrappers or _in_proc_script_path could yield the module path.

This team obviously understands the architecture of their code better than i do, but is there some expectation that the file yielded by _in_proc_script_path is outside of the python installation or otherwise does not follow the if __name__ = "__main__": main() convention?

[pfmoore]

There's no guarantee that the in-proc script is being run by the Python installation that has pep517 installed.

I suggest we stop debating theory here, and if you want this, get #130 working and test it in various sourceless environments (including some complex pip builds, which means you'll need to set up a copy of pip with the patched version vendored in) to confirm it works.

[jameshilliard]

There was a discussion if those scripts should be installed to /usr/bin instead of /usr/lib/python but I don't think that would make a terrible amount of sense for the in_process.py script.

Shouldn't you just be using entry points, our pep517 infrastructure uses installer which should support those?

Do you have any other ideas how to handle this in a more generic way than special casing this one file for both pep517 and pip (and any other build tool that decides to vendor pep517)?

I'm confused here, buildroot doesn't really support pip in general and also doesn't vendor pep517(buildroot uses normal unvendored pep517 for the pep517 build frontend).

[BubuOT]

@jameshilliard

I'm confused here, buildroot doesn't really support pip in general and also doesn't vendor pep517(buildroot uses normal unvendored pep517 for the pep517 build frontend).

This is talking about using pip on the target. (In combination with venv most likely)

[jameshilliard]

This is talking about using pip on the target.

That's not really maintained AFAIU, and is unlikely to work properly due to the lack of a target compiler.

Why are you trying to use pip on the target? I typically just sync/share a project source tree from my development system to buildroot and run the project without installing if I need to do development testing on my target system, there's other techniques as well that one can use depending on requirements.

[commodo]

Why are you trying to use pip on the target?

I'll give you the OpenWrt answer here (which may be similar to Buildroot's): at least in the OpenWrt case, since it's a distribution (on it's own) running on routers, you get users wanting to run Python on it and have some basic way of writing scripts, and if possible install (Python-only) packages via pip. Even the ability to install Python only packages on the device was enough (for most users; I suspect [1] ) . At least that was the idea back in ~2015.

I don't have an answer as to why people want to run pip on the routers, (i.e.) on systems with 64 MB of RAM, and 8-16 MB of flash, but they do. Packaging pip2 & pip3 alongside Python2/3 opens the gate to not having to maintain many packages in the OpenWrt buildsystem. Otherwise, if people want some Python package, they need to package it via OpenWrt's build-system (which can be cumbersome for your average hobbyist).
This offered, some flexibilty to hack easily on the router (which I guess is fun for a few people).

Luckily, there's no more Python2 these days, which (as a maintainers) means half-work (no more need to care about Py2 compatibility for each Python2 variant).

[1] I say that "I suspect", because over time, people never told me what they do, just what doesn't work. Silence also means "it's probably working"

Hope this helps :)
I didn't understand my users, and still don't.
But if it makes them happy ¯_(ツ)_/¯

[jameshilliard]

Even the ability to install Python only packages on the device was enough (for most users; I suspect [1] ) .

You don't really need pip for that though, you can just copy a virtualenv from a build host or use various other techniques.

Otherwise, if people want some Python package, they need to package it via OpenWrt's build-system (which can be cumbersome for your average hobbyist).

Buildroot doesn't really support openwrt style package installation in general.

[commodo]

You don't really need pip for that though, you can just copy a virtualenv from a build host or use various other techniques.

Yeah, but the average use-case for OpenWrt is:

• get an image from here: https://downloads.openwrt.org/
• flash it to the router
• boot router
• ssh to the router
• on the router:
  • opkg update
  • opkg install python3
  • opkg install python3-pip
• done

[commodo]

Yeah, but the average use-case for OpenWrt is:

I will correct this here, and say that I don't know what the average use-case for OpenWrt is (because I have no data) :)
I will say, it's a use-case (or steps) that people have used when reporting issues to Python2/3 on OpenWrt