make pyenv available in Docker #96
Comments
|
@viceice wdyt? Transfer this to the buildpack repo? |
|
Sure, move it to buildpack, pyenv should be pretty small. |
|
Ok, I've looked into pyenv. It's not clear to me if pyenv will autoinstall the required python version or do we need to install it before it is usable by pyenv? |
|
Questions:
|
|
I'm running some tests now. pushing a branch in a few minutes |
|
looks like even when pyenv is available, we need to preinstall python versions whith pyenv
|
|
@wyardley can you comment on the preinstallation requirement? I don't think it's feasible for us to bloat the full image any further by preinstalling all possible python releases people would want, so we need a way around that if one exists. @viceice maybe we need to add a "precommand" even if binarySource=global? (assuming we can detect which version of python that pipenv will want) |
|
Yes, preinstall command ca work. But we need to do a user install if we aren't root. Our python installer already skips install, if the specific version is already exists. I'll try to create a dynamic python install tool, which can be run as non root. Maybe we can adapt it for other tools later. So we can really remove the full image and let all tools install dynamic on global mode. |
|
It would be nice if we could let the tools use |
|
Pyenv doesn't install any python version automatically. You need to explicit call |
|
Looks like pipenv would try through? |
|
Not sure. We can try it, but my knowledge is very basic for python universe. I need some help to create a real test case, see linked pr. The pr fails with missing python. No message about pyenv. 🤔 |
|
I think the research/testing burden can fall on @wyardley for now :) |
|
Yeah, I was thinking this as well last nightl -- that maybe it wouldn’t handle dynamically installing it. I don’t use pyenv locally. I agree that bloating the install image with every possible version of Python is not ideal. In our case, we do run it on premise so in theory, it wouldn’t be hard for us to build on top of the base renovate image and add additional Python versions ourselves (I guess we could add pyenv ourselves too, if needed). I am less familiar with https://asdf-vm.com/, but I will play with it as well, to see if that is capable of actually installing versions of tools. |
|
Had a short look at asdf. it can install multiple tools with multiple versions without root, Maybe it can be used to preinstall our tools and also to install tools dynamically. It also uses plugins to add support for installing / handling tools. So we can easily add new tools, which are not yet supported. |
Was having the same thought. A lot of positives, though I guess potentially may be some security implications of using such a magical tool. Side question: with the new approach, there's still no way for us to set a (single) Python version that's different from the one baked into the Renovate image without building our own container, right? Even if it's not called out as a breaking change, it would be nice if the version change were noted in the release notes and / or if there were some way to influence what that version was. w/r/t the pyenv and bloating images thing - if the |
|
This will bloat too, as other people suggesting to do this for node, java and other tools too. So that isn't an option. So let me check my previous idea of allowing install python dynamically as non-root. So renovate can try to install missing tools at runtime when binarySource is global. |
|
No hurry, but just curious if any of these ideas / approaches have panned out at all? |
|
So have we exhausted the idea that pyenv can install at runtime rather than just switch version? |
|
Sounds like Edit: With pyenv, it looks like it would work if you first ran https://pipenv.pypa.io/en/latest/advanced/#automatic-python-installation mentions that pipenv can prompt to install the missing version via pyenv. Let me test to see if it can do this without confirmation if a specific option is passed. |
|
@viceice: nice! and thanks Yean, was just about to post that it seems like it will auto-install after all, at least with |
|
with @rarkins which pipenv version is used in 1.2.0 image ? Is it the pro image? |
|
Currently WhiteSource Renovate On-Premises (can I say "WSROP"?) is built off the |
|
we should wait until the latest python and dotnet issues are fixed in buildpack |
docker run --rm -it whitesource/renovate:1.2.0 bash
ubuntu@d1b0aa913826:/usr/src/app$ cd /tmp/
ubuntu@d1b0aa913826:/tmp$ curl -sSLO https://raw.githubusercontent.com/renovatebot/docker-buildpack/feat/pyenv/test/python/test/b-pipenv/Pipfile
ubuntu@d1b0aa913826:/tmp$ pipenv lock
Warning: Python 3.6 was not found on your system...
Neither 'pyenv' nor 'asdf' could be found to install Python.
You can specify specific versions of Python with:
$ pipenv --python path/to/python
ubuntu@d1b0aa913826:/tmp$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
Cloning into '/home/ubuntu/.pyenv'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (74/74), done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 18684 (delta 32), reused 14 (delta 4), pack-reused 18610
Receiving objects: 100% (18684/18684), 3.78 MiB | 5.53 MiB/s, done.
Resolving deltas: 100% (12670/12670), done.
ubuntu@d1b0aa913826:/tmp$ export export PYENV_ROOT="$HOME/.pyenv"
ubuntu@d1b0aa913826:/tmp$ export PATH="$PYENV_ROOT/bin:$PATH"
ubuntu@d1b0aa913826:/tmp$ pipenv lock
Warning: Python 3.6 was not found on your system...
Would you like us to install CPython 3.6.12 with Pyenv? [Y/n]: n
You can specify specific versions of Python with:
$ pipenv --python path/to/python
ubuntu@d1b0aa913826:/tmp$ ./node_modules/.bin/renovate --version^C
ubuntu@d1b0aa913826:/tmp$ echo "" | pipenv lock
Warning: Python 3.6 was not found on your system...
Would you like us to install CPython 3.6.12 with Pyenv? [Y/n]: Installing CPython 3.6.12 with /home/ubuntu/.pyenv/bin/pyenv (this may take a few minutes)...
✘ Failed...
Something went wrong...
Downloading Python-3.6.12.tar.xz...
-> https://www.python.org/ftp/python/3.6.12/Python-3.6.12.tar.xz
Installing Python-3.6.12...
BUILD FAILED (Ubuntu 18.04 using python-build 1.2.22-47-g4c302a02)
Inspect or clean up the working tree at /tmp/python-build.20210205090452.183
Results logged to /tmp/python-build.20210205090452.183.log
Last 10 log lines:
sys.exit(ensurepip._main())
File "/tmp/python-build.20210205090452.183/Python-3.6.12/Lib/ensurepip/__init__.py", line 204, in _main
default_pip=args.default_pip,
File "/tmp/python-build.20210205090452.183/Python-3.6.12/Lib/ensurepip/__init__.py", line 117, in _bootstrap
return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
File "/tmp/python-build.20210205090452.183/Python-3.6.12/Lib/ensurepip/__init__.py", line 27, in _run_pip
import pip._internal
zipimport.ZipImportError: can't decompress data; zlib not available
Makefile:1102: recipe for target 'install' failed
make: *** [install] Error 1
Warning: The Python you just installed is not available on your PATH, apparently. |
|
compiling on the fly should not hte way we want to go, as we already have all python versions precompiled, see https://github.com/renovatebot/python/tree/releases i think we need to be able to install those tools as non-root to home dir and allow renovate to install them on the fly in global mode |
|
Let's invent a new mode for now, e.g. |
|
It would only be valid within our buildpack-based images anyway, e.g. not when |
yes, as it will need our buildpack tools, but anyone can add the tools to a custom image and use that as base. |
|
OK, but let's still retain the meaning of |
|
ok, so new |
|
we can use |
Hrm, maybe I misunderstood? I thought the goal was for Renovate not to ship with all of these? It would be a bit more work, but renovate could (theoretically) parse the expected version in the |
|
We have precompiled in separate repo. So we can build a solution to download on the fly. We already extract the required python constraint. If you use |
|
Ok, I think I'm understanding a little better now. Will |
|
It would need a sidecar dind container. Wich can be hopefully the new rootless one. You need to share the Checkout our helm chart repo, there is an issue for this to integrate it to the chart. Maybe you can help to get that running. |
|
We also planning a new install binarySource mode, which is like the docker slim one, but downloads the required tools on the fly and running them locally |
|
So with 1.3.0, where 3.8 and 3.9 are both installed, I get a different error: We can play around with that since we're already messing around with a custom docker image, but would it be possible to get distutils installed in there? |
I'm fine with that but not experienced with it. Are you familiar enough to PR it to this repo? |
In a general sense, I understand what's needed, but not w/r/t the buildpack stuff. Maybe just adding python3-distutils here would work? docker-buildpack/src/python/buildpack/tools/python.sh Lines 42 to 49 in 6e0909f |
FROM whitesource/renovate:1.3.0
USER root
RUN install-tool golang 1.15.10
RUN install-tool golang 1.16.2
RUN apt-get update && apt-get -y install python3-distutils && apt-get clean all
USER 1000While I don't think this is the right fix in terms of this buildpack setup, the above does allow both Python 3.8 and 3.9 to work for me with latest pro image. |
|
The problem is, that other tools bring in the system python, which is not expected. eg this: docker-buildpack/src/golang/buildpack/tools/golang.sh Lines 25 to 26 in 7fe5151 So we do not explicit install any system python. so if we would install I'm currently not sure about the best way to solve this issue. 🤔 |
|
Is the 3.8.5 that ends up on the standard image not from the system one? I looked to see if there were a way to install distutils via another method and didn’t come up with one. |
|
|
Right - and that's the issue - pipenv finds it, but it doesn't have a working distutils. In practical terms, this only seems to add two dependencies currently: of course, I can see the what you're saying, as if python3 at some point weren't builtin or a transitive dependency, explicitly installing python3-distutils. If bzr isn't needed, it looks like you could potentially remove the system python3 |
|
|
|
@wyardley If you're still interested in this, please open a new issue at containerbase/buildpack repo, as this repo is now deprecated and will be archived soon |
|
Is containerbase/base#101 the duplicate/continuation of this @viceice @rarkins ? |
|
Yes, i think so |
What would you like Renovate to be able to do?
Currently, Renovate can seemingly only operate on the version of Python that it has installed globally (currently 3.9 in the 1.2.0 image). While we like to keep Renovate up to date, we have certain use cases where we need to stick with a specific Python version for one reason or another.
Since
pipenv(a supported provider) can usepyenvto get a Python version, I noticed this in the "artifact update" error:Thus, I think that if pyenv were made available in the Docker image, it might be possible to have Renovate work for non-default Python versions. Happy to hear suggestions if you have an alternate way of solving this.
Did you already have any implementation ideas?
The text was updated successfully, but these errors were encountered: