Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update jupyter-tensorflow-full rock for 1.8 #49

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

NohaIhab
Copy link
Contributor

@NohaIhab NohaIhab commented Sep 21, 2023

Update the jupyter-tensorflow-full rock for 1.8, based on the upstream image

Summary of changes

  • change the branch to v1.8-branch
  • bump the rock version to 1.8
  • pin pyjuju to <4.0 in the unit test requirements due to migration to juju 3 in 1.8
  • update kubectl to 1.25

Testing

Note: the rock was built before running the test

tox -e unit
unit installed: anyio==4.0.0,asttokens==2.4.0,attrs==23.1.0,backcall==0.2.0,bcrypt==4.0.1,cachetools==5.3.1,certifi==2023.7.22,cffi==1.15.1,charmed-kubeflow-chisme==0.2.0,charset-normalizer==3.2.0,cryptography==41.0.4,decorator==5.1.1,deepdiff==6.2.1,exceptiongroup==1.1.3,executing==1.2.0,google-auth==2.23.0,h11==0.14.0,httpcore==0.18.0,httpx==0.25.0,hvac==1.2.1,idna==3.4,importlib-resources==6.1.0,iniconfig==2.0.0,ipdb==0.13.13,ipython==8.12.2,jedi==0.19.0,Jinja2==3.1.2,jsonschema==4.17.3,juju==3.2.2,kubernetes==28.1.0,lightkube==0.14.0,lightkube-models==1.28.1.4,macaroonbakery==1.3.1,MarkupSafe==2.1.3,matplotlib-inline==0.1.6,mypy-extensions==1.0.0,oauthlib==3.2.2,ops==2.6.0,ordered-set==4.1.0,packaging==23.1,paramiko==2.12.0,parso==0.8.3,pexpect==4.8.0,pickleshare==0.7.5,pkgutil-resolve-name==1.3.10,pluggy==1.3.0,prompt-toolkit==3.0.39,protobuf==3.20.3,ptyprocess==0.7.0,pure-eval==0.2.2,pyasn1==0.5.0,pyasn1-modules==0.3.0,pycparser==2.21,Pygments==2.16.1,pyhcl==0.4.5,pymacaroons==0.13.0,PyNaCl==1.5.0,pyRFC3339==1.1,pyrsistent==0.19.3,pytest==7.4.2,pytest-asyncio==0.21.1,pytest-operator==0.29.0,python-dateutil==2.8.2,pytz==2023.3.post1,PyYAML==6.0.1,requests==2.31.0,requests-oauthlib==1.3.1,rsa==4.9,ruamel.yaml==0.17.32,ruamel.yaml.clib==0.2.7,serialized-data-interface==0.7.0,six==1.16.0,sniffio==1.3.0,stack-data==0.6.2,tenacity==8.2.3,tomli==2.0.1,toposort==1.10,traitlets==5.10.0,typing-extensions==4.8.0,typing-inspect==0.9.0,urllib3==1.26.16,wcwidth==0.2.6,websocket-client==1.6.3,websockets==8.1,zipp==3.17.0
unit run-test-pre: PYTHONHASHSEED='4273105998'
unit run-test: commands[0] | pytest -v --tb native --show-capture=all --log-cli-level=INFO /home/ubuntu/kubeflow-rocks/jupyter-tensorflow-full/tests
====================================================================================== test session starts ======================================================================================
platform linux -- Python 3.8.10, pytest-7.4.2, pluggy-1.3.0 -- /home/ubuntu/kubeflow-rocks/jupyter-tensorflow-full/.tox/unit/bin/python
cachedir: .tox/unit/.pytest_cache
rootdir: /home/ubuntu/kubeflow-rocks/jupyter-tensorflow-full
plugins: anyio-4.0.0, asyncio-0.21.1, operator-0.29.0
asyncio: mode=strict
collected 1 item                                                                                                                                                                                

tests/test_rock.py::test_rock 
---------------------------------------------------------------------------------------- live log setup -----------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:647 Adding model microk8s-localhost:test-rock-e1rg on cloud microk8s
PASSED                                                                                                                                                                                    [100%]
--------------------------------------------------------------------------------------- live log teardown ---------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:783 Model status:

Model           Controller          Cloud/Region        Version  SLA          Timestamp
test-rock-e1rg  microk8s-localhost  microk8s/localhost  3.1.5    unsupported  09:49:55Z

INFO     pytest_operator.plugin:plugin.py:789 Juju error logs:


INFO     pytest_operator.plugin:plugin.py:877 Resetting model test-rock-e1rg...
INFO     pytest_operator.plugin:plugin.py:882 Not waiting on reset to complete.
INFO     pytest_operator.plugin:plugin.py:855 Forgetting main...


====================================================================================== 1 passed in 10.21s =======================================================================================
unit run-test: commands[1] | python /home/ubuntu/kubeflow-rocks/jupyter-tensorflow-full/tests/test_access.py
Running jupyter-tensorflow-full_v1.8.0_20.04_1_amd64:v1.8.0_20.04_1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3914  100  3914    0     0   254k      0 --:--:-- --:--:-- --:--:--  254k
84ab5d20dbeb
84ab5d20dbeb
____________________________________________________________________________________________ summary ____________________________________________________________________________________________
  unit: commands succeeded
  congratulations :)

@NohaIhab NohaIhab requested a review from a team as a code owner September 21, 2023 09:50
@@ -25,10 +25,10 @@ def main():
container_id = container_id[0:12]

# to ensure container is started
time.sleep(5)
time.sleep(10)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to add tenacity here instead of sleep?

@i-chvets
Copy link
Contributor

@NohaIhab We need to update comments to indicate new branch:

# Based on the following Dockerfiles:
# https://github.com/kubeflow/kubeflow/blob/v1.7-branch/components/example-notebook-servers/base/Dockerfile
# https://github.com/kubeflow/kubeflow/blob/v1.7-branch/components/example-notebook-servers/jupyter/Dockerfile
# https://github.com/kubeflow/kubeflow/blob/v1.7-branch/components/example-notebook-servers/jupyter-tensorflow-full/cpu.Dockerfile

@i-chvets
Copy link
Contributor

i-chvets commented Sep 28, 2023

We are hitting incompatibility issues in our Jupyter Tensorflow ROCK. kfp* is pinned in upstream (has been for 6 months):

kfp==1.6.3
kfp-server-api==1.6.0

This is causing issues with urllib3:

ImportError: cannot import name 'appengine' from 'urllib3.contrib'

Looks like issue is fixed in kfp-1.8.21 and kfp-2.0.0b16

Options:

  • Bump version to either 1.8.21 or 2.0.0b16 (dangerous). Will require us to have our own patch for ROCK (as per our spec) and create PR upstream.

@i-chvets
Copy link
Contributor

i-chvets commented Sep 28, 2023

Upstream v1.8-branch Docker is building with errors (container image is created successfully):

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jupyterlab-server 2.25.0 requires jsonschema>=4.18.0, but you have jsonschema 3.2.0 which is incompatible.
tensorflow 2.9.3 requires absl-py>=1.0.0, but you have absl-py 0.11.0 which is incompatible.

Summary of changes:
- Added test imports script and updated tox.ini to call it.
@i-chvets i-chvets force-pushed the kf-4211-update-jupyter-tensorflow-rock branch from 143628d to 27e2ae8 Compare September 28, 2023 18:14
@i-chvets
Copy link
Contributor

Added imports tests with commented out (TO-DO) kfp import which causes dependency issue. This is the same way other ROCKs are handling this issue.

Created issue to track it: #53

Ivan Chvets added 2 commits October 10, 2023 16:44
Summary of changes:
- Updated environment variables handling and service command to use env
  variables.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants