-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker Image Change #1207
Merged
+425
−398
Merged
Docker Image Change #1207
Changes from 26 commits
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
16aa5b4
Add new AL2 based image and gunicorn
jawadqur 84e63b9
Update poetry
jawadqur 9883f05
updating the command
EliseCastle23 a243631
Merge branch 'master' into feat/al2
EliseCastle23 0238816
updating poetry lock
EliseCastle23 016636e
fixing link
EliseCastle23 0fe5859
fixing deadlinks
EliseCastle23 6708a84
fixing case
EliseCastle23 c514982
Merge branch 'master' into feat/al2
EliseCastle23 9fc2412
Merge branch 'master' into feat/al2
Avantol13 2031c74
feat(docker): update to use new base image
Avantol13 cc0e908
feat(mcrypt): add wip work to have 2 images
Avantol13 351d6e5
Merge branch 'master' into feat/al2
Avantol13 c82765b
WIP
nss10 a179a04
Update ci.yaml
nss10 2502803
Changing the basic dockerfile
nss10 43a98f4
Merge branch 'master' into chore/ccrypt_usersync
BinamB f198e66
update for arm
BinamB f8b848b
Fix downloads
BinamB fab15db
add y
BinamB 5937738
fix docker
BinamB 2bb0b1e
single image
BinamB 31e50ae
fix image
BinamB 8e664c0
add poetry run to gunicorn
BinamB d448672
update poetry lock
BinamB 63217d7
update dockerfile
BinamB 108360f
make sub into string
BinamB 536e11b
fix token
BinamB 57f1233
Merge branch 'master' into chore/ccrypt_usersync
BinamB 9b8bd74
testing import
BinamB 7d96128
fix get config
BinamB 162c58b
set default
BinamB 3df68fc
remove try block
BinamB aab5a84
Add tar
jawadqur 7286c02
Remove mcrypt docker image
BinamB 29bf703
Update ccrypt for arm
BinamB 57b1b2b
Merge branch 'master' into chore/ccrypt_usersync
nss10 be49983
Add `xz` alongside `tar` in Dockerfile
nss10 78da528
Update integration tests cloud auto branch
nss10 f9e96d5
Change cloud auto branch
nss10 b7fae52
Fix Indentation on the `uses` tab
nss10 3e60965
Merge branch 'master' into chore/ccrypt_usersync
nss10 4e9e6be
Updating integration tests path. Also cleaning up Dockerfile
nss10 9e0d31e
Remove CUSTOM_TEST_BRANCH
nss10 652578e
Adding poetry's venv/bin to PATH
nss10 8c6cca0
Updating ssh command in sync_users for dbgap
nss10 1f89888
Chanigng the ccrypt install location from builder stage to final stage
nss10 7bd4997
Add ccdecrypt install commands upon download
nss10 a83fc06
[WIP]: fix command errors to get it to work
nss10 1faa0dc
remove tragetarch
BinamB 1abb3da
remove which
BinamB 132f5c0
Revert: Custom cloud automation branch in integration tests
nss10 4a4de95
Merge branch 'master' into chore/ccrypt_usersync
nss10 52232d3
Change pyproject version and update gitignore
nss10 4780e10
Merge branch 'master' into chore/ccrypt_usersync
Avantol13 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -102,7 +102,6 @@ ENV/ | |
.mypy_cache/ | ||
|
||
# jwt keys | ||
keys | ||
tests/resources/keys/*.pem | ||
|
||
.DS_Store | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,56 +1,74 @@ | ||
# To run: docker run --rm -d -v /path/to/fence-config.yaml:/var/www/fence/fence-config.yaml --name=fence -p 80:80 fence | ||
# To check running container do: docker exec -it fence /bin/bash | ||
# To build: docker build -t fence:latest . | ||
# To run interactive: | ||
# docker run -v ~/.gen3/fence/fence-config.yaml:/var/www/fence/fence-config.yaml -v ./keys/:/fence/keys/ fence:latest | ||
# To check running container do: docker exec -it CONTAINER bash | ||
|
||
FROM quay.io/cdis/python:python3.9-buster-2.0.0 | ||
ARG AZLINUX_BASE_VERSION=feat_python-nginx | ||
|
||
# ------ Base stage ------ | ||
FROM quay.io/cdis/python-nginx-al:${AZLINUX_BASE_VERSION} AS base | ||
|
||
# Comment this in, and comment out the line above, if quay is down | ||
# FROM 707767160287.dkr.ecr.us-east-1.amazonaws.com/gen3/python-nginx-al:${AZLINUX_BASE_VERSION} as base | ||
|
||
ENV appname=fence | ||
|
||
RUN pip install --upgrade pip | ||
RUN pip install --upgrade poetry | ||
RUN apt-get update \ | ||
&& apt-get install -y --no-install-recommends curl bash git \ | ||
&& apt-get -y install vim \ | ||
libmcrypt4 mcrypt \ | ||
&& apt-get clean \ | ||
&& rm -rf /var/lib/apt/lists/ | ||
|
||
RUN mkdir -p /var/www/$appname \ | ||
&& mkdir -p /var/www/.cache/Python-Eggs/ \ | ||
&& mkdir /run/nginx/ \ | ||
&& ln -sf /dev/stdout /var/log/nginx/access.log \ | ||
&& ln -sf /dev/stderr /var/log/nginx/error.log \ | ||
&& chown nginx -R /var/www/.cache/Python-Eggs/ \ | ||
&& chown nginx /var/www/$appname | ||
|
||
# aws cli v2 - needed for storing files in s3 during usersync k8s job | ||
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \ | ||
&& unzip awscliv2.zip \ | ||
&& ./aws/install \ | ||
&& /bin/rm -rf awscliv2.zip ./aws | ||
|
||
WORKDIR /$appname | ||
|
||
# copy ONLY poetry artifact, install the dependencies but not fence | ||
# this will make sure than the dependencies is cached | ||
COPY poetry.lock pyproject.toml /$appname/ | ||
RUN poetry config virtualenvs.create false \ | ||
&& poetry install -vv --no-root --no-dev --no-interaction \ | ||
&& poetry show -v | ||
|
||
# copy source code ONLY after installing dependencies | ||
COPY . /$appname | ||
COPY ./deployment/uwsgi/uwsgi.ini /etc/uwsgi/uwsgi.ini | ||
COPY ./deployment/uwsgi/wsgi.py /$appname/wsgi.py | ||
COPY clear_prometheus_multiproc /$appname/clear_prometheus_multiproc | ||
|
||
# install fence | ||
RUN poetry config virtualenvs.create false \ | ||
&& poetry install -vv --no-dev --no-interaction \ | ||
&& poetry show -v | ||
|
||
RUN COMMIT=`git rev-parse HEAD` && echo "COMMIT=\"${COMMIT}\"" >$appname/version_data.py \ | ||
&& VERSION=`git describe --always --tags` && echo "VERSION=\"${VERSION}\"" >>$appname/version_data.py | ||
|
||
WORKDIR /var/www/$appname | ||
|
||
CMD ["sh","-c","bash /fence/dockerrun.bash && /dockerrun.sh"] | ||
WORKDIR /${appname} | ||
|
||
RUN chown -R gen3:gen3 /${appname} | ||
|
||
# ------ Builder stage ------ | ||
FROM base AS builder | ||
|
||
# Install ccrypt to decrypt dbgap telmetry files | ||
RUN if [ "$TARGETARCH" = "amd64" ]; then \ | ||
echo "Upgrading dnf"; \ | ||
dnf upgrade -y && \ | ||
echo "Installing Packages"; \ | ||
dnf install -y \ | ||
libxcrypt-compat-4.4.33 \ | ||
libpq-15.0 && \ | ||
echo "Installing RPM"; \ | ||
rpm -i https://ccrypt.sourceforge.net/download/1.11/ccrypt_1.11-1_amd64.deb; \ | ||
fi | ||
|
||
RUN if [ "$TARGETARCH" = "arm64" ]; then \ | ||
echo "Upgrading dnf"; \ | ||
dnf upgrade -y && \ | ||
echo "Installing Packages"; \ | ||
dnf install -y \ | ||
libxcrypt-compat-4.4.33 \ | ||
libpq-15.0 && \ | ||
echo "Installing RPM"; \ | ||
rpm -i https://ccrypt.sourceforge.net/download/1.11/ccrypt-1.11-1.x86_64.rpm; \ | ||
fi | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I noticed we are using an x86 binary for ARM. Will this be emulated for ARM during use? Has any testing been done to confirm its functionality? |
||
|
||
# Install just the deps without the code as it's own step to avoid redoing this on code changes | ||
COPY poetry.lock pyproject.toml /${appname}/ | ||
RUN poetry lock -vv --no-update \ | ||
&& poetry install -vv --only main --no-interaction | ||
|
||
# Move app files into working directory | ||
COPY --chown=gen3:gen3 . /$appname | ||
COPY --chown=gen3:gen3 ./deployment/wsgi/wsgi.py /$appname/wsgi.py | ||
|
||
# Do the install again incase the app itself needs install | ||
RUN poetry lock -vv --no-update \ | ||
&& poetry install -vv --only main --no-interaction | ||
|
||
ENV PATH="$(poetry env info --path)/bin:$PATH" | ||
|
||
nss10 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# Setup version info | ||
RUN git config --global --add safe.directory /${appname} && COMMIT=`git rev-parse HEAD` && echo "COMMIT=\"${COMMIT}\"" > /$appname/version_data.py \ | ||
&& VERSION=`git describe --always --tags` && echo "VERSION=\"${VERSION}\"" >> /$appname/version_data.py | ||
|
||
# install tar | ||
# RUN yum install tar -y | ||
# do we need to untar jwt-keys? | ||
|
||
# ------ Final stage ------ | ||
FROM base | ||
|
||
COPY --chown=gen3:gen3 --from=builder /$appname /$appname | ||
|
||
CMD ["/bin/bash", "-c", "/fence/dockerrun.bash"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,67 @@ | ||
# old Dockerfile -- specifically for the use of Mcrypt, since Al2 image does not support Mcrypt (which is used in usersyncjob) | ||
# To run: docker run --rm -d -v /path/to/fence-config.yaml:/var/www/fence/fence-config.yaml --name=fence -p 80:80 fence | ||
# To check running container do: docker exec -it fence /bin/bash | ||
|
||
FROM quay.io/cdis/python:python3.9-buster-2.0.0 | ||
|
||
ENV appname=fence | ||
WORKDIR /$appname | ||
|
||
|
||
RUN pip install --upgrade pip | ||
RUN pip install --upgrade poetry | ||
RUN apt-get update \ | ||
&& apt-get install -y --no-install-recommends curl bash git \ | ||
&& apt-get -y install vim \ | ||
libmcrypt4 mcrypt \ | ||
&& apt-get clean \ | ||
&& rm -rf /var/lib/apt/lists/ | ||
|
||
RUN mkdir -p /var/www/$appname \ | ||
&& mkdir -p /var/www/.cache/Python-Eggs/ \ | ||
&& mkdir /run/nginx/ \ | ||
&& ln -sf /dev/stdout /var/log/nginx/access.log \ | ||
&& ln -sf /dev/stderr /var/log/nginx/error.log \ | ||
&& chown nginx -R /var/www/.cache/Python-Eggs/ \ | ||
&& chown nginx /var/www/$appname | ||
|
||
# aws cli v2 - needed for storing files in s3 during usersync k8s job | ||
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \ | ||
&& unzip awscliv2.zip \ | ||
&& ./aws/install \ | ||
&& /bin/rm -rf awscliv2.zip ./aws | ||
|
||
|
||
# copy ONLY poetry artifact, install the dependencies but not fence | ||
# this will make sure than the dependencies is cached | ||
COPY poetry.lock pyproject.toml /$appname/ | ||
RUN pwd | ||
|
||
RUN sed -i 's/psycopg2-binary = "<3"/psycopg2 = "<3"/g' /$appname/pyproject.toml | ||
|
||
RUN poetry lock -vv --no-update \ | ||
&& poetry config virtualenvs.create false \ | ||
&& poetry install -vv --no-root --no-dev --no-interaction \ | ||
&& poetry show -v | ||
|
||
# copy source code ONLY after installing dependencies | ||
COPY . /$appname | ||
RUN sed -i 's/psycopg2-binary = "<3"/psycopg2 = "<3"/g' /$appname/pyproject.toml | ||
|
||
# Changing the ownership of gunicorn to root for this dockerfile to run | ||
RUN sed -i 's/gen3/root/g' /$appname/deployment/wsgi/gunicorn.conf.py | ||
|
||
COPY ./deployment/uwsgi/uwsgi.ini /etc/uwsgi/uwsgi.ini | ||
COPY ./deployment/wsgi/wsgi.py /$appname/wsgi.py | ||
COPY clear_prometheus_multiproc /$appname/clear_prometheus_multiproc | ||
|
||
# install fence | ||
RUN poetry config virtualenvs.create false \ | ||
&& poetry lock -vv --no-update \ | ||
&& poetry install -vv --no-dev --no-interaction \ | ||
&& poetry show -v | ||
|
||
RUN COMMIT=`git rev-parse HEAD` && echo "COMMIT=\"${COMMIT}\"" >$appname/version_data.py \ | ||
&& VERSION=`git describe --always --tags` && echo "VERSION=\"${VERSION}\"" >>$appname/version_data.py | ||
|
||
CMD ["poetry", "run", "gunicorn", "-c", "deployment/wsgi/gunicorn.conf.py"] |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
wsgi_app = "deployment.wsgi.wsgi:application" | ||
bind = "0.0.0.0:8000" | ||
workers = 1 | ||
preload_app = True | ||
user = "root" | ||
group = "root" | ||
timeout = 300 | ||
keepalive = 2 | ||
keepalive_timeout = 5 |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why was this removed? The instructions have people placing .pem keys in a directory named "keys". I think we need to keep this