Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Image Change #1207

Merged
merged 55 commits into from
Feb 19, 2025
Merged
Changes from 26 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
16aa5b4
Add new AL2 based image and gunicorn
jawadqur Sep 4, 2024
84e63b9
Update poetry
jawadqur Sep 4, 2024
9883f05
updating the command
EliseCastle23 Sep 6, 2024
a243631
Merge branch 'master' into feat/al2
EliseCastle23 Oct 16, 2024
0238816
updating poetry lock
EliseCastle23 Oct 16, 2024
016636e
fixing link
EliseCastle23 Oct 17, 2024
0fe5859
fixing deadlinks
EliseCastle23 Oct 17, 2024
6708a84
fixing case
EliseCastle23 Oct 18, 2024
c514982
Merge branch 'master' into feat/al2
EliseCastle23 Oct 18, 2024
9fc2412
Merge branch 'master' into feat/al2
Avantol13 Nov 1, 2024
2031c74
feat(docker): update to use new base image
Avantol13 Nov 1, 2024
cc0e908
feat(mcrypt): add wip work to have 2 images
Avantol13 Nov 8, 2024
351d6e5
Merge branch 'master' into feat/al2
Avantol13 Nov 8, 2024
c82765b
WIP
nss10 Nov 22, 2024
a179a04
Update ci.yaml
nss10 Nov 22, 2024
2502803
Changing the basic dockerfile
nss10 Nov 22, 2024
43a98f4
Merge branch 'master' into chore/ccrypt_usersync
BinamB Nov 22, 2024
f198e66
update for arm
BinamB Nov 22, 2024
f8b848b
Fix downloads
BinamB Nov 22, 2024
fab15db
add y
BinamB Nov 22, 2024
5937738
fix docker
BinamB Nov 22, 2024
2bb0b1e
single image
BinamB Dec 13, 2024
31e50ae
fix image
BinamB Dec 13, 2024
8e664c0
add poetry run to gunicorn
BinamB Dec 15, 2024
d448672
update poetry lock
BinamB Dec 15, 2024
63217d7
update dockerfile
BinamB Dec 15, 2024
108360f
make sub into string
BinamB Dec 18, 2024
536e11b
fix token
BinamB Dec 20, 2024
57f1233
Merge branch 'master' into chore/ccrypt_usersync
BinamB Dec 21, 2024
9b8bd74
testing import
BinamB Dec 24, 2024
7d96128
fix get config
BinamB Jan 3, 2025
162c58b
set default
BinamB Jan 3, 2025
3df68fc
remove try block
BinamB Jan 3, 2025
aab5a84
Add tar
jawadqur Jan 7, 2025
7286c02
Remove mcrypt docker image
BinamB Jan 8, 2025
29bf703
Update ccrypt for arm
BinamB Jan 8, 2025
57b1b2b
Merge branch 'master' into chore/ccrypt_usersync
nss10 Jan 27, 2025
be49983
Add `xz` alongside `tar` in Dockerfile
nss10 Jan 28, 2025
78da528
Update integration tests cloud auto branch
nss10 Jan 29, 2025
f9e96d5
Change cloud auto branch
nss10 Jan 29, 2025
b7fae52
Fix Indentation on the `uses` tab
nss10 Jan 29, 2025
3e60965
Merge branch 'master' into chore/ccrypt_usersync
nss10 Jan 29, 2025
4e9e6be
Updating integration tests path. Also cleaning up Dockerfile
nss10 Jan 29, 2025
9e0d31e
Remove CUSTOM_TEST_BRANCH
nss10 Jan 29, 2025
652578e
Adding poetry's venv/bin to PATH
nss10 Jan 30, 2025
8c6cca0
Updating ssh command in sync_users for dbgap
nss10 Feb 4, 2025
1f89888
Chanigng the ccrypt install location from builder stage to final stage
nss10 Feb 4, 2025
7bd4997
Add ccdecrypt install commands upon download
nss10 Feb 5, 2025
a83fc06
[WIP]: fix command errors to get it to work
nss10 Feb 5, 2025
1faa0dc
remove tragetarch
BinamB Feb 5, 2025
1abb3da
remove which
BinamB Feb 5, 2025
132f5c0
Revert: Custom cloud automation branch in integration tests
nss10 Feb 7, 2025
4a4de95
Merge branch 'master' into chore/ccrypt_usersync
nss10 Feb 7, 2025
52232d3
Change pyproject version and update gitignore
nss10 Feb 12, 2025
4780e10
Merge branch 'master' into chore/ccrypt_usersync
Avantol13 Feb 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -10,19 +10,22 @@ jobs:
Security:
name: Security Pipeline
uses: uc-cdis/.github/.github/workflows/securitypipeline.yaml@master
secrets: inherit
secrets: inherit # pragma: allowlist secret

UnitTest:
name: Python Unit Test with Postgres
uses: uc-cdis/.github/.github/workflows/python_unit_test.yaml@master
with:
python-version: '3.9'
test-script: 'tests/ci_commands_script.sh'
run-coveralls: true
ci:
python-version: '3.9'
test-script: 'tests/ci_commands_script.sh'
run-coveralls: true

BuildImageAndPush:
name: Build Image and Push
# TODO Uncomment after PXP-9212
# needs: Security
needs: Security
with:
BUILD_PLATFORMS: "linux/amd64"
# https://github.com/uc-cdis/.github/blob/master/.github/workflows/image_build_push.yaml
uses: uc-cdis/.github/.github/workflows/image_build_push.yaml@master
secrets:
ECR_AWS_ACCESS_KEY_ID: ${{ secrets.ECR_AWS_ACCESS_KEY_ID }}
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -102,7 +102,6 @@ ENV/
.mypy_cache/

# jwt keys
keys
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was this removed? The instructions have people placing .pem keys in a directory named "keys". I think we need to keep this

tests/resources/keys/*.pem

.DS_Store
39 changes: 6 additions & 33 deletions .secrets.baseline
Original file line number Diff line number Diff line change
@@ -115,13 +115,13 @@
}
],
"results": {
".github/workflows/ci.yaml": [
".github/workflows/buildpipeline.yaml": [
{
"type": "Secret Keyword",
"filename": ".github/workflows/ci.yaml",
"filename": ".github/workflows/buildpipeline.yaml",
"hashed_secret": "3e26d6750975d678acb8fa35a0f69237881576b0",
"is_verified": false,
"line_number": 13
"line_number": 17
}
],
"deployment/scripts/postgresql/postgresql_init.sql": [
@@ -210,22 +210,13 @@
"line_number": 137
}
],
"fence/resources/storage/storageclient/cleversafe.py": [
{
"type": "Secret Keyword",
"filename": "fence/resources/storage/storageclient/cleversafe.py",
"hashed_secret": "7cb6efb98ba5972a9b5090dc2e517fe14d12cb04",
"is_verified": false,
"line_number": 274
}
],
"fence/utils.py": [
{
"type": "Secret Keyword",
"filename": "fence/utils.py",
"hashed_secret": "8318df9ecda039deac9868adf1944a29a95c7114",
"is_verified": false,
"line_number": 129
"line_number": 128
}
],
"migrations/versions/a04a70296688_non_unique_client_name.py": [
@@ -268,14 +259,14 @@
"filename": "tests/conftest.py",
"hashed_secret": "1348b145fa1a555461c1b790a2f66614781091e9",
"is_verified": false,
"line_number": 1570
"line_number": 1556
},
{
"type": "Base64 High Entropy String",
"filename": "tests/conftest.py",
"hashed_secret": "227dea087477346785aefd575f91dd13ab86c108",
"is_verified": false,
"line_number": 1594
"line_number": 1579
}
],
"tests/credentials/google/test_credentials.py": [
@@ -394,24 +385,6 @@
"line_number": 300
}
],
"tests/storageclient/storage_client_mock.py": [
{
"type": "Secret Keyword",
"filename": "tests/storageclient/storage_client_mock.py",
"hashed_secret": "37bbea9557f9efd1eeadb25dda9ab6514f08fde9",
"is_verified": false,
"line_number": 158
}
],
"tests/storageclient/test_cleversafe_api_client.py": [
{
"type": "Secret Keyword",
"filename": "tests/storageclient/test_cleversafe_api_client.py",
"hashed_secret": "f683c485d521c2e45830146dd570111770baea29",
"is_verified": false,
"line_number": 130
}
],
"tests/test-fence-config.yaml": [
{
"type": "Basic Auth Credentials",
122 changes: 70 additions & 52 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,56 +1,74 @@
# To run: docker run --rm -d -v /path/to/fence-config.yaml:/var/www/fence/fence-config.yaml --name=fence -p 80:80 fence
# To check running container do: docker exec -it fence /bin/bash
# To build: docker build -t fence:latest .
# To run interactive:
# docker run -v ~/.gen3/fence/fence-config.yaml:/var/www/fence/fence-config.yaml -v ./keys/:/fence/keys/ fence:latest
# To check running container do: docker exec -it CONTAINER bash

FROM quay.io/cdis/python:python3.9-buster-2.0.0
ARG AZLINUX_BASE_VERSION=feat_python-nginx

# ------ Base stage ------
FROM quay.io/cdis/python-nginx-al:${AZLINUX_BASE_VERSION} AS base

# Comment this in, and comment out the line above, if quay is down
# FROM 707767160287.dkr.ecr.us-east-1.amazonaws.com/gen3/python-nginx-al:${AZLINUX_BASE_VERSION} as base

ENV appname=fence

RUN pip install --upgrade pip
RUN pip install --upgrade poetry
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl bash git \
&& apt-get -y install vim \
libmcrypt4 mcrypt \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/

RUN mkdir -p /var/www/$appname \
&& mkdir -p /var/www/.cache/Python-Eggs/ \
&& mkdir /run/nginx/ \
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log \
&& chown nginx -R /var/www/.cache/Python-Eggs/ \
&& chown nginx /var/www/$appname

# aws cli v2 - needed for storing files in s3 during usersync k8s job
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install \
&& /bin/rm -rf awscliv2.zip ./aws

WORKDIR /$appname

# copy ONLY poetry artifact, install the dependencies but not fence
# this will make sure than the dependencies is cached
COPY poetry.lock pyproject.toml /$appname/
RUN poetry config virtualenvs.create false \
&& poetry install -vv --no-root --no-dev --no-interaction \
&& poetry show -v

# copy source code ONLY after installing dependencies
COPY . /$appname
COPY ./deployment/uwsgi/uwsgi.ini /etc/uwsgi/uwsgi.ini
COPY ./deployment/uwsgi/wsgi.py /$appname/wsgi.py
COPY clear_prometheus_multiproc /$appname/clear_prometheus_multiproc

# install fence
RUN poetry config virtualenvs.create false \
&& poetry install -vv --no-dev --no-interaction \
&& poetry show -v

RUN COMMIT=`git rev-parse HEAD` && echo "COMMIT=\"${COMMIT}\"" >$appname/version_data.py \
&& VERSION=`git describe --always --tags` && echo "VERSION=\"${VERSION}\"" >>$appname/version_data.py

WORKDIR /var/www/$appname

CMD ["sh","-c","bash /fence/dockerrun.bash && /dockerrun.sh"]
WORKDIR /${appname}

RUN chown -R gen3:gen3 /${appname}

# ------ Builder stage ------
FROM base AS builder

# Install ccrypt to decrypt dbgap telmetry files
RUN if [ "$TARGETARCH" = "amd64" ]; then \
echo "Upgrading dnf"; \
dnf upgrade -y && \
echo "Installing Packages"; \
dnf install -y \
libxcrypt-compat-4.4.33 \
libpq-15.0 && \
echo "Installing RPM"; \
rpm -i https://ccrypt.sourceforge.net/download/1.11/ccrypt_1.11-1_amd64.deb; \
fi

RUN if [ "$TARGETARCH" = "arm64" ]; then \
echo "Upgrading dnf"; \
dnf upgrade -y && \
echo "Installing Packages"; \
dnf install -y \
libxcrypt-compat-4.4.33 \
libpq-15.0 && \
echo "Installing RPM"; \
rpm -i https://ccrypt.sourceforge.net/download/1.11/ccrypt-1.11-1.x86_64.rpm; \
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed we are using an x86 binary for ARM. Will this be emulated for ARM during use? Has any testing been done to confirm its functionality?


# Install just the deps without the code as it's own step to avoid redoing this on code changes
COPY poetry.lock pyproject.toml /${appname}/
RUN poetry lock -vv --no-update \
&& poetry install -vv --only main --no-interaction

# Move app files into working directory
COPY --chown=gen3:gen3 . /$appname
COPY --chown=gen3:gen3 ./deployment/wsgi/wsgi.py /$appname/wsgi.py

# Do the install again incase the app itself needs install
RUN poetry lock -vv --no-update \
&& poetry install -vv --only main --no-interaction

ENV PATH="$(poetry env info --path)/bin:$PATH"

# Setup version info
RUN git config --global --add safe.directory /${appname} && COMMIT=`git rev-parse HEAD` && echo "COMMIT=\"${COMMIT}\"" > /$appname/version_data.py \
&& VERSION=`git describe --always --tags` && echo "VERSION=\"${VERSION}\"" >> /$appname/version_data.py

# install tar
# RUN yum install tar -y
# do we need to untar jwt-keys?

# ------ Final stage ------
FROM base

COPY --chown=gen3:gen3 --from=builder /$appname /$appname

CMD ["/bin/bash", "-c", "/fence/dockerrun.bash"]
67 changes: 67 additions & 0 deletions DockerfileMcrypt
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# old Dockerfile -- specifically for the use of Mcrypt, since Al2 image does not support Mcrypt (which is used in usersyncjob)
# To run: docker run --rm -d -v /path/to/fence-config.yaml:/var/www/fence/fence-config.yaml --name=fence -p 80:80 fence
# To check running container do: docker exec -it fence /bin/bash

FROM quay.io/cdis/python:python3.9-buster-2.0.0

ENV appname=fence
WORKDIR /$appname


RUN pip install --upgrade pip
RUN pip install --upgrade poetry
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl bash git \
&& apt-get -y install vim \
libmcrypt4 mcrypt \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/

RUN mkdir -p /var/www/$appname \
&& mkdir -p /var/www/.cache/Python-Eggs/ \
&& mkdir /run/nginx/ \
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log \
&& chown nginx -R /var/www/.cache/Python-Eggs/ \
&& chown nginx /var/www/$appname

# aws cli v2 - needed for storing files in s3 during usersync k8s job
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install \
&& /bin/rm -rf awscliv2.zip ./aws


# copy ONLY poetry artifact, install the dependencies but not fence
# this will make sure than the dependencies is cached
COPY poetry.lock pyproject.toml /$appname/
RUN pwd

RUN sed -i 's/psycopg2-binary = "<3"/psycopg2 = "<3"/g' /$appname/pyproject.toml

RUN poetry lock -vv --no-update \
&& poetry config virtualenvs.create false \
&& poetry install -vv --no-root --no-dev --no-interaction \
&& poetry show -v

# copy source code ONLY after installing dependencies
COPY . /$appname
RUN sed -i 's/psycopg2-binary = "<3"/psycopg2 = "<3"/g' /$appname/pyproject.toml

# Changing the ownership of gunicorn to root for this dockerfile to run
RUN sed -i 's/gen3/root/g' /$appname/deployment/wsgi/gunicorn.conf.py

COPY ./deployment/uwsgi/uwsgi.ini /etc/uwsgi/uwsgi.ini
COPY ./deployment/wsgi/wsgi.py /$appname/wsgi.py
COPY clear_prometheus_multiproc /$appname/clear_prometheus_multiproc

# install fence
RUN poetry config virtualenvs.create false \
&& poetry lock -vv --no-update \
&& poetry install -vv --no-dev --no-interaction \
&& poetry show -v

RUN COMMIT=`git rev-parse HEAD` && echo "COMMIT=\"${COMMIT}\"" >$appname/version_data.py \
&& VERSION=`git describe --always --tags` && echo "VERSION=\"${VERSION}\"" >>$appname/version_data.py

CMD ["poetry", "run", "gunicorn", "-c", "deployment/wsgi/gunicorn.conf.py"]
19 changes: 0 additions & 19 deletions deployment/fence.conf

This file was deleted.

9 changes: 9 additions & 0 deletions deployment/wsgi/gunicorn.conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
wsgi_app = "deployment.wsgi.wsgi:application"
bind = "0.0.0.0:8000"
workers = 1
preload_app = True
user = "root"
group = "root"
timeout = 300
keepalive = 2
keepalive_timeout = 5
File renamed without changes.
8 changes: 3 additions & 5 deletions dockerrun.bash
Original file line number Diff line number Diff line change
@@ -1,10 +1,5 @@
#!/bin/bash

#
# Update certificate authority index -
# environment may have mounted more authorities
#
update-ca-certificates
#
# Kubernetes may mount jwt-keys as a tar ball
#
@@ -18,3 +13,6 @@ if [ -f /fence/jwt-keys.tar ]; then
fi
)
fi

nginx
poetry run gunicorn -c "/fence/deployment/wsgi/gunicorn.conf.py"
24 changes: 0 additions & 24 deletions dockerrunshib.bash

This file was deleted.

Loading