Skip to content

Onboarding New Developers

Jump to bottom
Chloe edited this page Jun 5, 2024 · 5 revisions

Introduction

One of the biggest challenges beginners face in web development is deployment. It creates a hurdle – working with the cloud requires understanding complex concepts like Kubernetes and choosing the right service. This can be overwhelming and slow down their learning.

Our internal developer platform tackles this issue head-on by automating the deployment process for the two main frameworks used in the web development basics module: Node.js Express and Flask. This simplifies the experience for students, allowing them to focus on core development concepts.

The platform acts as an abstraction layer, hiding the complexities of cloud configuration. Students can access and manage the configurations needed to run their applications without needing in-depth cloud knowledge.

We built this platform using Backstage, a framework chosen for its flexibility and ease of use. Backstage seamlessly integrates continuous integration and deployment pipelines, further streamlining the process. Additionally, Backstage offers a user-friendly graphical interface, perfect for beginners who are unfamiliar with command lines.

This document outlines the automated deployment process facilitated by our platform. It details the interaction between various components, from choosing application templates to the final deployment on Google Cloud using Docker and Kubernetes.

User Experience (UX)

img

Initiate deployment template

User Workflow:

  1. Select Initiate Deployment Template: The user begins by selecting the Initiate deployment template in the Backstage interface tailored to their project needs, whether starting a new deployment or updating an existing one. This template is the initial interaction point, providing a user-friendly interface for beginner developers.
  2. Input Requirements: The user provides the necessary information, such as the GitHub repository URL and email address. Validation ensures all inputs are correct, preventing errors in the deployment process.
  3. Deployment Trigger: Upon successful authentication and validation of inputs, the template triggers the corresponding GitHub Actions to initiate the deployment process.

Screenshots:

img

Login page.

img

Home page after successful login.

img

Templates to choose from

Form validation

We implemented form validation in the template to help limit errors resulting from invalid user input by utilizing regex expressions that are set in place to validate the user's input.

The following screenshot shows the error message received upon clicking review after providing an invalid GitHub repo.

GitHub Repo URL Validation

Upon Entering an invalid Github Repository URL, the user is prompted to match the regex pattern:

^https:\/\/github\.com\/**[**a-zA-Z0-9_-**]**+\/**[**a-zA-Z0-9_-**]**

img

Error message displayed for entering wrong GitHub repo URL

Email Address Validation

Upon Entering an invalid email address, the user is prompted to match the regex pattern: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

img

Error message displayed upon entering an invalid email address

2. System Architecture

We utilize Backstage, which is an open-source software framework designed specifically to create developer portals, to be the user-facing portal. This portal acts as a central hub, consolidating various tools and resources developers need to manage their projects in one place. It is an all-in-one platform for development activities.

The platform is structured around two main GitHub repositories; one handles the Backstage setup and essential functions, while the other manages deployment pipelines and project-specific configurations. This separation adheres to the single responsibility principle, allowing for clearer maintenance and error handling. The CODE-IDP repository includes authentication components, templates, deployment files, and Backstage's CI/CD pipeline. The idp-hosted-projects repository contains the deployment pipelines for the students’ projects, Docker images, submodules, Terraform files, and Helm chart files.

We centered our backend strategy around Docker and GitHub Actions which can be easily accessed through Backstage. These technologies form the core logical components of our project.

Docker:

One requirement to deploy to the Google Kubernetes engine is to use containers, as the pods run on a container-optimized operating system. To meet this requirement without expecting the students to provide us with Dockerfiles in their repositories, we created two Dockerfiles that create images of the students’ applications.

Github actions:

To adhere to continuous integration and deployment, we chose GitHub Actions as our pipeline due to the feasibility of integrating Backstage and Google Cloud, providing services to access Google Cloud applications through GitHub Actions. They also enabled us to structure our logic by using reusable workflows.

Detailed Components:

  • Frontend: Hosted on a cloud service, built using React framework, providing a graphical interface for user interactions.
  • Backend: The backend, also cloud-hosted, runs on Node.js, handling API requests and integration with GitHub for deployment operations.
  • Database: Utilizes a SQL database to store user data and session information, with secured interactions via backend APIs.

Architecture Diagram:

img

System components

Abstract mention of third-party tools:

We rely on multiple third-party APIs to handle the different system requirements, such as:

  1. Supporting two frameworks dynamically and determining what framework the students use for their application:
    1. We used GitHub dependencies API to get repository dependencies.
  2. Low usage of the system at the beginning of the semester and higher usage at the end:
    1. Terraform to facilitate the management of the cluster.
    2. Kubernetes Engine on Google Cloud that manages containers.
  3. Supporting multiple deployments and Managing them:
    1. Helm Chart.
  4. Dockerizing the students’ applications to run on the Google Kubernetes engine:
    1. Docker
    2. Google Artifact Registry.
  5. Allowing only CODE students to use the application:
    1. Google OAuth.

3. Technical Implementation

Data-flow Diagram

The following diagram outlines the sequence from the GitHub Actions triggered to the deployment on Kubernetes. Detailed workflows include adding submodules to repositories, deducing the framework used, and using this information to build Docker images.img

Docker and Kubernetes Usage

  • Docker: Packages student applications into containers, guaranteeing that the application runs consistently across different environments. We utilize custom Dockerfiles that are selected based on the application framework detected earlier in the process.
  • Kubernetes: Automates the deployment, scaling, and management of these containerized applications on Google Kubernetes Engine. The deployment process involves:
    • Creating Kubernetes Deployment and Services: Utilizing Helm Charts to define and manage these resources ensures that deployments are repeatable, scalable, and manageable.
    • Automation and Orchestration: Automated through GitHub workflows, making the process entirely driven by inputs from users via the Backstage interface.

GitHub Workflows and Actions

GitOps Practices

We adhere to GitOps principles by maintaining a single source of truth for our infrastructure definitions in the idp-hosted-projects repository. This setup is fully automated and guided by user inputs via the Backstage interface.img

For Initiating New Deployment

The deployment process begins with the initiation of the create-image.yaml GitHub Action, which is critical in managing the structure and versioning of code within our system. The action operates as follows:

  1. Directory Check: The action first verifies the presence of a directory named submissions-current-year (e.g., submissions-2024) in the idp-hosted-projects repository, which stores all student code. If this directory does not exist, it is created using parameters supplied from the Backstage template. This step ensures that all student projects are organized by year, simplifying management and retrieval.
  2. Submodule Verification: Next, GitHub Actions checks if a submodule for the student’s repository already exists within the appropriate yearly directory. If it exists, the deployment process is halted, and the user is prompted to initiate an update process instead of creating a new deployment. This prevents duplicate entries and maintains the integrity of the deployment history.
  3. Submodule Creation: If no existing submodule is found, the student's repository is then added as a new submodule, and is stored in the following format:
  • idp-hosted-projects/submissions-(current-year)/(GitHub-username)-(repository-name)/(GitHub-username)-(repository-name)
  • For example, for a student named John Doe with a repository named example-repository in the year 2024, the path would be:
  • idp-hosted-projects/submissions-2024/john-done-example-repository/john-done-example-repository

Rationale Behind Using Submodules

The reasoning behind choosing submodules is that they enable us to keep track of updates after the initial deployment, resulting in a direct path to track the Docker images’ tags to update the deployment later. This also supports GitOps’ model by allowing version control at a more detailed level, ensuring that updates can be systematically tracked and managed,

Triggering Subsequent Actions

Once a submodule is successfully created or verified with no required updates, the GitHub Action flags the next job in the pipeline. This job is responsible for fetching dependencies necessary for building the Docker image, utilizing the framework details extracted from the project.

Updating an Existing Deployment

The update process mirrors the initial deployment checks but with added steps to ensure the currency of the deployment:

  • Update Trigger: The update-image.yaml workflow is activated by user selection from the Backstage interface. It begins by checking for the existence of the submodule and its current state.
  • Commit Comparison: It compares the SHA of the latest commit at the HEAD of the submodule against the SHA of the latest commit in the student's repository. If there is a discrepancy between these SHAs, indicating changes in the repository, the workflow pulls these changes into the submodule, otherwise it is terminated.
  • Docker Image Creation: Following the update of the submodule with the latest changes, the process to create a new Docker image is initiated. This ensures that the deployment reflects the most current version of the student’s project.

Automated Dockerization:

The dockerise-image.yaml workflow is integral to our Docker operations, ensuring that our codebase adheres to the DRY (Don’t Repeat Yourself) principle. This workflow is automatically triggered for both new deployments and updates, performing the following functions:

  1. Framework Detection:

    1. API Call: The workflow initiates by calling the GitHub dependencies API, which scans the student’s repository to identify all the dependencies used within their application.
    2. Data Handling: The API returns a JSON response listing these dependencies. The data is then saved to a JSON file for further processing.
    3. Framework Analysis: Using the Jq command, the workflow extracts values from the JSON file to determine the application’s framework, such as whether it’s built with Express or Flask.

  2. Dockerfile Selection and Execution:

    1. Dockerfile Matching: Depending on the identified framework, the appropriate Dockerfile, either for Node.js or Flask, is selected.
    2. Docker Build Process: The selected Dockerfile is then used to build a Docker image of the student’s application. This process makes the Docker image suited to the specific needs of the application.
    3. Execution Trigger: The Docker build process is initiated once the Dockerfile has been successfully copied to the student’s submodule.

  3. Handling Diverse Application Entry Points:

    1. Entry Point Identification: When constructing Docker images, especially for Express-based applications, we encounter variations in the naming conventions of the entry file (e.g. main.js, app.js, index.js).
    2. Dynamic File Search: If the standard names aren't found, the Dockerfile defaults to searching for a server.js. This flexibility helps in correctly configuring the Docker images regardless of the project’s internal file structure.
    3. Command-Line Integration: We pass the identified entry file name as an argument to the Dockerfile during the build process to ensure that the correct application entry point is configured.

  4. Authentication and Security:

    1. Google Cloud Authentication: Before pushing to Docker images, the process authenticates to Google Cloud using a service account coupled with OAuth credentials to ensure secure access to cloud services.
    2. Artifacts Registry: After authentication, the Docker images are pushed to the on idp-artifact-registry Google Cloud. This step is secured by the permissions associated with the service account.
    3. API Security: The GitHub dependencies API call requires a secure GitHub token, which we store as a secret in our repository under DEPENDENCY_GRAPH_TOKEN. This makes sure that our API interactions are secure and that sensitive credentials are managed appropriately.

  5. Image Build and Storage:

    1. Naming Conventions: The Docker images are tagged and named following a naming convention that includes the SHA values of the latest commit from the submodule, enhancing traceability and version control:
  • europe-west10-docker.pkg.dev/code-idp/idp-artifact-registry/john-doe-repository-example:d6cd1e2bd19e03a81132a23b2025920577f84e37
  1. Push and Trigger: Following a successful push of the Docker image, the deployment workflow is automatically triggered, moving forward with the Kubernetes deployment using the newly created Docker image.

Best Practices and Workflow Efficiency

Both Docker images adhere to the best practice for Docker container construction, ensuring they're lightweight, secure and efficient. The automated workflows designed around these practices ensure that the Dockerization process is reliable and consistent across various deployments.

Kubernetes

The deployment process in Kubernetes begins by integrating a GitHub submodule and building a Docker image. This image is then deployed to a Kubernetes cluster using Helm charts, which are configured to manage and orchestrate the application efficiently.

Kubernetes Cluster Information

GKE Cluster Setup:

  1. Initial Configuration: Before deploying Docker images, a Google Kubernetes Engine (GKE) cluster must be operational. If a cluster does not exist, it is created using specific Terraform configurations.

    1. Cluster Existence: Before deploying Docker images, a Google Kubernetes Engine (GKE) cluster must be operational. If a cluster does not exist, it is created using specific Terraform configurations.
    2. Terraform Activation: The creation is triggered by the create-gke-cluster.yaml workflow, which initializes the necessary Terraform configurations.

  2. Configuring the Kubernetes Cluster Using Terraform:

    1. Inputs Required:

      1. Service Account: Specifies the service account (TF_VAR_service_account) used by Terraform to manage the cluster's resources.
      2. Project ID: Identifies the Google Cloud project (TF_VAR_project_id) where the cluster resides.
      3. Node Count: Determines the maximum (TF_VAR_max_node_count) and initial (TF_VAR_initial_node_count) number of nodes that the cluster can scale to accommodate varying loads.

    2. Node Pool Customization:

      1. Horizontal Scaling: The cluster’s scaling strategy is set to horizontal scaling, which divides work among pools rather than increasing individual pod specifications.
      2. Custom Node Pool: Replaces the default node pool with a custom one that supports preemptible VMs, optimizing both cost and resource usage. These VMs are configured to last up to 24 hours without an expiration time, ideal for short-term project assessments.

    3. Resource Optimisation:

      1. System Upgrade and Repairs: Ensures that the node pool is capable of auto-repairing and upgrading, allowing for most optimal system performance and integrity.

    4. Namespace and Service Account Management:

      1. Kubernetes namespaces and service accounts: Creates Kubernetes namespaces and service accounts, assigning appropriate roles and permissions to each, thus isolating and securing deployments within the cluster.

Deploying Docker Images Using Helm Charts

We opted to use Helm as we expect approximately 20-50 deployments per semester, and managing the Kubernetes files required for the deployments independently would be a challenge as our internal developer platform scales up.

Helm Configurations:

  • Deployment Naming and Updates: Since Kubernetes utilizes namespaces to organize deployments, we use the submissions year as the namespace, where all deployments for that year reside. The deployment name itself follows the format github-username-github-repo-name-deployment. This way we can have multiple deployments without conflicts if needed.

  • Helm Templates: HELM uses templates within the CodeIdpChart. These templates generate new Kubernetes configuration files with new or updated information provided through values.yaml so we end up with customized deployment, ingress and service for each application independently. This approach guarantees consistency in values and steps, regardless whether a deployment is being created or updated. In addition to this Helm helps streamline the creates and updates of Kubernetes configurations by removing a need for various commands, which we utilize by using the helm upgrade -i command.

4. Tearing down clusters

Since the deployments are only needed during assessment time, there is no need to keep the cluster running afterward and exhaust resources and finances. We leverage the benefits of using infrastructure as code (Terraform) and run a GitHub workflow that enables us to kill all the existing deployments simply by running the delete-kubernetes-default-services.yaml workflow which deletes the services and the delete-gke-cluster.yaml workflow destroys the cluster itself.

5. Security and Authentication

OAuth Implementation and Security Considerations

This section aims to explain our current authentication approach and future plans for access control within Backstage.

Current Setup:

Security Considerations:

  • Limited Access: The sign-in method serves our MVP needs, however, currently all users have full access to backstage functionalities.
  • Future Permissions: We acknowledge the potential security risks associated with unrestricted access as new components are added to Backstage. However, as there is currently only a single component, no immediate risk is posed.
  • Planned Access Control: We plan to add permission-based access control to restrict users to relevant components like templates. This will enhance our security by granting access only to functionalities necessary for users’ roles.

Security Best Practices

  • Authentication: We use service accounts to authenticate to Google Cloud and OAuth, eliminating the use of less secure JSON configurations.
  • Kubernetes Service Account Roles: We enforce least privilege by limiting the roles assigned to Kubernetes service accounts within their designated namespaces.
  • Secure Credential Management: We prioritize secure credentials storage, avoiding storing sensitive information directly within GitHub workflows.
  • Backstage Data Validation: Error handling mechanisms are integrated within Backstage templates, preventing data from being inserted.
  • Action Output Control: Sensitive information is excluded from being printed during GitHub Actions executions.
  • Limited Service Account Scope: Google Cloud service accounts have their scope restricted, granting them only minimum permissions necessary to fulfill their tasks.

Local Development Setup and Testing

Authentication with Google OAuth:

Local development utilizes Google OAuth for authentication, which requires two environment variables to exist within the .env file:

  • GOOGLE_CLIENT_ID
  • GOOGLE_CLIENT_SECRET

These secrets are retrievable from your Google Cloud project under the name code-idp-local. You don’t need to recreate them unless absolutely necessary. If complete replacement is required, refer to the Backstage documentation for guidance.

Granting Repository Access:

Follow the tutorial to grant the token permissions to modify code-idp within the codeuniversity organization. This repository utilizes SAML single sign-on for enhanced security.

Testing Backstage Templates Locally:

Testing changes made to Backstage templates requires a GitHub secret stored in your .env file. This secret also allows you to modify another repository (idp-hosted-projects).

Creating and Configuring the GitHub Token:

  1. Follow this tutorial to create a GitHub token.
  2. During token creation, enable both workflow permissions and repo permissions.
  3. Add the generated token to your .env file with the name GITHUB_TOKEN.

Granting Repository Access:

Follow this tutorial to grant the token permissions to modify code-idp within the codeuniversity organization. This repository utilizes SAML single sign-on for enhanced security.

Creating New Templates:

For creating entirely new templates, refer to the Backstage documentation on template creation.

6. Next Steps after MVP

For scaling the MVP up and making it capable of handling more students, it’s crucial to tackle the issues that would be introduced as the project grows:

  • Currently, the MVP isn’t equipped to handle real-world sensitive data.
  • The MVP doesn’t have test coverage as not all tests are implemented, refer to this document for a detailed explanation of the current test state.
  • The MVP doesn’t handle dealing with malicious code from the student’s repository and has not investigated its impact on deployment and creating Docker images.
  • The credits available on Google Cloud are limited and would only handle the scaling to up to 50 students.
  • Multiple GitHub workflows with many values passed between them.
  • There is no available permission handling on Backstage per user.
  • There’s no available monitoring of the cluster integrated into the Backstage.

As the application grows, it’s worth investigating these technical aspects:

  • Developers should investigate whether GitHub workflows are the best tool versus switching to other existing tools or even creating a custom backend to handle the deployment logic.
  • Developers should investigate whether Backstage should remain the customer-facing application or switch to another framework.
  • Developers should set up interviews with beginner web developers to see if the product meets all their needs.
  • Developers should consider using Backstage components such as enabling Kubernetes and Prometheus plugins to monitor deployments.
  • Multiple GitHub workflows with many values passed between them using a tool such as Argo CD.
  • Developers should consider adding user account roles and management to Backstage.

7. Appendices

CODE Internal developer platform is a semester learning project led by Professor Adam Roe at CODE University. Please contact the designated person for questions.

Clone this wiki locally