BU EC 528 Project Proposals

[jennydaman]

Below in this discussion thread, I will propose projects for the masters-level BU EC 528 class on cloud computing. Each project proposal will have a description and estimated difficulty.

[jennydaman]

Cloud Observability and ChRIS Plugins

Difficulty: 2/5
Key words: Prometheus, Grafana, Kubernetes, time-series database, front-end

Introduction

ChRIS brings medical apps to Kubernetes. Observability data, e.g. CPU and memory consumption over time of ChRIS plugins, is valuable information to know for ChRIS admins, everyday users, and plugin developers alike.

Learning Objectives

• Understand what observability is
• How to deploy things in Kubernetes
• How observability data is collected and stored
• Compare and contrast current observability solutions, e.g. Grafana stack v.s. OpenObserve v.s. Elastic

Deliverables

• Deploy ChRIS on OpenShift/Kubernetes
• Create observability pipelines (collection, parsing, storage) for data produced by ChRIS pipelines
• Create observability dashboards (Grafana or OpenObserve) for observability data

Reach Goals

• Integrate observability visualizations with ChRIS_ui, i.e. show average CPU, memory usage of a plugin next to its details in the plugins page

[jennydaman]

Logs management in Kubernetes

Difficulty: 5/5
Key words: streaming, object storage, Kubernetes

Introduction

In UNIX (Linux) programs write text output to "stdout" and "stderr." Typically, stdout and stderr are displayed to the terminal window. In Kubernetes, these logs are stored as timestamped lines of plaintext. Services such as Loki and OpenObserve offer querying and long-term storage of log data.

ChRIS is a platform which runs medical apps on Kubernetes. Its current log storage solution has problems described here: FNNDSC/ChRIS_ultron_backEnd#374
In summary:

• timestamps from logs are ignored
• logs are polled for and pulled
• logs must be a valid UTF-8 string and fewer than 4000 bytes (because they are store in Postgres)

Learning Objectives

• How to deploy stuff on Kubernetes
• Learn about the ways data can be stored (filesystem, database, object storage) and their trade-offs
• Understand the challenges of log storage
• Compare/contrast log storage solutions of Grafana Loki, Elastic, and OpenObserve

Deliverables

• Deploy ChRIS on OpenShift
• Deploy either Loki or OpenObserve on OpenShift using NooBaa ObjectBucketClaim for storage
• Configure log service to receive logs from Kubernetes

Reach Goals

• Integrate ChRIS backend with the log service so that logs can be retrieved from behind the ChRIS API
• Implement real-time streaming of log data to ChRIS_ui

[jennydaman]

Crunchy Postgres Operator and Disaster Recovery

Difficulty: 3/5
Key words: Postgres, Operators, backups, Kubernetes

Introduction

ChRIS is easy to deploy on Kubernetes using helm: https://github.com/FNNDSC/charts
Currently, the Helm charts deploy a single-replica Postgres database using a PersistentVolumeClaim for data storage. This should be improved to have the option to use the Crunchy Postgres operator.

Learning Objectives

• How to deploy stuff on Kubernetes
• Writing Helm charts
• Database administration including backups and disaster recovery

Deliverables

• Deploy ChRIS on OpenShift
• Use the Crunchy Postgres operator to deploy a highly-available, backed up Postgres service on OpenShift
• Connect ChRIS to the Postgres created by Crunchy
• Modify the Helm Chart so that it supports the Crunchy Postgres operator

Reach Goals

• Demonstrate fault-tolerance and disaster recovery of solution by simulating database corruption than recovering from a backup.
• Modify the Helm Chart so that is supports the option of choosing either a single-replica StatefulSet Postgres deployment using PVC or the Crunchy Postgres Operator

[jennydaman]

Advanced Searching in the Cloud

Difficulty: 2/5
Key words: search engines, Kubernetes, Helm

Introduction

The ChRIS backend's storage and database hold unstructured data (such as logs) and structured data (such as past job info), respectively. The API provides some searching functionality, however it is limited in scope and the supported queries are hard-coded. Search engine databases are databases which specialize in searching over data. Integration with a search engine can vastly improve the functionality of ChRIS by providing efficient search with features such as fuzzy search and searching across fields.

Learning Objectives

• How to deploy stuff on Kubernetes
• Basic knowledge on relational databases
• How to deploy a search engine database

Deliverables

• Deploy ChRIS on OpenShift and generate some example user data
• Research search engine database solutions, such as: ElasticSearch, Meilisearch, Quickwit
• Decide whether to focus on unstructured or structured data
• Use a search engine database to perform searches over data in ChRIS
• Integrate the search engine with the ChRIS backend

Reach Goals

• Add the search engine database to the Helm chart