Skip to content

Commit

Permalink
updated readme (#167)
Browse files Browse the repository at this point in the history
  • Loading branch information
@aaron-wu1
aaron-wu1 authored Feb 12, 2024
1 parent bf50bca commit c8d2a55
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 192 deletions.
146 changes: 1 addition & 145 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,31 +19,7 @@ A local version of Dirtviz can be started using `docker-compose.yml`. This will
docker compose up --build -d
```

At this point the portal is accessible at [http://localhost:3000/frontend](http://localhost:5006/frontend), but will likely show a blank page in your web browser and throw an error. This is due to the database being empty, therefore there is no data to display.

### Setup Connection Stings

The following table shows the default values to connect to the postgresql instance.

> **NOTE:** The database connection strings are set to static simple values as defined in `docker-compose.yml`. Do **_NOT_** use the default values for any publicly facing deployment.
| Name | Key | Value |
| -------- | ----------- | --------- |
| User | DB_USER | dirtviz |
| Password | DB_PASS | password |
| Hostname | DB_HOST | localhost |
| Port | DB_PORT | 5432 |
| Database | DB_DATABASE | dirtviz |

The following commands will set the necessary environment variables to allow utilities within Dirtviz to connect to the database.

```bash
export DB_USER=dirtviz
export DB_PASS=password
export DB_HOST=localhost
export DB_PORT=5432
export DB_DATABASE=dirtviz
```
At this point the portal is accessible at [http://localhost:3000/](http://localhost:3000/), but will likely show a blank page in your web browser and throw an error. This is due to the database being empty, therefore there is no data to display.

<!-- for reference OLD -->

Expand Down Expand Up @@ -72,126 +48,6 @@ Now some graphs should appear on the website and look like the following.

![Example screenshot of Dirtviz](.github/assets/img/dashboard.png)

## Running in Production

### TL;DR

To create a new development environment you must create a new branch. For each issue/feature, a new branch should be created. The environment will automatically be created and simultaneously deleted once the branch is deleted. The branch name **_MUST NOT_** be more than 55 characters, see [#52](https://github.com/jlab-sensing/DirtViz/issues/52). The website can be access via [https://dirtviz.jlab.ucsc.edu/dev/branch/portal/](https://dirtviz.jlab.ucsc.edu/dev/branch/portal/). The services `cs-http` and `http-api` are also assessable under [https://dirtviz.jlab.ucsc.edu/dev/branch/cs-http/](https://dirtviz.jlab.ucsc.edu/dev/branch/cs-http/) and [https://dirtviz.jlab.ucsc.edu/dev/branch/http-api/](https://dirtviz.jlab.ucsc.edu/dev/branch/http-api/) respectively. Replace `branch` with the name of the newly created branch.

### Overview

Dirtviz is intended to be deployed to a Kubernetes cluster. This way development work can be completed using a locally running Docker instance via `docker-compose` using the same containers that run in production. The only difference is that the Kubernetes deployments connects to a production database rather than the docker compose stack that runs the postgresql and Adminer instances as services. For our labs purposes, deployments are controlled through Github actions to build the containers then deploy the full stack to our locally hosted Kubernetes cluster.

### Github Setup

The Github repo is automatically setup to deploy to a Kubernetes instance. The only setup required is to setup Github secrets for the hostname, ssh key, and database connection parameters. The following gives the values needed to deploy to production.

| Name | Example | Description |
| --------------- | ----------- | ------------------------------------------ |
| USER | johndoe | User to perform k8s actions |
| HOSTNAME | example.com | Hostname/IP of production server |
| SSH_KNOWN_HOSTS | | Fingerprint to populate ~/.ssh/known_hosts |
| SSH_PRIVATE_KEY | | Private SSH key for USER |
| DB_HOST | example.com | Hostname/IP of database |
| DB_PORT | 5432 | Port of database |
| DB_USER | johndoe | User login for database |
| DB_PASS | password | Password for DB_USER |

### Accessing Deployments

> **NOTE**: The following talks about accessing a deployment in a generic manner, not distinguishing between production and development environments. Think of production as a development branch under the name `main`.
#### Updating a Deployment

A branch is deployed whenever there are changes pushed to the repository. It does **NOT** run tests or lint on the code before being pushed to production. The following are the steps for deployment.

1. Build `frontend`, `backend`, `cs-http`, and `http-api` containers
2. Generate k8s yaml files from templates
3. Apply configurations to k8s via ssh
<!-- 4. Apply database migrations. -->

> **NOTE**: Steps 3. and 4. are completed simultaneously.
#### Creating New Deployment

> **NOTE**: Due to name limits of resources in k8s the length of the branch name is limited to 55 characters. See [#52](https://github.com/jlab-sensing/DirtViz/issues/52) for more information.
Whenever a new branch on Github is created a new development environment with a separate database on the postgresql server and separate resources on k8s under a namespace that follows that of the branch. For example if a new branch is created with the name `test-branch`, a new postgres database is created with the name `test-branch` and a new k8s namespace is created with the name `dirtviz-test-branch`. The postgres database is created under the `DB_USER` defined in github secrets. The follows is the order of jobs run when a new branch is created:

1. Create k8s namespace
2. Create new database
3. Populate database with temporary data

Access to the running containers are available from the following paths. Internally, the deployment leverages the [NGINX Ingress Controller](https://github.com/kubernetes/ingress-nginx) to setup paths.

| Service | Path |
| -------- | ---------------------------------------------- |
| frontend | https://localhost/dirtviz/dev/BRANCH/frontend/ |
| cs-http | https://localhost/dirtviz/dev/BRANCH/cs-http/ |
| http-api | https://localhost/dirtviz/dev/BRANCH/http-api/ |

It is recommended to have a external (aka non k8s) nginx instance to forward traffic to production and development environments and simplify handling of ssl certificates. The following is a sample nginx configuration to do exactly this.

```
server {
listen 443 ssl default_server;
server_name YOUR_HOSTNAME;
include snippets/ssl.conf;
#access_log /tmp/bokeh.access.log;
#error_log /tmp/bokeh.error.log debug;
location / {
proxy_pass http://127.0.0.1:6080/dirtviz/main/portal/;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
}
location /integrations/ {
proxy_pass http://127.0.0.1:6080/dirtviz/main/;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
}
location /dev/ {
proxy_pass http://127.0.0.1:6080/dirtviz/;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
}
}
```

## Web server

Currently the frontend is served using an Nginx proxy server configured under `./frontend/frontend_server.conf`. The backend is accessed by route: [http://localhost:3000/frontend/api](http://localhost:5006/frontend/api) through a proxy pass.

## Integrations

Currently there are two integrations that allow for data to uploaded to the database.

### Chirpstack

Chirpstack handles data uploaded via the [LoRa](https://en.wikipedia.org/wiki/LoRa) protocol which works over extremely long ranges. The Chirpstack integration makes use of the HTTP integration that is a part of the Chirpstack software package. More information can be found at [https://www.chirpstack.io/application-server/integrations/http/](https://www.chirpstack.io/application-server/integrations/http/). Below is an example configuration within Chirpstack.

![Chirpstack HTTP Integration Configuration](.github/assets/img/screenshot_cs.png)

### HTTP

The HTTP integration is currently under development and will be changed significantly coming soon. Clients connected over Ethernet can send HTTP POST request to `http://localhost:8090` in CSV format. See the source code for formatting.

## FAQ

### How do I create database migrations?
Expand Down
61 changes: 14 additions & 47 deletions frontend/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
# Getting Started with Create React App

This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

## Available Scripts

In the project directory, you can run:
Expand All @@ -12,59 +8,30 @@ Runs the app in the development mode.\
Open [http://localhost:3000](http://localhost:3000) to view it in your browser.

The page will reload when you make changes.\
You may also see any lint errors in the console.

### `npm test`

Launches the test runner in the interactive watch mode.\
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.

### `npm run build`

Builds the app for production to the `build` folder.\
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.\
Your app is ready to be deployed!

See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.

### `npm run eject`
### `npm run test`

**Note: this is a one-way operation. Once you `eject`, you can't go back!**
Starts the vitest

If you aren't satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.
### `npm run test:ui`

Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you're on your own.
Starts the vitest with interactive UI

You don't have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn't feel obligated to use this feature. However we understand that this tool wouldn't be useful if you couldn't customize it when you are ready for it.
### `npm run coverage`

## Learn More
Runs a coverage test with vitest using --coverage flag

You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).
### `npm run lint`

To learn React, check out the [React documentation](https://reactjs.org/).
Runs lint

### Code Splitting
### `npm run lint:fix`

This section has moved here: [https://facebook.github.io/create-react-app/docs/code-splitting](https://facebook.github.io/create-react-app/docs/code-splitting)
Runs lint and fixes fixable lint errors

### Analyzing the Bundle Size

This section has moved here: [https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size](https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size)

### Making a Progressive Web App

This section has moved here: [https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app](https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app)

### Advanced Configuration

This section has moved here: [https://facebook.github.io/create-react-app/docs/advanced-configuration](https://facebook.github.io/create-react-app/docs/advanced-configuration)

### Deployment

This section has moved here: [https://facebook.github.io/create-react-app/docs/deployment](https://facebook.github.io/create-react-app/docs/deployment)
### `npm run build`

### `npm run build` fails to minify
Builds the app for production to the `build` folder.\

This section has moved here: [https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify](https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify)
The build is minified and the filenames include the hashes.\
Your app is ready to be deployed!

0 comments on commit c8d2a55

Please sign in to comment.