Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User Management backend - Base #26

Closed
wants to merge 27 commits into from
Closed
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ee7e3ef
Created db models and configured app
DoRTaL94 Apr 10, 2020
1b7aaf8
adding .gitignore
ahinoamta Apr 11, 2020
345daf4
editing the models User, Token and ResponseUser
ahinoamta Apr 11, 2020
1bf76b8
adding db_hanlder
ahinoamta Apr 11, 2020
92482f1
Updating db handler
DoRTaL94 Apr 11, 2020
5bde4d5
Added required packages to setup.sh and forwarded additional port for…
DoRTaL94 Apr 11, 2020
daf8484
added new line
ahinoamta Apr 12, 2020
82e7381
Merge branch 'master' into user-management
ahinoamta Apr 12, 2020
4a1e0ce
added necessary packages to requirements.txt file
ahinoamta Apr 12, 2020
ba95634
CR fixes
ahinoamta Apr 12, 2020
aeac940
Modified user, db_handler and token files
DoRTaL94 Apr 12, 2020
9580175
added abstract base class BaseStorageHandler that DbHandler derives from
ahinoamta Apr 13, 2020
21dad4f
Fixed token query
DoRTaL94 Apr 14, 2020
e605ac5
Modified generic dbhandler
DoRTaL94 Apr 15, 2020
ced0fc6
Added packages versions in requirements file
DoRTaL94 Apr 15, 2020
24650eb
Changed flask_init name
DoRTaL94 Apr 16, 2020
fa860bb
Moved to standard way of configuring DB connection string
DoRTaL94 Apr 16, 2020
75f3a5c
Removed unnecessary DBHandler.
DoRTaL94 Apr 16, 2020
54500cc
Added config file
DoRTaL94 Apr 16, 2020
f6c312f
Fixed token model repr function
DoRTaL94 Apr 20, 2020
61e4b1c
Updated setup.sh and changed app file name
DoRTaL94 Apr 16, 2020
7deba85
created test that verifies application is running
timshik Apr 18, 2020
9937b84
Restructured project layout
DoRTaL94 Apr 21, 2020
922a891
Merged both flask servers into one
DoRTaL94 Apr 21, 2020
2f2d231
Added use of Flask-Migrate extension
DoRTaL94 Apr 21, 2020
8fb4f37
Changed Vagrantfile
DoRTaL94 Apr 16, 2020
93122d9
Updated gitignore
DoRTaL94 Apr 20, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@

#Build result

*.log
_pycache_/
*/*.log
/*.log
simzacks marked this conversation as resolved.
Show resolved Hide resolved
/.vagrant/*
*/.idea/*
*.box
*.pyc
_pycache_/
8 changes: 4 additions & 4 deletions Vagrantfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
FLASK_PORT=5000
POSTGRESQL_PORT=5432
CLIENT_PORT=5000
BE_PORT=3000

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.provision :shell, path: "setup.sh"
config.vm.network :forwarded_port, guest: FLASK_PORT, host: FLASK_PORT
config.vm.network :forwarded_port, guest: POSTGRESQL_PORT, host: POSTGRESQL_PORT
config.vm.network :forwarded_port, guest: CLIENT_PORT, host: CLIENT_PORT
DoRTaL94 marked this conversation as resolved.
Show resolved Hide resolved
config.vm.network :forwarded_port, guest: BE_PORT, host: BE_PORT
DoRTaL94 marked this conversation as resolved.
Show resolved Hide resolved
config.vm.provider "virtualbox" do |v|
v.gui = false
v.name = "Edison_test"
Expand Down
16 changes: 16 additions & 0 deletions backend/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
import os


db_role = 'postgres'
db_password = 'edison'
db_url = '0.0.0.0'
DoRTaL94 marked this conversation as resolved.
Show resolved Hide resolved
db_name = 'edison'

# Put app and db here so the entire app could import them.
app = Flask(__name__)
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://{}:{}@{}/{}'.format(db_role, db_password, db_url, db_name)
DoRTaL94 marked this conversation as resolved.
Show resolved Hide resolved
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db = SQLAlchemy(app)
22 changes: 22 additions & 0 deletions backend/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from flask_restful import Api
from flask_jwt_extended import JWTManager
import secrets
import backend

# API description in swagger - https://app.swaggerhub.com/apis/DoRTaL94/UserManagment/1.0.0

app = backend.app
# Creates all tables in app. Table's name defined with __tablename__ variable.
backend.db.create_all()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may not be a good idea to do this every time you start your app. Consider what would happen in production when your database contains production information and your app restarts from some reason...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

db.create_all() creates only tables that not exists in the db. If a table exists in the db it won't be changed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the process for updating the structure of an existing table, when you decide you need to add a field to a model class for example?

Copy link
Collaborator Author

@DoRTaL94 DoRTaL94 Apr 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, currently we have two choices that I can think of:

  1. Delete the relevant table, and when db.create_all being called again it will create the updated table with the new column. (Not a good option and could be used only in development)
  2. New field = new column in the table, so we could add the new column manually by connecting to PostgreSQL from the command line.

For more advanced solution we could use Flask-Migrate extension that handles SQLAlchemy database migrations for Flask applications using Alembic. It can emit ALTER statements to a database in order to change the structure of tables.

# Enables response message for unauthenticated requests
app.config['PROPAGATE_EXCEPTIONS'] = True
# JWTManager uses this secret key for creating tokens
app.config['JWT_SECRET_KEY'] = secrets.token_hex(24)
# This tells the JWTManager to call 'check_if_token_in_blacklist' function below on every request
app.config['JWT_BLACKLIST_ENABLED'] = True
# We're going to check if both access_token and refresh_token are black listed
app.config['JWT_BLACKLIST_TOKEN_CHECKS'] = ['access', 'refresh']
api = Api(app)
# Creation of Json-Web-Token manager.
# In order to reach secured endpoints client should add an authorization header with the value Bearer <token>.
jwt = JWTManager(app)
2 changes: 2 additions & 0 deletions backend/models/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
from .token import Token
from .user import User
12 changes: 12 additions & 0 deletions backend/models/token.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from backend import db


class Token(db.Model):
__tablename__ = 'token_blacklist'

id = db.Column(db.Integer, primary_key=True)
jti = db.Column(db.String(150), nullable=False, unique=True)
creation_timestamp = db.Column(db.TIMESTAMP(timezone=False), nullable=False)

def __repr__(self):
return f"<Token: jti: {self.jti}, creation time: {self.creation_timestamp}."
simzacks marked this conversation as resolved.
Show resolved Hide resolved
25 changes: 25 additions & 0 deletions backend/models/user.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from backend import db


class User(db.Model):
__tablename__ = 'users'
__table_args__ = {'extend_existing': True}

id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(50), nullable=False, unique=True)
password = db.Column(db.String(150), nullable=False)
first_name = db.Column(db.String(50), nullable=False)
last_name = db.Column(db.String(50), nullable=False)
email = db.Column(db.String(150), nullable=False)

def to_json(self):
return {
"username": self.username,
"first_name": self.first_name,
"last_name": self.last_name,
"email": self.email
}

def __repr__(self):
return f"<User: id = {self.id}, first_name = {self.first_name}, " \
f"last_name = {self.last_name}, username = {self.username}, email = {self.email}>"
29 changes: 29 additions & 0 deletions backend/services/base_storage_handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from abc import ABC, abstractmethod
from backend import db
from typing import Dict

class BaseStorageHandler(ABC):
DoRTaL94 marked this conversation as resolved.
Show resolved Hide resolved

@abstractmethod
def get_by_filters(self, model: db.Model, filters: Dict[str, str]):
pass

@abstractmethod
def get_by_id(self, model: db.Model, _id: int):
pass

@abstractmethod
def get_all(self, model: db.Model):
pass

@abstractmethod
def add(self, model: db.Model):
pass

@abstractmethod
def delete(self, model: db.Model):
pass

@abstractmethod
def update(self):
pass
27 changes: 27 additions & 0 deletions backend/services/db_handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
from backend import db
from datetime import datetime
from backend.services.base_storage_handler import BaseStorageHandler
from typing import Dict


class DBHandler(BaseStorageHandler):
DoRTaL94 marked this conversation as resolved.
Show resolved Hide resolved

def get_by_filters(self, model: db.Model, filters: Dict[str, str]):
return model.query.filter_by(**filters).first()

def get_by_id(self, model: db.Model, _id: int):
return model.query.get(_id)

def get_all(self, model: db.Model):
return model.query.all()

def add(self, model: db.Model):
db.session.add(model)
db.session.commit()

def delete(self, model: db.Model):
db.session.delete(model)
db.session.commit()

def update(self):
db.session.commit()
5 changes: 5 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
Flask==1.1.1
flask-restful
simzacks marked this conversation as resolved.
Show resolved Hide resolved
flask-jwt-extended
passlib
flask-sqlalchemy
psycopg2-binary
16 changes: 15 additions & 1 deletion setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,20 @@ sudo apt-get install -y postgresql postgresql-contrib
echo "install requirements"
pip3 install -r /vagrant/requirements.txt

simzacks marked this conversation as resolved.
Show resolved Hide resolved
echo "installing postgresql"
sudo apt-get install -y postgresql postgresql-contrib

echo "configuring database"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Executing this kind of SQL manually during the setup stage means you get a new database every time you bring up the system - this might work out in development but not in staging or production.

The technique for managing database configuration over time with Git is called "database migrations" here is a post introducing this concept:
https://rollout.io/blog/database-migration/
The examples there are for Ruby-on-Rails but the same concepts apply to Flask and SQLAlchemy

Here is a DB migration framework you can use with SQLAlchemy:
https://alembic.sqlalchemy.org/en/latest/

Copy link
Collaborator Author

@DoRTaL94 DoRTaL94 Apr 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Executing this kind of SQL manually during the setup stage means you get a new database every time you bring up the system

Should we create the database manually ? But on the next vagrant up we'll need to create it again... That is why I added those lines to setup.sh so we wouldn't need to do it over and over again.

The technique for managing database configuration over time with Git is called "database migrations"

Doesn't it migrate only the tables schema without the data ? and if so on vagrant destroy all the data will be gone. How does it different from creating a new database on every vagrant up ?

Edit: I've added Flask-Migrate extension to the project that handles SQLAlchemy database migrations for Flask applications using Alembic. It super easy to use and it configures Alembic in the proper way to work with our Flask and SQLAlchemy application.
Regarding of the question I asked:

Doesn't it migrate only the tables schema without the data ? and if so on vagrant destroy all the data will be gone.

I solved this issue of the data being destroyed by saving it inside the vagrant synced folder.
I used vagrant triggers so just before vagrant destroy is executed I save the data and
after vagrant up is executed I restore the data.
But I still need the SQL commands inside setup.sh to be executed so the database will be created.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I squashed some commits so this file is no longer reachable through this commit.
This file could be found here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what vagrant workflow you're aiming for, but I think the current solution looks too complex and may not be needed.

They way I think of it, your workflow should include several commands:

  1. A command that brings up a dev environment from scratch on a new machine (Typically the first time you run vagrant up)
  2. A command that resyncs your code changes to the dev environment (May not be needed if your intended workflow is to develop directly on the Vagrant machine). It should probably also do things like:
    1. Run any DB migrations that were not yet applied to the DB
    2. Restart the application server or otherwise have some other way to make it reload the code if it does not do that automatically
  3. A command that shuts down the environment without destroying the data in it (Typically vagrant halt)
  4. A command that brings an existing environment back up (Typically vagrant up when its not the first time you've ran it)
  5. A command that clears up the environment so it can be created from scratch (Typically vagrant destroy)

You can use different combinations of Vagrant commands and scripts to achieve the commands above, but you should have them available because they are all needed in a typical development workflow.

sudo su postgres <<POSTGRESQL_COMMANDS
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIR Postgeas has some command-line shortcuts for DB and user creation, could be cleaner to use those then inlined SQL.

psql
CREATE DATABASE edison;
ALTER ROLE postgres WITH PASSWORD 'edison';
POSTGRESQL_COMMANDS

echo "running flask_init.py"
export FLASK_APP=/vagrant/flask_init.py
python3 -m flask run --host=0.0.0.0 >> /vagrant/log.log 2>&1 &
flask run -h 0.0.0.0 -p 5000 >> /vagrant/flask_init.log 2>&1 &

echo "running app.py"
export FLASK_APP=/vagrant/backend/app.py
flask run -h 0.0.0.0 -p 3000 >> /vagrant/app.log 2>&1 &
simzacks marked this conversation as resolved.
Show resolved Hide resolved