Skip to content

Latest commit

 

History

History
827 lines (659 loc) · 35.5 KB

README.md

File metadata and controls

827 lines (659 loc) · 35.5 KB

Budget Management Backend API

Welcome to the Budget Management API, a robust backend application designed to manage budgets, expenses, users, orders, notifications, and more. This API integrates cutting-edge technologies and supports advanced features like gRPC, GraphQL, WebSockets, Elasticsearch, PostgreSQL, MySQL, Redis, RabbitMQ, Kafka, and Docker.

Moreover, it also includes a CLI tool for interacting with the backend and a comprehensive Swagger documentation for testing and exploring the API.

Below is a comprehensive guide to setting up, running, and utilizing this API.

Table of Contents

  1. Overview
  2. Live Deployment
  3. Technologies Used
  4. Project Structure
  5. Setup Instructions
  6. Available Endpoints
  7. Schemas
  8. Features and Integrations
  9. Environment Variables
  10. CLI Usage
  11. Demo Frontend UI
  12. Swagger Documentation
  13. GraphQL Integration
  14. NGINX Configuration
  15. gRPC Integration
  16. Dockerization
  17. Kubernetes Deployment
  18. Continuous Integration and Deployment with Jenkins
  19. Testing
  20. Contributing
  21. Author

Overview

The Budget Management API is designed to handle complex budget management requirements, including:

  • Budget and expense tracking.
  • User management and authentication.
  • Real-time notifications via WebSockets.
  • Asynchronous task handling using RabbitMQ and Kafka.
  • Advanced search capabilities with Elasticsearch.
  • CLI operations for direct interaction with the system.
  • Compatibility with modern cloud environments like Docker and Kubernetes.

The purpose of this API is to demonstrate the capabilities of modern backend technologies and provide a foundation for building scalable, real-time applications. It can be used as a reference for developers looking to implement similar features in their projects. Simply clone the repository, set up the environment, and start building the frontend or additional functionality on top of the existing API!

Live Deployment

The Budget Management API is deployed live at https://budget-management-backend-api.onrender.com.

Additionally, the frontend UI is also available, hosted at https://budget-manage-app.vercel.app.

You can access the API and test the endpoints directly from the browser. Feel free to use the API for your own projects or applications. Simply add some attribution to the original repository and the creator. Also, be sure that you use your own credentials and tokens, otherwise your data may clash with mine and other users' data!

Note: Be mindful of the rate limits and usage policies when testing the live API. Additionally, because the API is hosted on the free plan of Render, it may take a while (1-2 minutes) to wake up if it has been inactive for some time. Kindly be patient during this process!

Backup Frontend: https://budget-management-system.netlify.app

Deployment and Technology Status

API Status MongoDB Redis PostgreSQL RabbitMQ Kafka Elasticsearch gRPC GraphQL Swagger Docs Docker Kubernetes WebSockets Nginx OpenAPI Jenkins CI/CD Prometheus Grafana Vercel Render Netlify

Technologies Used

Technology Purpose
Node.js Core application framework.
Express.js Web application framework for building APIs.
MongoDB Primary NoSQL database for managing budgets and expenses.
PostgreSQL Relational database for transaction logs.
MySQL Optional relational database support.
Redis In-memory database for caching.
RabbitMQ Message broker for task queuing.
Kafka Distributed event streaming platform.
Elasticsearch Advanced search engine for querying data.
gRPC High-performance remote procedure call framework.
GraphQL Query language for fetching and manipulating data.
WebSocket Real-time communication for notifications.
Swagger/OpenAPI API documentation and testing.
Docker Containerization for easy deployment.
Kubernetes Orchestrating containerized applications at scale.
Nginx Reverse proxy and load balancer.
Prometheus Monitoring and alerting toolkit.
Grafana Observability and visualization platform.
Jenkins CI/CD pipeline for automated testing and deployment.

Project Structure

Budget-Management-Backend-API/
├── .env                           # Environment variables configuration
├── .env.example                   # Example environment configuration file
├── .gitignore                     # Git ignore file
├── .prettierrc                    # Prettier configuration for code formatting
├── LICENSE                        # License information
├── README.md                      # Documentation for the project
├── app.test.js                    # Main test file for application
├── cli.js                         # CLI tool for interacting with the backend
├── docker-compose.yml             # Docker Compose configuration
├── Dockerfile                     # Dockerfile for containerizing the application
├── grpcServer.js                  # gRPC server implementation
├── index.js                       # Main entry point for the application
├── nodemon.json                   # Nodemon configuration file
├── openapi.yaml                   # OpenAPI specification for the API
├── package.json                   # NPM package configuration file
├── start.sh                       # Script to start the application
├── prometheus.yml                 # Prometheus configuration for monitoring
├── redis-mongo-flow/              # Directory for Redis-Mongo integration flow
│   ├── app.js                     # Express app for Redis-Mongo flow
│   ├── config.js                  # Configuration for Redis-Mongo flow
│   ├── package.json               # NPM configuration for this module
│   ├── README.md                  # Documentation specific to Redis-Mongo flow
│   ├── seed.js                    # Data seeder for Redis-Mongo flow
│   ├── test.js                    # Test file for Redis-Mongo flow
├── round-robin/                   # Directory for round-robin load balancer
│   ├── config.js                  # Configuration for round-robin implementation
│   ├── index.js                   # Main entry point for round-robin logic
│   ├── README.md                  # Documentation for round-robin functionality
├── proto/                         # Protocol Buffers directory
│   ├── budget.proto               # gRPC proto file for budgets
├── nginx/                         # NGINX configuration directory
│   ├── docker-compose.yml         # Docker Compose for NGINX
│   ├── Dockerfile                 # Dockerfile for NGINX
│   ├── nginx.conf                 # NGINX configuration file
│   ├── start_nginx.sh             # Script to start NGINX
│   ├── README.md                  # Documentation for NGINX
├── docs/                          # Documentation directory
│   ├── swaggerConfig.js           # Swagger configuration for API docs
├── graphql/                       # GraphQL-related files
│   ├── schema.js                  # GraphQL schema definition
├── services/                      # Services and utilities
│   ├── dataSeeder.js              # Seeder for MongoDB
│   ├── elasticService.js          # Elasticsearch client and utility functions
│   ├── jwtService.js              # JSON Web Token (JWT) utilities
│   ├── postgresService.js         # PostgreSQL client and utilities
│   ├── rabbitMQService.js         # RabbitMQ client and utilities
│   ├── redisService.js            # Redis client and utilities
│   ├── websocketService.js        # WebSocket server and utilities
├── controllers/                   # Route controllers for the API
│   ├── authController.js          # Authentication-related endpoints
│   ├── budgetController.js        # Budget management endpoints
│   ├── customerController.js      # Customer management endpoints
│   ├── expenseController.js       # Expense management endpoints
│   ├── notificationController.js  # Notification management endpoints
│   ├── orderController.js         # Order management endpoints
│   ├── searchController.js        # Search-related endpoints
│   ├── taskController.js          # Task management endpoints
│   ├── transactionController.js   # Transaction log endpoints
│   ├── userController.js          # User profile management endpoints
├── middleware/                    # Middleware utilities
│   ├── authMiddleware.js          # JWT authentication middleware
├── models/                        # Mongoose schemas
│   ├── budget.js                  # Schema for budgets
│   ├── customer.js                # Schema for customers
│   ├── expense.js                 # Schema for expenses
│   ├── order.js                   # Schema for orders
│   ├── task.js                    # Schema for tasks
│   ├── user.js                    # Schema for users
├── routes/                        # Express router files
│   ├── authRoutes.js              # Routes for authentication
│   ├── budgetRoutes.js            # Routes for budgets
│   ├── customerRoutes.js          # Routes for customers
│   ├── expenseRoutes.js           # Routes for expenses
│   ├── graphqlRoutes.js           # Routes for GraphQL
│   ├── index.js                   # Main router entry point
│   ├── notificationRoutes.js      # Routes for notifications
│   ├── orderRoutes.js             # Routes for orders
│   ├── searchRoutes.js            # Routes for Elasticsearch
│   ├── taskRoutes.js              # Routes for tasks
│   ├── transactionRoutes.js       # Routes for transactions
│   ├── userRoutes.js              # Routes for user profiles
├── views/                         # Static assets and templates
│   ├── android-chrome-192x192.png # Android Chrome app icon
│   ├── android-chrome-512x512.png # Android Chrome high-res icon
│   ├── apple-touch-icon.png       # Apple Touch icon
│   ├── favicon.ico                # Favicon
│   ├── favicon-16x16.png          # 16x16 favicon
│   ├── favicon-32x32.png          # 32x32 favicon
│   ├── home.html                  # HTML template for homepage
│   ├── manifest.json              # Web app manifest

Setup Instructions

Prerequisites

  • Node.js (>= 16)
  • Docker and Docker Compose (if using containerized setup)
  • MongoDB, PostgreSQL, MySQL, RabbitMQ, Redis, and Elasticsearch services.

Local Installation

  1. Clone the repository:

    git clone https://github.com/hoangsonww/Budget-Management-Backend-API.git
cd Budget-Management-Backend-API

  2. Install dependencies:

    npm install

  3. Set up environment variables:

  • Create a .env file in the root directory: 
    MONGO_DB_URI=mongodb://localhost:27017/budget_manager
POSTGRES_URI=postgres://user:password@localhost:5432/budget_manager
REDIS_URL=redis://localhost:6379
RABBITMQ_URL=amqp://localhost
KAFKA_BROKER=localhost:9092
JWT_SECRET=your_secret_key
  • Replace placeholders with your actual configuration.

  1. Start the application:

    npm start

  2. Access the application:

  • API: http://localhost:3000
  • Swagger: http://localhost:3000/docs

Available Endpoints

Endpoint Method Description
/api/auth/register POST Register a new user.
/api/auth/login POST Login and receive a JWT token.
/api/auth/logout POST Logout and invalidate the token.
/api/auth/verify-email POST Verify the user's email address.
/api/auth/reset-password POST Reset the user's password.
/api/users/profile GET Get the authenticated user's profile.
/api/budgets GET Get all budgets.
/api/budgets POST Create a new budget.
/api/budgets/:id GET Get a specific budget.
/api/budgets/:id PUT Update a budget.
/api/budgets/:id DELETE Delete a budget.
/api/customers GET Get all customers.
/api/customers POST Create a new customer.
/api/customers/:id GET Get a specific customer.
/api/customers/:id PUT Update a customer.
/api/customers/:id DELETE Delete a customer.
/api/expenses GET Get all expenses.
/api/expenses POST Add a new expense.
/api/expenses/:budgetId GET Get all expenses for a budget.
/api/expenses/:id PUT Update an expense.
/api/expenses/:id DELETE Delete an expense.
/api/orders GET Get all orders.
/api/orders POST Create a new order.
/api/orders/:id GET Get a specific order.
/api/orders/:id PUT Update an order.
/api/orders/:id DELETE Delete an order.
/api/transactions GET Get all transactions.
/api/transactions POST Add a new transaction.
/api/transactions/:id GET Get a specific transaction.
/api/transactions/:id PUT Update a transaction.
/api/transactions/:id DELETE Delete a transaction.
/api/tasks GET Get all tasks.
/api/tasks POST Add a new task.
/api/tasks/:id GET Get a specific task.
/api/tasks/:id PUT Update a task.
/api/tasks/:id DELETE Delete a task.
/api/graphql POST Perform a GraphQL query.
/api/notifications POST Send a real-time notification.
/api/search POST Search for expenses using Elasticsearch.

Additionally, the root / endpoint provides a welcome message and information about the API.

Available Endpoints

More endpoints and features are available in the API. Refer to the Swagger documentation for detailed information.

Schemas

User

Field Type Description
username String Unique username.
email String Unique email address.
password String Hashed password.
createdAt Date User creation date.

Budget

Field Type Description
name String Budget name.
limit Number Budget limit.
createdAt Date Budget creation date.

Expense

Field Type Description
budgetId String ID of the associated budget.
description String Expense description.
amount Number Expense amount.
createdAt Date Expense creation date.

Order

Field Type Description
customerId String ID of the associated customer.
amount Number Order amount.
status String Order status.
createdAt Date Order creation date.

Customer

Field Type Description
name String Customer name.
email String Customer email address.
phone String Customer phone number.

Task

Field Type Description
description String Task description.
status String Task status.
createdAt Date Task creation date.

Features and Integrations

gRPC

  • High-performance RPC framework.
  • Start the gRPC server using: 
    npm start

GraphQL

  • Flexible data queries and mutations.
  • Access the GraphQL endpoint at http://localhost:3000/graphql.

WebSocket

  • Real-time notifications for clients.
  • Notifications can be sent using the /api/notifications endpoint or CLI.

Docker

  • Build and run the app with Docker: 
    docker-compose up --build

Elasticsearch

  • Advanced search for expenses.
  • Search endpoint: /api/search/expenses.

RabbitMQ

  • Asynchronous task handling.
  • Use the budget-manager CLI to add tasks.
  • Tasks are processed in the background.

Kafka

  • Distributed event streaming platform.
  • Kafka broker URL: localhost:9092.
  • Kafka producer and consumer are integrated.

Redis

  • In-memory caching for improved performance.
  • Redis URL: redis://localhost:6379.
  • Caching is used for user sessions and other data.

PostgreSQL

  • Relational database for transaction logs.
  • PostgreSQL URL: postgres://user:password@localhost:5432/budget_manager.
  • Used for storing transaction logs and other relational data.
  • MySQL is also supported as an alternative.

MongoDB

  • Primary NoSQL database for managing budgets and expenses.
  • MongoDB URL: mongodb://localhost:27017/budget_manager.
  • Used for storing budgets, expenses, and user data.

Nginx

  • Reverse proxy and load balancer.
  • Nginx configuration is included in the nginx directory.
  • Load balancing can be configured for multiple instances.
  • SSL termination and caching can be added.

Kubernetes

  • Deployment manifests are available in the kubernetes directory.
  • Deploy the application to a Kubernetes cluster using: 
    kubectl apply -f kubernetes/

Prometheus and Grafana

  • Monitoring and observability tools.
  • Prometheus configuration is available in prometheus.yml.
  • Grafana can be used for visualization and monitoring.
  • Metrics and dashboards can be configured.
  • Monitor the health and performance of the API.

Services Interaction

The Budget Management API interacts with various services and databases to provide a comprehensive backend solution. The architecture includes a frontend UI, a CLI tool, an API gateway, a gRPC server, and multiple external services. Here is a high-level overview of the service interaction:

          +--------------------+               +------------------+
          |    Frontend UI     |               |                  |
          |    (not impl.)     |               |  CLI Tool / gRPC |
          +--------------------+               +------------------+
                    |                               |
                    | HTTP/GraphQL Requests         | CLI Commands / gRPC Calls
                    |                               |
          +--------------------+                 +------------------+
          |   API Gateway /    |<--------------->|    gRPC Server   |
          |    Express.js      |                 +------------------+
          +--------------------+
                    |
                    | RESTful API / WebSocket / GraphQL Responses
                    |
          +--------------------+
          |  Application Core  |
          |--------------------|
          |  Controllers /     |
          |  Services          |
          +--------------------+
            |        |         |
     +------+        |         +-------------+
     |               |                       |
+----------+   +-----------+           +----------------+
| MongoDB  |   | PostgreSQL |          | Elasticsearch  |
| NoSQL DB |   | Relational |          | Search Engine  |
+----------+   +-----------+           +----------------+
     |               |                       |
     |               |                       |
+----------+    +-------------+      +----------------+
|   Redis  |    | RabbitMQ /  |      | Kafka (Event   |
|   Cache  |    | Kafka Queue |      |   Streaming)   |
+----------+    +-------------+      +----------------+
     |              |                        |
     +--------------|-(Asynchronous Tasks)---+
                    |
          +----------------------+
          |   External Services  |
          |  (Email, SMS, etc.)  |
          +----------------------+

Environment Variables

Ensure your .env file looks like this before getting started:

# Server Configuration
PORT=

# MongoDB Configuration
MONGO_DB_URI=
MONGO_DB_USERNAME=
MONGO_DB_PASSWORD=

# Redis Configuration
REDIS_HOST=
REDIS_PORT=
REDIS_URL=

# RabbitMQ Configuration
RABBIT_MQ_HOST=
RABBITMQ_URL=

# Kafka Configuration
KAFKA_BROKER=

# JWT Secret Key
JWT_SECRET=

# Elasticsearch Configuration
ELASTIC_SEARCH_URL=

# PostgreSQL Configuration
POSTGRES_URL=

CLI Usage

The budget-manager CLI provides a convenient way to interact with the application from the command line.

Follow these steps to use the CLI:

  1. Install globally:

    npm link

  2. Use commands:

    budget-manager seed
budget-manager notify "Hello!"
budget-manager add-task "Task description"

  3. View available commands:

     budget-manager --help

This will display a list of available commands and options:

budget-manager --help
Usage: budget-manager [options] [command]

A CLI for managing budgets, tasks, orders, and more.

Options:
  -V, --version                        output the version number
  -h, --help                           display help for command

Commands:
  seed                                 Seed the MongoDB database with initial data
  notify <message>                     Send a real-time notification to WebSocket clients
  list-budgets                         List all budgets in the database
  add-task <description>               Add a new task to the task queue
  list-orders                          List all orders in the database
  add-order <customerId> <amount>      Add a new order
  list-customers                       List all customers in the database
  add-customer <name> <email> [phone]  Add a new customer
  search-expenses <query>              Search for expenses using a query
  graphql-query <query>                Perform a GraphQL query
  help [command]                       display help for command

Examples:
  $ budget-manager seed
  $ budget-manager notify "Hello World!"
  $ budget-manager list-budgets
  $ budget-manager add-task "New Task Description"
  1. View version information: 
    budget-manager --version

The CLI provides a simple way to interact with the backend API and perform various operations. It can be used for testing, debugging, and managing the application without a frontend interface or using the Swagger documentation.

Demo Frontend UI

The Budget Management API includes a demo frontend UI for interacting with the backend.

It gives developers an idea of how the API can be used in a real-world application. The frontend UI is built using React, Redux, and Material-UI components.

To run the frontend UI, follow these steps:

  1. Navigate to the frontend directory:

    cd frontend

  2. Install dependencies: (use npm install --legacy-peer-deps if you encounter peer dependency issues)

     npm install

  3. Start the development server:

    npm start

  4. Access the frontend UI at http://localhost:3001 (or whichever port is specified in the console).

Alternatively, it is also deployed live at https://budget-manage-app.vercel.app. Feel free to use the live version for testing and exploration.

For more information, refer to the Frontend README in the frontend directory to learn about the frontend UI components, features, and usage.

UI Images

Here are some screenshots of the frontend UI:

Home:

Frontend UI

Dashboard:

Frontend UI

Budgets:

Frontend UI

Expenses:

Frontend UI

Profile:

Frontend UI

Login:

Frontend UI

Register:

Frontend UI

Forgot Password:

Frontend UI

Users:

Frontend UI

Swagger Documentation

  • Comprehensive API documentation is available at /docs.
  • Includes all endpoints, schemas, and examples.
  • Use Swagger UI to test and interact with the API.
  • The Swagger UI looks like this:

Swagger UI

GraphQL Integration

  • The Budget Management API supports GraphQL queries and mutations.
  • Access the GraphQL endpoint at http://localhost:3000/api/graphql.
  • Use the GraphiQL interface to interact with the API.
  • The GraphiQL interface looks like this:

GraphiQL

  • Interact with the API using GraphQL queries and mutations. Examples include:
query {
  budgets {
    id
    name
    limit
    createdAt
  }
}

Or:

query {
  expenses(budgetId: "64c9f8f2a73c2f001b3c68f4") {
    id
    description
    amount
    budgetId
    createdAt
  }
}

When you run these queries, you will receive a response with the requested data. GraphQL provides a flexible and efficient way to fetch and manipulate data from the backend. Here is an example:

GraphQL

NGINX Configuration

  • The nginx directory contains an Nginx configuration for reverse proxy and load balancing.
  • Use Nginx to route requests to multiple instances of the API.
  • Configure SSL termination and caching for improved performance.
  • The Nginx configuration looks like this:
server {
    listen 80;
    server_name localhost;

    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

gRPC Integration

The Budget Management API includes support for gRPC to enable high-performance remote procedure calls.

Getting Started

  1. Start the gRPC Server: Run the following command:

    node grpcServer.js

  2. Use the gRPC Client: Execute the client to interact with the server:

    node grpcClient.js

  3. Proto File: The .proto file for defining gRPC services and messages is located in the protos directory.

That's it! Your gRPC server and client should now be operational. 🚀

Dockerization

The Budget Management API can be run in a Docker container for easy deployment and scaling.

You can build and run the app using Docker Compose:

docker-compose up --build

Kubernetes Deployment

  1. Create Kubernetes manifests for the services.

  2. Deploy to a cluster:

    kubectl apply -f kubernetes/

  3. Access the application using the service URL.

Continuous Integration and Deployment with Jenkins

The Budget Management API includes a Jenkins pipeline for continuous integration and deployment.

  1. Pipeline Configuration: The Jenkinsfile defines the CI/CD pipeline stages, including code checkout, dependency installation, testing, building, and deployment. Add it to the root of the project.

  2. Job Setup: Create a pipeline job in Jenkins, point it to the repository, and configure it to use the Jenkinsfile.

  3. Automated Testing: The pipeline runs npm test to ensure all tests pass before proceeding to the build or deployment stages.

  4. Environment Variables: Use Jenkins environment variables to securely manage secrets like API keys and credentials for services such as MongoDB, Redis, or Render.

  5. Deployment: The pipeline supports deploying the application using Render or directly to a server using SSH and PM2.

  6. Webhooks: Integrate GitHub/GitLab webhooks to trigger builds automatically on code changes.

  7. Notifications: Add Slack or email notifications in the pipeline to inform team members about build and deployment statuses.

Testing

The Budget Management API includes unit tests for all endpoints and services.

To run the tests, use the following command:

npm test

The test results will be displayed in the console.

Contributing

Contributions are welcome! Please fork the repository, create a feature branch, and submit a pull request:

  1. Fork the repository.
  2. Create a new branch (git checkout -b feature-branch).
  3. Make changes and commit them (git commit -am 'Add new feature').
  4. Push to the branch (git push origin feature-branch).
  5. Create a new pull request. We will review your changes and merge them if they look good.

Author

Thank you for using the Budget Management API. For questions, feedback, or support, please open an issue or contact me directly.

Created with ❤️ by Son Nguyen in 2024. All rights reserved.

⬆️ Back to Top