Merge pull request #344 from Progress1/docker_ver

Removed .env password support

docker/.env.example

@@ -14,12 +14,6 @@ TZ=Europe/Bratislava
 # Force language
 VUE_APP_TARANIS_NG_LOCALE=""
 
-# Taranis-NG now supports Docker secrets. Use this in production! More here: https://docs.docker.com/compose/use-secrets/
-# These variables bellow are only for testing purposes and cannot be mixed with use of Docker secrets!
-# POSTGRES_PASSWORD=supersecret
-# JWT_SECRET_KEY=supersecret
-# COLLECTOR_PRESENTER_PUBLISHER_API_KEY=supersecret
-
 # Paths
 CVE_UPDATE_FILE=/data/cve_dictionary.xml
 CPE_UPDATE_FILE=/data/cpe_dictionary.xml

docker/README.md

@@ -56,7 +56,7 @@ git clone https://github.com/SK-CERT/Taranis-NG.git
 cd Taranis-NG
 ```
 
-_Then_, remove `.example` extension from file `docker/.env.example` and files in `docker/secrets`. Use your favorite text editor and change default passwords. Taranis NG uses [Docker secrets](https://docs.docker.com/compose/use-secrets/) to store sensitive data. (Saving passwords in variables defined in `docker/.env` is not advised and you will need to modify Docker compose YAML files to make it work correctly. Also, make sure you do not have both POSTGRES_PASSWORD and POSTGRES_PASSWORD_FILE set - they are mutually exclusive)
+_Then_, remove `.example` extension from file `docker/.env.example` and files in `docker/secrets`. Use your favorite text editor and change default passwords. Taranis NG uses [Docker secrets](https://docs.docker.com/compose/use-secrets/) to store sensitive data. (Saving passwords in variables defined in `docker/.env` is not advised and you will need to modify Docker compose YAML files to make it work correctly.
 
 ```bash
 vim docker/.env
@@ -147,25 +147,33 @@ Any configuration options are available at [https://hub.docker.com/_/postgres](h
 | `DB_URL`                    | PostgreSQL database URL. | `127.0.0.1` |
 | `DB_DATABASE`               | PostgreSQL database name. | `taranis-ng` |
 | `DB_USER`                   | PostgreSQL database user. | `taranis-ng` |
-| `DB_PASSWORD`               | PostgreSQL database password. | `supersecret` |
 | `DB_POOL_SIZE`              | SQLAlchemy QueuePool number of active connections to the database. | `100` |
 | `DB_POOL_RECYCLE`           | SQLAlchemy QueuePool maximum connection age. | `300` |
 | `DB_POOL_TIMEOUT`           | SQLAlchemy QueuePool connection timeout. | `5` |
-| `JWT_SECRET_KEY`            | JWT token secret key. | `supersecret` |
 | `OPENID_LOGOUT_URL`         | Keycloak logout URL. | `https://example.com/auth/realms/master/protocol/openid-connect/logout` |
 | `WORKERS_PER_CORE`          | Number of gunicorn worker threads to spawn per CPU core. | `4` |
 | `SKIP_DEFAULT_COLLECTOR`    | Set to `true` to prevent initialization of a default docker collector at first run | `` |
 
+| Secrets file                | Description | Example |
+|-----------------------------|-------------|----------|
+| `postgres_password`         | PostgreSQL database password. | `supersecret` |
+| `jwt_secret_key`            | JWT token secret key. | `supersecret` |
+
+
 Taranis NG can use [connection pooling](https://docs.sqlalchemy.org/en/14/core/pooling.html) to maintain multiple active connections to the database server. Connection pooling is required when your deployment serves hundreds of customers from one instance. To enable connection pooling, set the `DB_POOL_SIZE`, `DB_POOL_RECYCLE`, and `DB_POOL_TIMEOUT` environment variables.
 
 ### `bots`, `collectors`, `presenters`, `publishers`
 
 | Environment variable        | Description | Example |
 |-----------------------------|-------------|----------|
 | `TARANIS_NG_CORE_URL`       | URL of the Taranis NG core API. | `http://127.0.0.1:8080/api/v1` |
-| `API_KEY`                   | Shared API key. | `cuBG/4H9lGTeo47F9X6DUg` |
 | `WORKERS_PER_CORE`          | Number of gunicorn worker threads to spawn per CPU core. | `4` |
 
+| Secrets file                | Description | Example |
+|-----------------------------|-------------|----------|
+| `api_key`            | Shared API key. | `cuBG/4H9lGTeo47F9X6DUg` |
+
+
 ### `gui`
 
 | Environment variable          | Description | Example |

docker/docker-compose.yml

@@ -1,5 +1,3 @@
-version: "3.9"
-
 services:
   redis:
     image: "redis:${REDIS_TAG}"
@@ -20,8 +18,6 @@ services:
     environment:
       POSTGRES_DB: "taranis-ng"
       POSTGRES_USER: "taranis-ng"
-      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
-      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
       TZ: "${TZ}"
       PGTZ: "${TZ}"
     command: ["postgres", "-c", "shared_buffers=${DB_SHARED_BUFFERS}", "-c", "max_connections=${DB_MAX_CONNECTIONS}"]
@@ -54,8 +50,6 @@ services:
       DB_URL: "database"
       DB_DATABASE: "taranis-ng"
       DB_USER: "taranis-ng"
-      DB_PASSWORD: "${POSTGRES_PASSWORD}"
-      DB_PASSWORD_FILE: /run/secrets/postgres_password
       DB_POOL_SIZE: 100
       DB_POOL_RECYCLE: 300
       DB_POOL_TIMEOUT: 30
@@ -64,8 +58,6 @@ services:
       LDAP_BASE_DN: "${LDAP_BASE_DN}"
       LDAP_CA_CERT_PATH:
 
-      JWT_SECRET_KEY: "${JWT_SECRET_KEY}"
-      JWT_SECRET_KEY_FILE: /run/secrets/jwt_secret_key
       OPENID_LOGOUT_URL: ""
       WORKERS_PER_CORE: "1"
 
@@ -76,9 +68,6 @@ services:
       TZ: "${TZ}"
       DEBUG: "true"
       DEBUG_SQL: "false"
-      # to allow automatic initialisation of collectors/presenters/publishers
-      API_KEY: "${COLLECTOR_PRESENTER_PUBLISHER_API_KEY}"
-      API_KEY_FILE: "/run/secrets/api_key"
     labels:
       traefik.enable: "true"
       traefik.http.services.taranis-api.loadbalancer.server.port: "80"
@@ -121,8 +110,6 @@ services:
         http_proxy: "${HTTP_PROXY}"
         https_proxy: "${HTTPS_PROXY}"
     environment:
-      API_KEY: "${COLLECTOR_PRESENTER_PUBLISHER_API_KEY}"
-      API_KEY_FILE: "/run/secrets/api_key"
       TARANIS_NG_CORE_URL: "http://core"
       TARANIS_NG_CORE_SSE: "http://core/sse"
       WORKERS_PER_CORE: "1"
@@ -151,8 +138,6 @@ services:
         https_proxy: "${HTTPS_PROXY}"
     environment:
       TARANIS_NG_CORE_URL: "http://core"
-      API_KEY: "${COLLECTOR_PRESENTER_PUBLISHER_API_KEY}"
-      API_KEY_FILE: "/run/secrets/api_key"
       WORKERS_PER_CORE: "1"
       DEBUG: "true"
       TZ: "${TZ}"
@@ -182,8 +167,6 @@ services:
         https_proxy: "${HTTPS_PROXY}"
     environment:
       TARANIS_NG_CORE_URL: "http://core"
-      API_KEY: "${COLLECTOR_PRESENTER_PUBLISHER_API_KEY}"
-      API_KEY_FILE: "/run/secrets/api_key"
       WORKERS_PER_CORE: "1"
       TZ: "${TZ}"
     ports:
@@ -214,8 +197,6 @@ services:
         https_proxy: "${HTTPS_PROXY}"
     environment:
       TARANIS_NG_CORE_URL: "http://core"
-      API_KEY: "${COLLECTOR_PRESENTER_PUBLISHER_API_KEY}"
-      API_KEY_FILE: "/run/secrets/api_key"
       WORKERS_PER_CORE: "1"
       TZ: "${TZ}"
     logging:

docker/prestart_core.sh

@@ -10,13 +10,8 @@ python /app/db_migration.py db upgrade head
 
 if [ "$(python ./manage.py collector --list | wc -l)" == 0 ] && [ x"$SKIP_DEFAULT_COLLECTOR" != "xtrue" ]; then
     (
-    echo "Adding default collector"
-    if [ -z "$API_KEY_FILE" ]; then
-        echo "API_KEY_FILE variable is not set, will use API_KEY..."
-    else
-        echo "Reading API key from file..."
-        API_KEY=$(cat "$API_KEY_FILE")
-    fi
+    echo "Reading API key from file..."
+    API_KEY=$(cat "/run/secrets/api_key")
     python ./manage.py collector --create --name "Default Docker Collector" --description "A local collector node configured as a part of Taranis NG default installation." --api-url "http://collectors/" --api-key "$API_KEY"
     ) &
 fi

src/bots/config.py

@@ -1,2 +1,13 @@
 class Config(object):
+
+    def read_secret(secret_name):
+        file_path = f"/run/secrets/{secret_name}"
+        try:
+            with open(file_path, "r") as secret_file:
+                return secret_file.read().strip()
+        except FileNotFoundError:
+            raise RuntimeError(f"Secret '{secret_name}' not found.")
+
+
     DEBUG = True
+    API_KEY = read_secret("api_key")

src/bots/managers/auth_manager.py

@@ -4,17 +4,11 @@
     wrapper: Wrapper function for the API endpoints.
 """
 
-from functools import wraps
-from flask import request
 import os
 import ssl
-
-try:
-    with open(os.getenv("API_KEY_FILE"), "r") as file:
-        api_key = file.read()
-except FileNotFoundError:
-    print("API_KEY_FILE not found. Please set the API_KEY_FILE environment variable to the path of the file containing the API key.")
-    api_key = os.getenv("API_KEY")
+from config import Config
+from flask import request
+from functools import wraps
 
 if os.getenv("SSL_VERIFICATION") == "False":
     try:
@@ -30,13 +24,14 @@ def api_key_required(fn):
 
     Arguments:
         fn -- The function to be decorated.
+
     Returns:
         wrapper: Wrapper function for the API endpoints.
     """
 
     @wraps(fn)
     def wrapper(*args, **kwargs):
-        if "Authorization" not in request.headers or request.headers["Authorization"] != ("Bearer " + api_key):
+        if "Authorization" not in request.headers or request.headers["Authorization"] != (f"Bearer {Config.API_KEY}"):
             return {"error": "not authorized"}, 401
         else:
             return fn(*args, **kwargs)

src/bots/managers/sse_manager.py

@@ -4,26 +4,17 @@
 import requests
 import sseclient
 import threading
-
+from config import Config
 from managers import bots_manager
 
-
-try:
-    with open(os.getenv("API_KEY_FILE"), "r") as file:
-        api_key = file.read()
-except FileNotFoundError:
-    print("API_KEY_FILE not found. Please set the API_KEY_FILE environment variable to the path of the file containing the API key.")
-    api_key = os.getenv("API_KEY")
-
-
 def initialize():
     """Start the SSE thread to listen to the Core's events."""
 
     class SSEThread(threading.Thread):
         @classmethod
         def run(cls):
             try:
-                response = requests.get(f"{os.getenv('TARANIS_NG_CORE_SSE')}?api_key={api_key}", stream=True)
+                response = requests.get(f"{os.getenv('TARANIS_NG_CORE_SSE')}?api_key={Config.API_KEY}", stream=True)
                 client = sseclient.SSEClient(response)
                 for event in client.events():
                     bots_manager.process_event(event.event, event.data)

src/bots/remote/core_api.py

@@ -1,16 +1,16 @@
 """This module provides a class to interact with the Taranis-NG Core API."""
 
-import os
 import json
+import os
 import requests
+from config import Config
 
 
 class CoreApi:
     """A class that provides methods to interact with the Taranis-NG Core API.
 
     Attributes:
         api_url (str): The URL of the Taranis-NG Core API.
-        api_key (str): The API key used for authentication.
         headers (dict): The headers used for API requests.
 
     Methods:
@@ -28,13 +28,7 @@ class CoreApi:
     api_url = os.getenv("TARANIS_NG_CORE_URL")
     if api_url.endswith("/"):
         api_url = api_url[:-1]
-    try:
-        with open(os.getenv("API_KEY_FILE"), "r") as file:
-            api_key = file.read()
-    except FileNotFoundError:
-        print("API_KEY_FILE not found. Please set the API_KEY_FILE environment variable to the path of the file containing the API key.")
-        api_key = os.getenv("API_KEY")
-    headers = {"Authorization": "Bearer " + api_key}
+    headers = {"Authorization": f"Bearer {Config.API_KEY}"}
 
     @classmethod
     def get_bots_presets(cls, bot_type):
@@ -53,7 +47,7 @@ def get_bots_presets(cls, bot_type):
         """
         try:
             response = requests.post(
-                cls.api_url + "/api/v1/bots/bots-presets", json={"api_key": cls.api_key, "bot_type": bot_type}, headers=cls.headers
+                cls.api_url + "/api/v1/bots/bots-presets", json={"api_key": Config.API_KEY, "bot_type": bot_type}, headers=cls.headers
             )
             return response.json(), response.status_code
         except (requests.exceptions.ConnectionError, json.decoder.JSONDecodeError):

src/collectors/config.py

@@ -1,2 +1,13 @@
 class Config(object):
+
+    def read_secret(secret_name):
+        file_path = f"/run/secrets/{secret_name}"
+        try:
+            with open(file_path, "r") as secret_file:
+                return secret_file.read().strip()
+        except FileNotFoundError:
+            raise RuntimeError(f"Secret '{secret_name}' not found.")
+
+
+    API_KEY = read_secret("api_key")
     DEBUG = True

src/collectors/managers/auth_manager.py

@@ -4,17 +4,12 @@
     wrapper: Wrapper function for the API endpoints.
 """
 
-from functools import wraps
-from flask import request
 import os
 import ssl
+from config import Config
+from flask import request
+from functools import wraps
 
-try:
-    with open(os.getenv("API_KEY_FILE"), "r") as file:
-        api_key = file.read()
-except FileNotFoundError:
-    print("API_KEY_FILE not found. Please set the API_KEY_FILE environment variable to the path of the file containing the API key.")
-    api_key = os.getenv("API_KEY")
 
 if os.getenv("SSL_VERIFICATION") == "False":
     try:
@@ -37,7 +32,7 @@ def api_key_required(fn):
 
     @wraps(fn)
     def wrapper(*args, **kwargs):
-        if "Authorization" not in request.headers or request.headers["Authorization"] != ("Bearer " + api_key):
+        if "Authorization" not in request.headers or request.headers["Authorization"] != (f"Bearer {Config.API_KEY}"):
             return {"error": "not authorized"}, 401
         else:
             return fn(*args, **kwargs)

src/collectors/remote/core_api.py

@@ -2,8 +2,9 @@
 
 import logging
 import os
-import urllib
 import requests
+import urllib
+from config import Config
 
 logger = logging.getLogger("gunicorn.error")
 logger.level = logging.INFO
@@ -19,7 +20,6 @@ class CoreApi:
 
     Attributes:
         api_url (str): The URL of the Taranis-NG API.
-        api_key (str): The API key used for authentication.
         headers (dict): The headers to be included in API requests.
 
     Methods:
@@ -31,13 +31,7 @@ class CoreApi:
     api_url = os.getenv("TARANIS_NG_CORE_URL")
     if api_url.endswith("/"):
         api_url = api_url[:-1]
-    try:
-        with open(os.getenv("API_KEY_FILE"), "r") as file:
-            api_key = file.read()
-    except FileNotFoundError:
-        print("API_KEY_FILE not found. Please set the API_KEY_FILE environment variable to the path of the file containing the API key.")
-        api_key = os.getenv("API_KEY")
-    headers = {"Authorization": "Bearer " + api_key}
+    headers = {"Authorization": f"Bearer {Config.API_KEY}"}
 
     @classmethod
     def get_osint_sources(cls, collector_type):
@@ -66,7 +60,7 @@ def get_osint_sources(cls, collector_type):
                 + "/api/v1/collectors/"
                 + urllib.parse.quote(id)
                 + "/osint-sources?api_key="
-                + urllib.parse.quote(cls.api_key)
+                + urllib.parse.quote(Config.API_KEY)
                 + "&collector_type="
                 + urllib.parse.quote(collector_type),
                 headers=cls.headers,

src/core/README.md

@@ -6,12 +6,14 @@
     `source venv/bin/activate`
     `pip3 install -r requirements.txt`
 4. Set environment variables for taranis-ng-core:
-    DB_URL=127.0.0.1:5432;DB_DATABASE=taranisdb;DB_USER=<YOUR-DB-USER>;DB_PASSWORD=<YOUR-DB-USER-PASSWORD>;JWT_SECRET_KEY=<YOUR-JWT-SECRET-KEY>
-5. Before first run uncomment line with `import test.py` in app.py to create set of test data. After first run comment this line again.
-6. Run taranis-ng-core: `python3 run.py`
-7. Set environment variables for taranis-ng-collectors:
+    DB_URL=127.0.0.1:5432;DB_DATABASE=taranisdb;DB_USER=<YOUR-DB-USER>
+5. Set secrets (passwords) for taranis-ng-core:
+    api_key.txt, jwt_secret_key.txt, postgres_password.txt
+6. Before first run uncomment line with `import test.py` in app.py to create set of test data. After first run comment this line again.
+7. Run taranis-ng-core: `python3 run.py`
+8. Set environment variables for taranis-ng-collectors:
     FLASK_RUN_PORT=5001;API_KEY=12345;TARANIS_NG_CORE_URL=http://127.0.0.1:5000;SSL_VERIFICATION=False
-8. Run taranis-ng-collectors: `python3 run.py` and it should collect first set of RSS news items from preconfigured test osint source
+9. Run taranis-ng-collectors: `python3 run.py` and it should collect first set of RSS news items from preconfigured test osint source
 
 # **TaranisNG frontend setup**
 1. Install Node.js from https://nodejs.org/en/