Skip to content

Commit

Permalink
integrate UCL into quickstart setup (#582)
Browse files Browse the repository at this point in the history
  • Loading branch information
@dtrai2
dtrai2 authored May 24, 2024
1 parent 3c22b7c commit 8311e0f
Show file tree
Hide file tree
Showing 13 changed files with 3,029 additions and 8,713 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

### Features

* add UCL into the Quickstart Setup

### Improvements

* remove logger from Components and Factory signatures
Expand Down
117 changes: 71 additions & 46 deletions doc/source/getting_started.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Run without Logprep Container (default)
.. code-block:: bash
docker compose up -d
It starts and connects `Kafka`, `logprep`, `Opensearch` and `Opensearch Dashboards`.
2. Run Logprep against loaded environment from main `Logprep` directory:

Expand All @@ -103,8 +103,8 @@ Run without Logprep Container (default)
export PROMETHEUS_MULTIPROC_DIR="tmp/logprep"
logprep run quickstart/exampledata/config/pipeline.yml
Run with Logprep Container
--------------------------
Expand All @@ -114,7 +114,7 @@ Run with Logprep Container
.. code-block:: bash
docker compose --profile logprep up -d
Run with getting config from http server with basic authentication
------------------------------------------------------------------
Expand All @@ -124,14 +124,14 @@ Run with getting config from http server with basic authentication
.. code-block:: bash
docker compose --profile basic_auth up -d
* Run within the project root directory:

.. code-block:: bash
export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
logprep run http://localhost:8081/config/pipeline.yml
Run with getting config from http server with mTLS authentication
-----------------------------------------------------------------
Expand All @@ -141,57 +141,43 @@ Run with getting config from http server with mTLS authentication
.. code-block:: bash
docker compose --profile mtls up -d
* Run within the project root directory:

.. code-block:: bash
export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
logprep run https://localhost:8082/config/pipeline.yml
Run with getting config from FDA with oauth2 authentication
-----------------------------------------------------------

Start logprep by using the oauth2 profile with docker compose:
* Run within the project root directory:

.. code-block:: bash
export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
docker compose --profile oauth2 up -d
Once they are set logprep can be started from the project root directory with:

.. code-block:: bash
logprep run "http://localhost:8002/api/v1/pipelines?stage=prod&logclass=ExampleClass"
logprep run https://localhost:8082/config/pipeline.yml
Interacting with the Quickstart Environment
-------------------------------------------

The start up takes a few seconds to complete, but once everything is up
and running it is possible to write JSON events into Kafka and read the processed events in
Opensearch Dashboards. Following services are available after start up:

====================== ================= ======== ========
Service Location User Password
====================== ================= ======== ========
Kafka: `localhost:9092` / /
Kafka Exporter: `localhost:9308` / /
Logprep metrics: `localhost:8001` / /
Opensearch: `localhost:9200` / /
Opensearch Dashboards: `localhost:5601` / /
Grafana Dashboards: `localhost:3000` admin admin
Prometheus: `localhost:9090` / /
Nginx Basic Auth: `localhost:8081` user password
Opensearch Dashboards.
Considering, you have started logprep.
Following services are available after start up:

====================== ================= ======================== =======================
Service Location User Password
====================== ================= ======================== =======================
Kafka: `localhost:9092` / /
Kafka Exporter: `localhost:9308` / /
Logprep metrics: `localhost:8001` / /
Opensearch: `localhost:9200` / /
Opensearch Dashboards: `localhost:5601` / /
Grafana Dashboards: `localhost:3000` admin admin
Prometheus: `localhost:9090` / /
Nginx Basic Auth: `localhost:8081` user password
Nginx mTLS: `localhost:8082`
Keycloak: `localhost:8080` admin admin
Keycloak Postgres: `localhost:5432` keycloak bitnami
FDA: `localhost:8002` logprep logprep
FDA Postgres: `localhost:25432` fda fda
====================== ================= ======== ========
Keycloak: `localhost:8080` admin admin
Keycloak Postgres: `localhost:5432` keycloak bitnami
FDA: `localhost:3002` (configure via keycloak) (configure via keycloak)
FDA Postgres: `localhost:5432` fda fda
UCL: `localhost:3001` (configure via keycloak) (configure via keycloak)
UCL Postgres: `localhost:5432` ucl ucl
====================== ================= ======================== =======================

The example rules that are used in the docker instance of Logprep can be found
in `quickstart/exampledata/rules`.
Expand All @@ -207,4 +193,43 @@ These events can be added to Kafka with the following command:
Once the events have been processed for the first time, the new indices *processed*, *sre*
and *pseudonyms* should be available in Opensearch Dashboards.

The environment can be stopped via :code:`docker compose down`.
The environment can be stopped via :code:`docker compose down`.


Utilizing FDA and UCL
---------------------

If you want to try out the FDA and UCL you first have to do some preparations.


0. Run the quickstart setup with the :code:`oauth2` profile:
:code:`docker compose --profile oauth2 up -d`.
1. Sign into the keycloak admin panel and create a logprep user in the :code:`logprep` realm.
Make sure that the user is part of the :code:`logprep-admin` group and has a password. If you
choose a password other than :code:`logprep` you have to update the credentials file
:code:`quickstart/exampledata/config/credentials.yml`, such that the password of
:code:`http://localhost:3001` and :code:`http://localhost:3002` reflects your choice.
2. You have to login to the FDA with the previously created user and create a release, as well
as your first logclass. It is also necessary to add an example event to this logclass in order
to initialize the first mapping flow. The logclass and its mapping flow has to be available in
order for logprep to load it's configuration.
3. If desired you can also create Use-Cases in the UCL. Similar to step two you have to sign in with
your created logprep user and then configure required Use-Cases.
At the current moment these configuration are not yet processed by logprep though, as the ucl
only provides a mock endpoint which doesn't contain your Use-Case configurations.
4. Set the env :code:`LOGPREP_CREDENTIALS_FILE` to :code:`quickstart/exampledata/config/credentials.yml`

Once you have set everything up you can run logprep with the following command.
Just consider that the first :code:`pipeline.yml` argument is used to define a proper :code:`input`
and :code:`output` as those are not part of the FDA/UCL output. Also, in the second argument
you should ensure that the :code:`stage` and :code:`loglcass` are set properly.

.. code-block:: bash
logprep run quickstart/exampledata/config/pipeline.yml "http://localhost:3002/api/v1/pipelines?stage=prod&logclass=ExampleClass" "http://localhost:3001/api/v1/general-predetection"
.. note::

If you did use the quickstart setup before and run into problems it is advised to first pull
all images again to update them to the latest version:
:code:`docker compose -f ./quickstart/docker-compose.yml pull`.
Loading

0 comments on commit 8311e0f

Please sign in to comment.