DFTT 2.0 release using updated schema, additional file format support…

…, and providing docker container options.

README.md

@@ -1,23 +1,27 @@
-# Detection Framework Testbed and Toolkit (DFTT), Version: 1.0
+# Detection Framework Testbed and Toolkit (DFTT), Version: 2.0
 
 The Detection Framework Testbed and Toolkit is a database and associated java programs intended to facilitate the development and testing of algorithms for operating suites of correlation and subspace detectors. This framework is a generalization of the system described in Harris and Dodge (2011). It allows retrospective processing of sequences of data using various system configurations. Results are saved in a database, so it is easy to compare the results obtained using different configurations of the system.
+
 ## Getting Started
 
 These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
 
 ### Prerequisites
 
-What things you need to install the software and how to install them
+The development system must have Maven 3.8 and git 2.* + installed as well as access to a git repo containing the necessary open source dependencies.
 
-```
-The development system must have Maven 3.1 and git 2.* + installed as well as access to a git repo containing the necessary open source dependencies.
 The git repo must also contain a suitable Oracle JDBC driver
+
 Java 8.* or greater is required to compile the source code.
-```
+
+The OregonDSP library is a required dependency and will need to be built as well. You can find the project at https://github.com/dbh441/OregonDSP
+
 
 ## Deployment
 
-DFFT requires an Oracle database hosting a schema in which configuration and results are stored. Current development has been on Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Instructions for creating the necessary schema and preparing it for use are in database/Instructions.txt
+DFFT requires an Oracle database hosting a schema in which configuration and results are stored. Current development has been on Oracle Enterprise Edition Release 21.3.0 - 64bit.
+
+Scripts creating the necessary schema and preparing it for use are in docker/database/ and a Dockerfile for an example database is provided for evaluation use.
 
 
 ## Built With
@@ -35,4 +39,4 @@ This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md
 
 LLNL-CODE-801881
 
-SPDX-License-Identifier: MIT
+SPDX-License-Identifier: MIT

docker/README.md

@@ -0,0 +1,52 @@
+# Manually running the project
+
+Install Docker or compatible runtime:
+
+### Windows
+https://docs.microsoft.com/en-us/windows/wsl/install
+
+https://docs.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers
+
+### Linux/Mac/Etc
+
+https://docs.docker.com/engine/
+
+
+Pull the official Oracle container repository Dockerfile and build the EE image at oracle/docker-images/tree/main/OracleDatabase/SingleInstance/dockerfiles/, download the LINUX.X64_213000_db_home.zip installation file for Oracle EE 21.3, place it in the 21.3.0 folder, and run:
+
+./buildContainerImage.sh -v 21.3.0 -t oracle:21.3.0-0-ee -e
+
+Pull or build the dftt and dftt-db images.
+
+Create a host-only bridge network for DFTT, this will enable the service discovery mechanism and insure that DFTT doesn't conflict with any existing network resources:
+
+docker network create --driver bridge dftt-net
+
+Start the database and set the necessary password:
+
+mkdir -p \$(pwd)/database
+
+docker run -e ORACLE_PDB=ORCL -e ORACLE_PWD=changeit -v $(pwd)/database/:/opt/oracle/oradata/ --network dftt-net --name dftt-db -d -p 127.0.0.1:1521:1521 dftt-db
+
+Start the gui:
+
+mkdir -p \$(pwd)/config
+
+mkdir -p \$(pwd)/data
+
+docker run -p 127.0.0.1:8080:8080 -v $(pwd)/config/:/opt/apps/config/ -v \$(pwd)/data/:/opt/apps/data/ -e DB_URL=dftt-db --name dftt-gui --network dftt-net -d dftt
+
+Log into the gui by opening a browser and visiting:
+http://127.0.0.1:8080/
+
+You can now connect to the database from the Builder to start working on DFTT projects.
+
+### FDSN example
+
+cd /opt/apps/config/ && ../ConfigCreator_2.0.sh dftt@dftt-db -c TJN -m 2022040 -M 2022046 -s TJN -I IRISDMC -type FDSN
+
+### Continuous Waveform table example
+
+./ConfigCreator_2.0.sh dftt@dftt-db -c JOKA_Z -D 1 -d 1 -e 10 -f 2 -F 20 -i 5 -m 2018119 -M 2018124 -n 80 -s JOKA -S 2 -t 0.5
+
+Note that for the continuous waveform usage you will need to have waveforms available and mounted in volumes into the docker container that match the paths entered into the CONTINUOUS_WAVEFORM table.

docker/database/Dockerfile

@@ -0,0 +1,12 @@
+FROM oracle:21.3.0-0-ee
+
+ARG NAME="Detection Framework Standalone Database"
+ARG DESCRIPTION="Basic Oracle database set up to allow testing the DFTT project in a standalone configuration, not for production use!"
+ARG LICENSE="Unreleased"
+ARG VENDOR="LLNL"
+
+LABEL name=$NAME
+LABEL maintainer="Justin Barno ([email protected])"
+LABEL gov.llnl.tags="utilities"
+
+ADD oracle-setup/ /opt/oracle/scripts/setup/

docker/database/build.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+docker build . -t dftt-db:latest

docker/database/database.yml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: oracle
+  namespace: dftt
+  labels:
+    app: oracle
+    tier: databases
+    version: v1
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: oracle   
+  template:
+    metadata:
+      labels:
+        name: oracle
+    spec:
+      containers:
+      - name: oracle
+        image: llnl/oracle:latest
+        env:
+         - name: ORACLE_SQL_DIR
+           value: /opt/oracle/scripts/startup
+         - name: ORACLE_PDB
+           value: ORCL
+         - name: ORACLE_NODE_NAME
+           valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        imagePullPolicy: IfNotPresent
+        ports:
+        - containerPort: 1521
+          name: oracle
+        - containerPort: 8080
+          name: oracle-studio
+        volumeMounts:
+        - mountPath: /opt/oracle/oradata
+          name: oracle-data
+        - mountPath: /dev/shm
+          name: dshm
+      volumes:
+      - name: oracle-data
+        hostPath: 
+          path: /oradata
+      - name: dshm
+        emptyDir:
+          medium: Memory
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: dftt-db-service
+spec:
+  selector:
+    app: oracle
+  ports:
+  - port: 1521
+    targetPort: oracle

docker/database/oracle-setup/00_fullInstall.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+set -e
+
+SYS_SCRIPTS=/opt/oracle/scripts/setup/sys_scripts
+DFTT_SCRIPTS=/opt/oracle/scripts/setup/dftt_scripts
+DFTT_USER=dftt
+DFTT_PWD=changeit
+DFTT_USER_AUTH=${DFTT_USER}/${DFTT_PWD}
+
+for f in $SYS_SCRIPTS/*; do
+    case "$f" in
+        *.sh)     echo "$0: running $f"; . "$f" ;;
+        *.sql)    echo "$0: running $f"; echo "exit" | $ORACLE_HOME/bin/sqlplus -s "sys/$ORACLE_PWD@localhost:1521/ORCL as SYSDBA" @"$f"; echo ;;
+        *)        echo "$0: ignoring $f" ;;
+    esac
+    echo "";
+done
+
+for f in $DFTT_SCRIPTS/*; do
+    case "$f" in
+        *.sh)     echo "$0: running $f"; . "$f" ;;
+        *.sql)    echo "$0: running $f"; echo "exit" | $ORACLE_HOME/bin/sqlplus -s "${DFTT_USER_AUTH}@localhost:1521/ORCL" @"$f"; echo ;;
+        *)        echo "$0: ignoring $f" ;;
+    esac
+    echo "";
+done