Skip to content

Latest commit

 

History

History
143 lines (94 loc) · 6.45 KB

README.md

File metadata and controls

143 lines (94 loc) · 6.45 KB

CircleCI

AI_Cybersecurity_IDS_PoC

and

Davids Udacity CloudDevOps Nanodegree Capstone Project

  • Winning Solution of BWI Data Analytics Hackathon 2020
  • CloudDevOps Pipeline with Green-Blue-Deployment for Davids Udacity CloudDevOps Nanodegree Capstone Project

bwi_hackathon_badge

App Screenshots

  • (as App is running on privately-owned real Internet-connected Infrastructure IPs are blurred)
Monitoring Dashboard Model Performance Anomaly Training Application of Models

Concept

  • unfortunately only in german :/

Features

  • Live-updating Webapp with DataPipeline from live running Zeek-Logs
    • extensive and easily extentable Monitoring Dashboard
  • Application of Neural Net and Random Forest models trained on pretrained labelled data against live Zeek logs
  • Training of Anomaly Detection using IsolationForest can be triggered during Runtime

Content

  • analysis contains all stuff Michael did for

    • exploring the used labelled data from UNSW-NB15 Datasets
    • checking out the performance of different models (mainly Random Forest and Neural Nets)
    • train and optimize the best model approaches using keras-tuner

  • app contains all stuff David did for

    • creating the live-updating Datapipeline using zeek logs
      • parsing them with an tinkered version of ParseZeekLogs for enabling continuously feeding the logs into the pipeline
      • and pygtail for also continuously feeding the logs into the pipeline
    • creating Webapp using plotly and Dash
    • Implementing live trained Anomaly Detection using Isolation Forest from scikit-learn

Installation/Deployment (CloudDevOps Nanodegree Part)

CircleCI Branch CI/CD Pipeline CircleCI Main CI/CD Pipeline

Local Docker-Compose Deployment

  1. Clone the repository:

    git clone https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC.git

  2. Go into Deploy Folder and run_compose.sh to run file-based or kafka-based Stack:

    deploy/run_compose.sh kafka
# OR
deploy/run_compose.sh file
  • first run will take very long because Docker Containers will be build locally and the zeek compilation and Kafka Plugin Install will take a while
  1. Go to http://127.0.0.1:8050/

Local Kubernetes Deployment

  1. You need to build the previous Compose-based stack at least once and upload the resulting Docker Container using the upload-docker.sh script or you relying on my public-built Container:

  1. You have to prepare and start minikube and run run_kube_local.sh:

    cd deploy
./run_kube_local.sh file
# OR (you can run booth as well)
./run_kube_local.sh file

  2. Now add local Ingress Rule to reach the broai endpoint:

    kubectl apply -f broai_kubernetes/ingress-local-service.yaml
# Check now these ingress service with
kubectl get svc

  3. Now add green.broai and blue.broai with your minikube IP to your /etc/hosts and visit this domains.

AWS Kubernetes Deployment

  1. You need to build the previous Compose-based stack at least once and upload the resulting Docker Container using the upload-docker.sh script or you relying on my public-built Container:

  1. Install aws-cli and deploy the Network and Cluster Requirements with the provided AWS Cloudformation Scripts:

    cd .circleci

scripts/push_cloudformation_stack.sh broainetwork cloudformation/network.yaml <your individual id>
scripts/push_cloudformation_stack.sh broaicluster cloudformation/cluster.yaml <your individual id>

  2. Get Access Token to acess your AWS EKS Cluster with kubectl:

    cd deploy

mkdir .kube
aws eks --region us-west-2 update-kubeconfig --kubeconfig .kube/config-aws --name AWSK8SCluster

  3. Deploy Kubernetes Manifests:

    ./run_kube_aws.sh

  4. Go to http://127.0.0.1:8050/

  5. Wait for finishing and check with kubectl --kubeconfig .kube/config-aws get svc the resulting Loadbalancer Hostnames and access them. :)

TODO

  • replacing filebased Datapipeline by Apache Kafka feed (DONE in scope of Davids Udacity CloudDevOps Nanodegree Capstone Project)
    • faster feeding into webapp
    • more elegant data management
  • also enabling Random Forest and Neural Net training during runtime
  • feeding predicted live-data into analysis workflow for automatic re-evaluation and re-training