diff --git a/README.rst b/README.rst index d61ca3a..adaad7b 100644 --- a/README.rst +++ b/README.rst @@ -17,7 +17,11 @@ Overview ======== -Automated Deployment of Lab Environments System (ADLES) +.. image:: documentation/system-overview-diagram.png +:alt: Overview of the system + :align: center + + Automated Deployment of Lab Environments System (ADLES) | ADLES automates the deterministic creation of virtualized environments for use in Cybersecurity and Information Technology (IT) education. @@ -34,25 +38,36 @@ Getting started pip3 install adles adles -h + adles --print-spec exercise + adles --print-spec infra + adles --list-examples + adles --print-example competition -- Read some of the examples, and try running a fews -- Read the exercise specification at specifications/exercise-specification.yaml -- Try writing your own! You can check it's syntax using ``adles -c example.yaml`` +Usage +===== +.. image:: documentation/usage-flowchart.png +:alt: Usage flowchart + :align: center + + - Read the exercise and infrastructure specifications and examples of them + - Write an infrastructure specification for your platform (Currently, VMware vSphere is the only platform supported) + - Write an exercise specification with the environment you want created + - Check its syntax, run the mastering phase, make your changes, and then run the deployment phase. + +.. code:: bash -System requirements -=================== -Client software ---------------- + adles -c my-competition.yaml + adles -m -s my-competition.yaml + adles -d -s my-competition.yaml -**Python**: 3.4+ (Recommended), 2.7.6+ +System requirements +=================== -Note on 2.7: While efforts have been made to maintain backward compatibility, Python 2.7 will be deprecated in -the future when full multi-lingual support and asyncronous operations are implemented. Therefore, -it is reccomended that Python 3 is used unless absolutely neccessary. (e.g CentOS 6) +**Python**: 3.4+ (Recommended), 2.7.6+ (Will be deprecated in the future) Python packages @@ -67,8 +82,8 @@ See ``requirements.txt`` for specific versions - colorlog - setuptools (If you are installing manually or developing) -Virtualization platforms ------------------------- +Platforms +--------- VMware vSphere ~~~~~~~~~~~~~~ @@ -85,141 +100,33 @@ Docker is currently in a pre-alpha development state. Eventually, the DockerInte support Docker Machine, Docker Compose, and potentially Docker Swarm. -Current goals -============= -The short-term goal is to create a system that allows instructors and students -teaching using virtual environments to automate their workloads. - -Currently, I am focusing on implementing the following components: - -- Overall system - - - Interface module system - - Specification injestion, parsing, and semantic checking - - Master-creation phase - - Deployment phase - - Post-phase cleanups - - User interface, logging, and basic result collection - -- Specifications - - - Exercise specification (The core spec that defines the exercise environment) - - Package specification - - Infrastructure specification +Contributing +============ -- VMware vSphere Interface module -- Documentation of the system as a whole and its component APIs -- Providing a number of examples of how to use the specifications in practice -- Basic Unit and Functional tests -- Packaging for PyPI +Contributions are most definitely welcome! See ``TODO.md`` for a list of what needs to be done. +Before submitting a pull request, do ensure you follow the general style and conventions used. +Just read the code for a bit to get a feel for how things are done, and stay consistent with that. -Future goals -============ +Goals and TODO +============== +The overall goal of ADLES is to create a easy to use and rock-solid system that allows instructors +and students teaching using virtual environments to automate their workloads. Long-term, I’d like to see the creation of a open-source repository, similiar to Hashicorp’s Atlas and Docker’s Hub, where educators can share packages and contribute to improving cyber education globally. -Support for additional platforms --------------------------------- - -- Docker: good for simulating large environments, with low resource overhead and quick load times - - - Docker Machine - - Docker Compose - - Docker Swarm - -- Hyper-V server: free academic license, good for schools invested in the Microsoft ecosystem -- Vagrant: brings the platform to workstations and personal machines. Enables interaction with - VirtualBox, desktop Hyper-V, and VMware Workstation -- Xen: free and open-source, scalable, robust. Rich introspection possibilities for monitoring - extensions using the Xen API -- KVM: free and open-source, good for schools with a strong Linux background. - LibVMI provides rich introspection possibilities here as well. -- Various cloud platforms, such as Microsoft Azure, Amazon AWS, Google Cloud Platform, - or DigitalOcean. Clouds are dynamic, scalable, and cost only for the time utilized, - making them perfect for short-lived tutorials or competitions. - - -Specification extensions ------------------------- - -- Monitoring extensions. This extension would add data collection configurations to relevant - areas of the specifications, enabling the implementation of high-fidelity data collection. This - would greatly enhance the system's research applicability, and enable other extensions such as - fully automated grading of results, visualizations, and data analytics. Some examples of these - configurations are: - - - Secondary interfaces on services for aggregating their log data, such as Windows Event Logs, - Unix Syslog, application logs, etc. - - Network packet captures. These could be obtained by enabling promiscuous mode on a vSwitch, or - enabling a SPAN monitoring port to aggregate the network traffic. - - Configuration of a centralized logging server to collect data, such as Splunk or ELK, including - specifying how the data aggregated should be "frozen" for inclusion with a package. - - Configuration of Virtual Machine Introspection (VMI) on supported platforms for a high-fidelity - view of exercises during execution. - - Instrumentation of the platforms and aggregation of the resulting log data, including the logs - created by ADLES itself. - -- Further Resource extensions for cyber-physical testbeds, and integration of Resources into more - aspects of the exercise and package specifications. Examples of resources include testbeds for: - ICS/SCADA, Wireless, USB devices, and car computers. -- Addition of ability to federate connections between separate lab environments, enabling the - sharing of testbed resources, virtualization infrastructure, and collaboration between - educational institutions. This could be implemented by extending the current Resources section - or the addition of a new section. -- Visualization of an exercise in progress, notably for competitive environments. -- Extension of the Groups section in the exercise specification with explicit specification - of user roles and permissions. -- Fully integrate and flesh out the role of exercise materials and other aspects of the Package - specification. -- Specification of system resources required for a service, e.g CPU, RAM, storage space. -- Collaboration and communications for an exercise, e.g video conferencing, TeamSpeak, - IRC channel, Discord server. -- Flag for selective toggling of parts of specifications where it would be useful to do - so without having to remove or comment out the content, e.g a folder. - - -System improvements -------------------- - -- Improved documentation on how to make a package, how to setup a platform for system, etc. -- Finish vSphere implementation (Users and Permissions) -- Redo the syntax verification component. Currently, any changes in spec involve a non-trivial - number of changes to the source code of the component. This is brittle and makes verifying new - extensions difficult, as most implementers will not bother in updating the component with the - new syntax of their extensions. - - - Swagger is a possible solution, as it enables automatic generation of APIs from a spec - - Implement verification of syntax for: - - - Non-vSphere platforms - - Package specification - - JSON files containing user information and platform logins - - Scoring criteria and other related files - -- Two-way transformation. Scan an existing environment and generate the specification for it. -- Graphical user interface. Ideally, this would be web-based for portability. In the short term, - this could be accomplished with a minimal amount of work using the EasyGUI Python module. -- Visualization of what the network and service structure of a given exercise or package - specification will look like without actually building the environment, including any - cyber-physical testbeds, connected labs, and monitoring components if their corresponding - extensions are implemented. -- Ability to pause/freeze an in-progress exercise, ideally as a simple commandline argument. -- Public repository of packages. -- More examples: - - - Examples of other types of competitions, notably CTFs - - Experiment examples - - Greater variety of tutorials - -- Simplify system setup for educators beyond what a Python package provides - - - Vagrantfile that builds a lightweight VM running the system - - Dockerfile that builds a lightweight Docker image running the system +Main things on the radar (see ``TODO.md`` for full list): +- User and group implementation for Vsphere +- Post-phase cleanups +- Result collection +- Provisioners +- Automated testing for utils and ideally Vsphere +- Working Docker platform implementation +- Implement a cloud platform interface, with Amazon AWS or Microsoft Azure being the easiest picks License diff --git a/TODO.md b/TODO.md index 5837c62..6785fba 100644 --- a/TODO.md +++ b/TODO.md @@ -109,8 +109,92 @@ This will help balance the load between the network connection to vCenter, vCent * Add a "edit-vms" script to do modifications en-masse? (e.g attach ISOs, edit hardware resources, etc.) -# Additions -* More interfaces +# Goals and Additions +Long-term, I’d like to see the creation of a open-source repository, similiar to +Hashicorp’s Atlas and Docker’s Hub, where educators can share packages +and contribute to improving cyber education globally. + + +## Support for additional platforms +* Docker: good for simulating large environments, with low resource overhead and quick load times + * Docker Machine + * Docker Compose + * Docker Swarm +* Hyper-V server: free academic license, good for schools invested in the Microsoft ecosystem +* Vagrant: brings the platform to workstations and personal machines. Enables interaction with + VirtualBox, desktop Hyper-V, and VMware Workstation +* Xen: free and open-source, scalable, robust. Rich introspection possibilities for monitoring + extensions using the Xen API +* KVM: free and open-source, good for schools with a strong Linux background. + LibVMI provides rich introspection possibilities here as well. +* Various cloud platforms, such as Microsoft Azure, Amazon AWS, Google Cloud Platform, + or DigitalOcean. Clouds are dynamic, scalable, and cost only for the time utilized, + making them perfect for short-lived tutorials or competitions. + +## Specification extensions +* Monitoring extensions. This extension would add data collection configurations to relevant + areas of the specifications, enabling the implementation of high-fidelity data collection. This + would greatly enhance the system's research applicability, and enable other extensions such as + fully automated grading of results, visualizations, and data analytics. Some examples of these + configurations are: + * Secondary interfaces on services for aggregating their log data, such as Windows Event Logs, + Unix Syslog, application logs, etc. + * Network packet captures. These could be obtained by enabling promiscuous mode on a vSwitch, or + enabling a SPAN monitoring port to aggregate the network traffic. + * Configuration of a centralized logging server to collect data, such as Splunk or ELK, including + specifying how the data aggregated should be "frozen" for inclusion with a package. + * Configuration of Virtual Machine Introspection (VMI) on supported platforms for a high-fidelity + view of exercises during execution. + * Instrumentation of the platforms and aggregation of the resulting log data, including the logs + created by ADLES itself. +* Further Resource extensions for cyber-physical testbeds, and integration of Resources into more + aspects of the exercise and package specifications. Examples of resources include testbeds for: + ICS/SCADA, Wireless, USB devices, and car computers. +* Addition of ability to federate connections between separate lab environments, enabling the + sharing of testbed resources, virtualization infrastructure, and collaboration between + educational institutions. This could be implemented by extending the current Resources section + or the addition of a new section. +* Visualization of an exercise in progress, notably for competitive environments. +* Extension of the Groups section in the exercise specification with explicit specification + of user roles and permissions. +* Fully integrate and flesh out the role of exercise materials and other aspects of the Package + specification. +* Specification of system resources required for a service, e.g CPU, RAM, storage space. +* Collaboration and communications for an exercise, e.g video conferencing, TeamSpeak, + IRC channel, Discord server. +* Flag for selective toggling of parts of specifications where it would be useful to do + so without having to remove or comment out the content, e.g a folder. + +## System improvements +* Improved documentation on how to make a package, how to setup a platform for system, etc. +* Finish vSphere implementation (Users and Permissions) +* Redo the syntax verification component. Currently, any changes in spec involve a non-trivial + number of changes to the source code of the component. This is brittle and makes verifying new + extensions difficult, as most implementers will not bother in updating the component with the + new syntax of their extensions. + * Swagger is a possible solution, as it enables automatic generation of APIs from a spec + * Implement verification of syntax for: + * Non-vSphere platforms + * Package specification + * JSON files containing user information and platform logins + * Scoring criteria and other related files +* Two-way transformation. Scan an existing environment and generate the specification for it. +* Graphical user interface. Ideally, this would be web-based for portability. In the short term, + this could be accomplished with a minimal amount of work using the EasyGUI Python module. +* Visualization of what the network and service structure of a given exercise or package + specification will look like without actually building the environment, including any + cyber-physical testbeds, connected labs, and monitoring components if their corresponding + extensions are implemented. +* Ability to pause/freeze an in-progress exercise, ideally as a simple commandline argument. +* Public repository of packages. +* More examples: + * Examples of other types of competitions, notably CTFs + * Experiment examples + * Greater variety of tutorials +* Simplify system setup for educators beyond what a Python package provides + * Vagrantfile that builds a lightweight VM running the system + * Dockerfile that builds a lightweight Docker image running the system + # Stretch/wack diff --git a/adles/__init__.py b/adles/__init__.py index d1e205d..65637d6 100644 --- a/adles/__init__.py +++ b/adles/__init__.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- -__version__ = "1.1.5" +__version__ = "1.1.6" __author__ = "Christopher Goes" __email__ = "" __license__ = "Apache 2.0" diff --git a/adles/scripts/adles_main.py b/adles/scripts/adles_main.py index 3e6fc75..3e6c832 100644 --- a/adles/scripts/adles_main.py +++ b/adles/scripts/adles_main.py @@ -37,7 +37,7 @@ --cleanup-masters Cleanup masters created by a specification --cleanup-enviro Cleanup environment created by a specification --nets Cleanup networks created during either phase - --print-spec NAME Prints the named: exercise, package, infrastructure + --print-spec NAME Prints the named specification: exercise, package, infrastructure --list-examples Prints the list of examples available --print-example NAME Prints the named example -h, --help Shows this help @@ -162,6 +162,7 @@ def main(): specs = ["exercise", "package", "infrastructure"] if spec in specs: # Find spec in package installation directory and print it + # This extracts the specifications from their package installation location filename = resource_filename(Requirement.parse("ADLES"), join("specifications", spec + "-specification.yaml")) with open(filename) as f: diff --git a/vsphere_testing.py b/vsphere_testing.py index 0f59c13..799d86a 100644 --- a/vsphere_testing.py +++ b/vsphere_testing.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""vsphere_testing. Entrypoint used to test the vSphere class. Has examples of API usage. +"""vsphere_testing. Used to experiment and test the Vsphere class. Has examples of API usage. Usage: vsphere_testing.py [-v] [options]