From ed6188e1ec4593539ac67bc49fbd08bddf4e2729 Mon Sep 17 00:00:00 2001 From: Jared Ondricek Date: Fri, 30 Jun 2023 13:36:50 -0500 Subject: [PATCH 001/159] Provide placeholder pages --- docs/index.rst | 9 ++++++++- docs/stix_overview/getting-started.rst | 4 ++++ docs/stix_overview/stix-recipes.rst | 4 ++++ 3 files changed, 16 insertions(+), 1 deletion(-) create mode 100644 docs/stix_overview/getting-started.rst create mode 100644 docs/stix_overview/stix-recipes.rst diff --git a/docs/index.rst b/docs/index.rst index a624e96d..0220f7ca 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -24,6 +24,13 @@ other modules in this library under "Additional Modules". mitre_attack_data/examples mitre_attack_data/custom_objects +.. toctree:: + :maxdepth: 1 + :caption: STIX Overview + + stix_overview/getting-started + stix_overview/stix-recipes + .. toctree:: :maxdepth: 1 :caption: Additional Modules @@ -31,4 +38,4 @@ other modules in this library under "Additional Modules". additional_modules/navlayers additional_modules/attackToExcel additional_modules/collections - additional_modules/diffStix \ No newline at end of file + additional_modules/diffStix diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/getting-started.rst new file mode 100644 index 00000000..c30c7bcf --- /dev/null +++ b/docs/stix_overview/getting-started.rst @@ -0,0 +1,4 @@ +Getting Started +=============== + +This is a placeholder page to put content from https://github.com/mitre/cti/blob/master/USAGE.md#accessing-attck-data-in-python into. diff --git a/docs/stix_overview/stix-recipes.rst b/docs/stix_overview/stix-recipes.rst new file mode 100644 index 00000000..08b19182 --- /dev/null +++ b/docs/stix_overview/stix-recipes.rst @@ -0,0 +1,4 @@ +Getting Started +=============== + +This is a placeholder page to put content from https://github.com/mitre/cti/blob/master/USAGE.md#python-recipes into. From bed8f9b4ca951b13c1d026d36d53f2017040b4b8 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 30 Jun 2023 16:34:41 -0400 Subject: [PATCH 002/159] Update getting-started.rst --- docs/stix_overview/getting-started.rst | 185 ++++++++++++++++++++++++- 1 file changed, 184 insertions(+), 1 deletion(-) diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/getting-started.rst index c30c7bcf..d63870a8 100644 --- a/docs/stix_overview/getting-started.rst +++ b/docs/stix_overview/getting-started.rst @@ -1,4 +1,187 @@ Getting Started =============== -This is a placeholder page to put content from https://github.com/mitre/cti/blob/master/USAGE.md#accessing-attck-data-in-python into. +## Accessing ATT&CK data in python + +There are several ways to acquire the ATT&CK data in Python. All of them will provide an object +implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. + +This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. + +### Requirements and imports + +Before installing requirements, we recommend setting up a virtual environment: + +1. Create virtual environment: + - macOS and Linux: `python3 -m venv env` + - Windows: `py -m venv env` +2. Activate the virtual environment: + - macOS and Linux: `source env/bin/activate` + - Windows: `env/Scripts/activate.bat` + +#### stix2 + +[stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: + +```python +from stix2 import Filter +``` + +However, if you are aiming to extend the ATT&CK dataset with new objects or implement complex workflows, you may need to use the `v20` specifier for some imports. This ensures that the objects use the STIX 2.0 API instead of the STIX 2.1 API. For example: + +```python +from stix2.v20 import AttackPattern +``` + +You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). + +#### taxii2client + +[taxii2-client can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-taxii-client#installation). The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of `taxii2client` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. + +If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. + +```python +from taxii2client.v20 import Collection +``` + +### Access local content + +Many users may opt to access the ATT&CK content via a local copy of the STIX data on this repo. This can be advantageous for several reasons: + +- Doesn't require internet access after the initial download +- User can modify the ATT&CK content if desired +- Downloaded copy is static, so updates to the ATT&CK catalog won't cause bugs in automated workflows. User can still manually update by cloning a fresh version of the data + +#### Access via FileSystemSource + +Each domain in this repo is formatted according to the [STIX2 FileSystem spec](https://stix2.readthedocs.io/en/latest/guide/filesystem.html). +Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: + +```python +from stix2 import FileSystemSource + +src = FileSystemSource('./cti/enterprise-attack') +``` + +#### Access via bundle + +If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: + +```python +from stix2 import MemoryStore + +src = MemoryStore() +src.load_from_file("enterprise-attack.json") +``` + +### Access live content + +Some users may instead prefer to access "live" ATT&CK content over the internet. This is advantageous for several reasons: + +- Always stays up to date with the evolving ATT&CK catalog +- Doesn't require an initial download of the ATT&CK content, generally requires less setup + +#### Access from the ATT&CK TAXII server + +Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII, the ATT&CK domains are represented as collections with static IDs: + +| domain | collection ID | +|:-------|:--------------| +| `enterprise-attack` | `95ecc380-afe9-11e4-9b6c-751b66dd541e` | +| `mobile-attack` | `2f669986-b40b-4423-b720-4396ca6a462b` | +| `ics-attack` | `02c3ef24-9cd4-48f3-a99f-b74ce24f1d34` | + +You can also get a list of available collection from the server directly: + +```python +from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 + +server = Server("https://cti-taxii.mitre.org/taxii/") +api_root = server.api_roots[0] +# Print name and ID of all ATT&CK domains available as collections +for collection in api_root.collections: + print(collection.title.ljust(20) + collection.id) +``` + +The following recipe demonstrates how to access the enterprise-attack data from the TAXII server. + +```python +from stix2 import TAXIICollectionSource +from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 + +collections = { + "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e", + "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b", + "ics-attack": "02c3ef24-9cd4-48f3-a99f-b74ce24f1d34" +} + +collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/") +src = TAXIICollectionSource(collection) +``` + +For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). + +#### Access from Github via requests + +Users can alternatively access the data from MITRE/CTI using HTTP requests, and load the resulting content into a MemoryStore. +While typically the TAXII method is more desirable for "live" access, this method can be useful if you want to +access data on a branch of the MITRE/CTI repo (the TAXII server only holds the master branch) or in the case of a TAXII server outage. + +```python +import requests +from stix2 import MemoryStore + +def get_data_from_branch(domain, branch="master"): + """get the ATT&CK STIX data from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'. Branch should typically be master.""" + stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/{branch}/{domain}/{domain}.json").json() + return MemoryStore(stix_data=stix_json["objects"]) + +src = get_data_from_branch("enterprise-attack") +``` + +### Access a specific version of ATT&CK + +ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.com/mitre/cti/tags). Tags prefixed with `ATT&CK-v` correspond to ATT&CK versions and tags prefixed with `CAPEC-v` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. + +In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: + +```python +import requests +from stix2 import MemoryStore + +def get_data_from_version(domain, version): + """get the ATT&CK STIX data for the given version from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'.""" + stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v{version}/{domain}/{domain}.json").json() + return MemoryStore(stix_data=stix_json["objects"]) + +src = get_data_from_version("enterprise-attack", "5.2") +``` + +You can get a list of ATT&CK versions programmatically using the github API: + +```python +import requests +import re + +refToTag = re.compile(r"ATT&CK-v(.*)") +tags = requests.get("https://api.github.com/repos/mitre/cti/git/refs/tags").json() +versions = list(map(lambda tag: refToTag.search(tag["ref"]).groups()[0] , filter(lambda tag: "ATT&CK-v" in tag["ref"], tags))) +# versions = ["1.0", "2.0", ...] +``` + +### Access multiple domains simultaneously + +Because ATT&CK is stored in multiple domains (as of this writing, enterprise-attack, mobile-attack and ics-attack), the above methodologies will only allow you to work +with a single domain at a time. While oftentimes the hard separation of domains is advantageous, occasionally it is useful to combine +domains into a single DataStore. Use any of the methods above to acquire the individual datastores, and then use the following approach to combine them into +a single CompositeDataSource: + +```python +from stix2 import CompositeDataSource + +src = CompositeDataSource() +src.add_data_sources([enterprise_attack_src, mobile_attack_src, ics_attack_src]) +``` + +You can then use this CompositeDataSource just as you would the DataSource for an individual domain. From 743723d6263be7689a86fe2976b418a18f5e32d2 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 30 Jun 2023 16:35:08 -0400 Subject: [PATCH 003/159] Update stix-recipes.rst --- docs/stix_overview/stix-recipes.rst | 667 +++++++++++++++++++++++++++- 1 file changed, 666 insertions(+), 1 deletion(-) diff --git a/docs/stix_overview/stix-recipes.rst b/docs/stix_overview/stix-recipes.rst index 08b19182..23dff9b4 100644 --- a/docs/stix_overview/stix-recipes.rst +++ b/docs/stix_overview/stix-recipes.rst @@ -1,4 +1,669 @@ Getting Started =============== -This is a placeholder page to put content from https://github.com/mitre/cti/blob/master/USAGE.md#python-recipes into. +## Python recipes + +Below are example python recipes which can be used to work with ATT&CK data. They assume the existence of an object implementing the DataStore API. Any of the methods outlined in the [Accessing ATT&CK data in python](#accessing-ATTCK-Data-in-Python) section should provide an object implementing this API. + +This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. See also the section on [Requirements and imports](#requirements-and-imports). + +### Getting an object + +The recipes in this section address how to query the dataset for a single object. + +#### By STIX ID + +The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. + +```python +g0075 = src.get("intrusion-set--f40eb8ce-2a74-4e56-89a1-227021410142") +``` + +#### By ATT&CK ID + +The following recipe can be used to retrieve an object according to its ATT&CK ID: + +```python +from stix2 import Filter + +g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] +``` + +Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. + +```python +from stix2 import Filter + +t1134 = src.query([ + Filter("external_references.external_id", "=", "T1134"), + Filter("type", "=", "attack-pattern") +])[0] +``` + +The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). + +#### By name + +The following recipe retrieves an object according to its name: + +```python +from stix2 import Filter + +def get_technique_by_name(thesrc, name): + filt = [ + Filter('type', '=', 'attack-pattern'), + Filter('name', '=', name) + ] + return thesrc.query(filt) +# get the technique titled "System Information Discovery" +get_technique_by_name(src, 'System Information Discovery') +``` + +#### By alias + +The following methodology can be used to find the group corresponding to a given alias: + +```python +from stix2 import Filter + +def get_group_by_alias(thesrc, alias): + return thesrc.query([ + Filter('type', '=', 'intrusion-set'), + Filter('aliases', '=', alias) + ])[0] + +get_group_by_alias(src, 'Cozy Bear') +``` + +### Getting multiple objects + +The recipes in this section address how to query the dataset for multiple objects. + +⚠ When working with queries to return objects based on a set of characteristics, it is likely that you'll end up with a few objects which are no longer maintained by ATT&CK. These are objects marked as deprecated or revoked. We keep these outdated objects around so that workflows depending on them don't break, but we recommend you avoid using them when possible. Please see the section [Working with deprecated and revoked objects](#Working-with-deprecated-and-revoked-objects) for more information. + +#### Objects by type + +See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. + +```python +from stix2 import Filter + +# use the appropriate STIX type in the query according to the desired ATT&CK type +groups = src.query([ Filter("type", "=", "intrusion-set") ]) +``` + +##### Getting techniques or sub-techniques + +ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. + +```python +from stix2 import Filter + +def get_techniques_or_subtechniques(thesrc, include="both"): + """Filter Techniques or Sub-Techniques from ATT&CK Enterprise Domain. + include argument has three options: "techniques", "subtechniques", or "both" + depending on the intended behavior.""" + if include == "techniques": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_is_subtechnique', '=', False) + ]) + elif include == "subtechniques": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_is_subtechnique', '=', True) + ]) + elif include == "both": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern') + ]) + else: + raise RuntimeError("Unknown option %s!" % include) + + return query_results + + +subtechniques = get_techniques_or_subtechniques(src, "subtechniques") +subtechniques = remove_revoked_deprecated(subtechniques) # see https://github.com/mitre/cti/blob/master/USAGE.md#removing-revoked-and-deprecated-objects +``` + +##### Getting software + +Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. + +```python +from itertools import chain +from stix2 import Filter + +def get_software(thesrc): + return list(chain.from_iterable( + thesrc.query(f) for f in [ + Filter("type", "=", "tool"), + Filter("type", "=", "malware") + ] + )) + +get_software(src) +``` + +#### Objects by content + +Sometimes it may be useful to query objects by the content of their description: + +```python +from stix2 import Filter + +def get_techniques_by_content(thesrc, content): + techniques = src.query([ Filter('type', '=', 'attack-pattern') ]) + return list(filter(lambda t: content.lower() in t.description.lower(), techniques)) + +# Get all techniques where the string LSASS appears in the description +get_techniques_by_content(src, 'LSASS') +``` + +#### Techniques by platform + +Techniques are associated with one or more platforms. You can query the techniques +under a specific platform with the following code: + +```python +from stix2 import Filter + +def get_techniques_by_platform(thesrc, platform): + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_platforms', '=', platform) + ]) + +# get techniques in the windows platform +get_techniques_by_platform(src, 'Windows') +``` + +#### Techniques by tactic + +Techniques are related to tactics by their kill_chain_phases property. +The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. + +```python +from stix2 import Filter + +def get_tactic_techniques(thesrc, tactic): + # double checking the kill chain is MITRE ATT&CK + # note: kill_chain_name is different for other domains: + # - enterprise: "mitre-attack" + # - mobile: "mitre-mobile-attack" + # - ics: "mitre-ics-attack" + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('kill_chain_phases.phase_name', '=', tactic), + Filter('kill_chain_phases.kill_chain_name', '=', 'mitre-attack'), + ]) + + +# use the x_mitre_shortname as argument +get_tactic_techniques(src, 'defense-evasion') +``` + +#### Tactics by matrix + +The tactics are individual objects (`x-mitre-tactic`), and their order in a matrix (`x-mitre-matrix`) is +found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches +the ordering of the tactics in that matrix. The following recipe returns a structured list of tactics within each matrix of the input DataStore. + +```python +from stix2 import Filter + +def getTacticsByMatrix(thesrc): + tactics = {} + matrix = thesrc.query([ + Filter('type', '=', 'x-mitre-matrix'), + ]) + + for i in range(len(matrix)): + tactics[matrix[i]['name']] = [] + for tactic_id in matrix[i]['tactic_refs']: + tactics[matrix[i]['name']].append(thesrc.get(tactic_id)) + + return tactics + +# get tactic layout +getTacticsByMatrix(src) +``` + +#### Objects created or modified since a given date + +Sometimes you may want to get a list of objects which have been created or modified after a certain time. + +```python +from stix2 import Filter + +def get_created_after(thesrc, timestamp): + filt = [ + Filter('created', '>', timestamp) + ] + return thesrc.query(filt) + +get_created_after(src, "2018-10-01T00:14:20.652Z") + + +def get_modified_after(thesrc, timestamp): + filt = [ + Filter('modified', '>', timestamp) + ] + return thesrc.query(filt) + +get_modified_after(src, "2018-10-01T00:14:20.652Z") +``` + +We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to [check the list of released versions of ATT&CK](https://github.com/mitre/cti/blob/master/USAGE.md#access-a-specific-version-of-attck). + +### Getting related objects + +A large part of working with ATT&CK revolves around parsing relationships between objects. It is useful +to track not only the related object but the relationship itself because a description is often +present to contextualize the nature of the relationship. The following recipes demonstrate +some common uses of relationships. + +#### Relationships microlibrary + +NOTE: The following code is intended to be used with the ATT&CK v12 release which includes Campaign Objects. +The examples are backwards-compatible for previous versions af ATT&CK that omit those objects. + +This microlibrary can be used to build a lookup table of stixID to related objects and relationships. +The argument to each accessor function is a STIX2 MemoryStore to build the relationship mappings from. + +```python +from pprint import pprint +from stix2 import MemoryStore, Filter + +# See section below on "Removing revoked and deprecated objects" +def remove_revoked_deprecated(stix_objects): + """Remove any revoked or deprecated objects from queries made to the data source""" + # Note we use .get() because the property may not be present in the JSON data. The default is False + # if the property is not set. + return list( + filter( + lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, + stix_objects + ) + ) + +def get_related(thesrc, src_type, rel_type, target_type, reverse=False): + """build relationship mappings + params: + thesrc: MemoryStore to build relationship lookups for + src_type: source type for the relationships, e.g "attack-pattern" + rel_type: relationship type for the relationships, e.g "uses" + target_type: target type for the relationship, e.g "intrusion-set" + reverse: build reverse mapping of target to source + """ + + relationships = thesrc.query([ + Filter('type', '=', 'relationship'), + Filter('relationship_type', '=', rel_type), + Filter('revoked', '=', False), + ]) + + # See section below on "Removing revoked and deprecated objects" + relationships = remove_revoked_deprecated(relationships) + + # stix_id => [ { relationship, related_object_id } for each related object ] + id_to_related = {} + + # build the dict + for relationship in relationships: + if src_type in relationship.source_ref and target_type in relationship.target_ref: + if (relationship.source_ref in id_to_related and not reverse) or (relationship.target_ref in id_to_related and reverse): + # append to existing entry + if not reverse: + id_to_related[relationship.source_ref].append({ + "relationship": relationship, + "id": relationship.target_ref + }) + else: + id_to_related[relationship.target_ref].append({ + "relationship": relationship, + "id": relationship.source_ref + }) + else: + # create a new entry + if not reverse: + id_to_related[relationship.source_ref] = [{ + "relationship": relationship, + "id": relationship.target_ref + }] + else: + id_to_related[relationship.target_ref] = [{ + "relationship": relationship, + "id": relationship.source_ref + }] + # all objects of relevant type + if not reverse: + targets = thesrc.query([ + Filter('type', '=', target_type), + Filter('revoked', '=', False) + ]) + else: + targets = thesrc.query([ + Filter('type', '=', src_type), + Filter('revoked', '=', False) + ]) + + # build lookup of stixID to stix object + id_to_target = {} + for target in targets: + id_to_target[target.id] = target + + # build final output mappings + output = {} + for stix_id in id_to_related: + value = [] + for related in id_to_related[stix_id]: + if not related["id"] in id_to_target: + continue # targeting a revoked object + value.append({ + "object": id_to_target[related["id"]], + "relationship": related["relationship"] + }) + output[stix_id] = value + return output + +# software:group +def software_used_by_groups(thesrc): + """returns group_id => {software, relationship} for each software used by the group and each software used by campaigns attributed to the group.""" + # get all software used by groups + tools_used_by_group = get_related(thesrc, "intrusion-set", "uses", "tool") + malware_used_by_group = get_related(thesrc, "intrusion-set", "uses", "malware") + software_used_by_group = {**tools_used_by_group, **malware_used_by_group} # group_id -> [{software, relationship}] + + # get groups attributing to campaigns and all software used by campaigns + software_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") + malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") + for id in malware_used_by_campaign: + if id in software_used_by_campaign: + software_used_by_campaign[id].extend(malware_used_by_campaign[id]) + else: + software_used_by_campaign[id] = malware_used_by_campaign[id] + campaigns_attributed_to_group = { + "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} + "software": software_used_by_campaign # campaign_id => {software, relationship} + } + + for group_id in campaigns_attributed_to_group["campaigns"]: + software_used_by_campaigns = [] + # check if attributed campaign is using software + for campaign in campaigns_attributed_to_group["campaigns"][group_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in campaigns_attributed_to_group["software"]: + software_used_by_campaigns.extend(campaigns_attributed_to_group["software"][campaign_id]) + + # update software used by group to include software used by a groups attributed campaign + if group_id in software_used_by_group: + software_used_by_group[group_id].extend(software_used_by_campaigns) + else: + software_used_by_group[group_id] = software_used_by_campaigns + return software_used_by_group + +def groups_using_software(thesrc): + """returns software_id => {group, relationship} for each group using the software and each software used by attributed campaigns.""" + # get all groups using software + groups_using_tool = get_related(thesrc, "intrusion-set", "uses", "tool", reverse=True) + groups_using_malware = get_related(thesrc, "intrusion-set", "uses", "malware", reverse=True) + groups_using_software = {**groups_using_tool, **groups_using_malware} # software_id => {group, relationship} + + # get campaigns attributed to groups and all campaigns using software + campaigns_using_software = get_related(thesrc, "campaign", "uses", "tool", reverse=True) + campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) + for id in campaigns_using_malware: + if id in campaigns_using_software: + campaigns_using_software[id].extend(campaigns_using_malware[id]) + else: + campaigns_using_software[id] = campaigns_using_malware[id] + groups_attributing_to_campaigns = { + "campaigns": campaigns_using_software,# software_id => {campaign, relationship} + "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} + } + + for software_id in groups_attributing_to_campaigns["campaigns"]: + groups_attributed_to_campaigns = [] + # check if campaign is attributed to group + for campaign in groups_attributing_to_campaigns["campaigns"][software_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in groups_attributing_to_campaigns["groups"]: + groups_attributed_to_campaigns.extend(groups_attributing_to_campaigns["groups"][campaign_id]) + + # update groups using software to include software used by a groups attributed campaign + if software_id in groups_using_software: + groups_using_software[software_id].extend(groups_attributed_to_campaigns) + else: + groups_using_software[software_id] = groups_attributed_to_campaigns + return groups_using_software + +# software:campaign +def software_used_by_campaigns(thesrc): + """returns campaign_id => {software, relationship} for each software used by the campaign.""" + tools_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") + malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") + return {**tools_used_by_campaign, **malware_used_by_campaign} + +def campaigns_using_software(thesrc): + """returns software_id => {campaign, relationship} for each campaign using the software.""" + campaigns_using_tool = get_related(thesrc, "campaign", "uses", "tool", reverse=True) + campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) + return {**campaigns_using_tool, **campaigns_using_malware} + +# campaign:group +def groups_attributing_to_campaign(thesrc): + """returns campaign_id => {group, relationship} for each group attributing to the campaign.""" + return get_related(thesrc, "campaign", "attributed-to", "intrusion-set") + +def campaigns_attributed_to_group(thesrc): + """returns group_id => {campaign, relationship} for each campaign attributed to the group.""" + return get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True) + +# technique:group +def techniques_used_by_groups(thesrc): + """returns group_id => {technique, relationship} for each technique used by the group and each + technique used by campaigns attributed to the group.""" + # get all techniques used by groups + techniques_used_by_groups = get_related(thesrc, "intrusion-set", "uses", "attack-pattern") # group_id => {technique, relationship} + + # get groups attributing to campaigns and all techniques used by campaigns + campaigns_attributed_to_group = { + "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} + "techniques": get_related(thesrc, "campaign", "uses", "attack-pattern") # campaign_id => {technique, relationship} + } + + for group_id in campaigns_attributed_to_group["campaigns"]: + techniques_used_by_campaigns = [] + # check if attributed campaign is using technique + for campaign in campaigns_attributed_to_group["campaigns"][group_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in campaigns_attributed_to_group["techniques"]: + techniques_used_by_campaigns.extend(campaigns_attributed_to_group["techniques"][campaign_id]) + + # update techniques used by groups to include techniques used by a groups attributed campaign + if group_id in techniques_used_by_groups: + techniques_used_by_groups[group_id].extend(techniques_used_by_campaigns) + else: + techniques_used_by_groups[group_id] = techniques_used_by_campaigns + return techniques_used_by_groups + +def groups_using_technique(thesrc): + """returns technique_id => {group, relationship} for each group using the technique and each campaign attributed to groups using the technique.""" + # get all groups using techniques + groups_using_techniques = get_related(thesrc, "intrusion-set", "uses", "attack-pattern", reverse=True) # technique_id => {group, relationship} + + # get campaigns attributed to groups and all campaigns using techniques + groups_attributing_to_campaigns = { + "campaigns": get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True), # technique_id => {campaign, relationship} + "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} + } + + for technique_id in groups_attributing_to_campaigns["campaigns"]: + campaigns_attributed_to_group = [] + # check if campaign is attributed to group + for campaign in groups_attributing_to_campaigns["campaigns"][technique_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in groups_attributing_to_campaigns["groups"]: + campaigns_attributed_to_group.extend(groups_attributing_to_campaigns["groups"][campaign_id]) + + # update groups using techniques to include techniques used by a groups attributed campaign + if technique_id in groups_using_techniques: + groups_using_techniques[technique_id].extend(campaigns_attributed_to_group) + else: + groups_using_techniques[technique_id] = campaigns_attributed_to_group + return groups_using_techniques + +# technique:campaign +def techniques_used_by_campaigns(thesrc): + """returns campaign_id => {technique, relationship} for each technique used by the campaign.""" + return get_related(thesrc, "campaign", "uses", "attack-pattern") + +def campaigns_using_technique(thesrc): + """returns technique_id => {campaign, relationship} for each campaign using the technique.""" + return get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True) + +# technique:software +def techniques_used_by_software(thesrc): + """return software_id => {technique, relationship} for each technique used by the software.""" + techniques_by_tool = get_related(thesrc, "tool", "uses", "attack-pattern") + techniques_by_malware = get_related(thesrc, "malware", "uses", "attack-pattern") + return {**techniques_by_tool, **techniques_by_malware} + +def software_using_technique(thesrc): + """return technique_id => {software, relationship} for each software using the technique.""" + tools_by_technique_id = get_related(thesrc, "tool", "uses", "attack-pattern", reverse=True) + malware_by_technique_id = get_related(thesrc, "malware", "uses", "attack-pattern", reverse=True) + return {**tools_by_technique_id, **malware_by_technique_id} + +# technique:mitigation +def mitigation_mitigates_techniques(thesrc): + """return mitigation_id => {technique, relationship} for each technique mitigated by the mitigation.""" + return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=False) + +def technique_mitigated_by_mitigations(thesrc): + """return technique_id => {mitigation, relationship} for each mitigation of the technique.""" + return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=True) + +# technique:sub-technique +def subtechniques_of(thesrc): + """return technique_id => {subtechnique, relationship} for each subtechnique of the technique.""" + return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern", reverse=True) + +def parent_technique_of(thesrc): + """return subtechnique_id => {technique, relationship} describing the parent technique of the subtechnique""" + return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern")[0] + +# technique:data-component +def datacomponent_detects_techniques(thesrc): + """return datacomponent_id => {technique, relationship} describing the detections of each data component""" + return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern") + +def technique_detected_by_datacomponents(thesrc): + """return technique_id => {datacomponent, relationship} describing the data components that can detect the technique""" + return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern", reverse=True) + +# Example usage: +src = MemoryStore() +src.load_from_file("path/to/enterprise-attack.json") + +group_id_to_software = software_used_by_groups(src) +pprint(group_id_to_software["intrusion-set--2a158b0a-7ef8-43cb-9985-bf34d1e12050"]) # G0019 +# [ +# { +# "object": Malware, # S0061 +# "relationship": Relationship # relationship between G0019 and S0061 +# }, +# { +# ... +# } +# ] +``` + +#### Getting techniques used by a group's software + +Because a group uses software, and software uses techniques, groups can be considered indirect users of techniques used by their software. +These techniques are oftentimes distinct from the techniques used directly by a group, although there are occasionally intersections in these two sets of techniques. + +The following recipe can be used to retrieve the techniques used by a group's software: + +```python +from stix2.utils import get_type_from_id +from stix2 import Filter + +def get_techniques_by_group_software(thesrc, group_stix_id): + # get the malware, tools that the group uses + group_uses = [ + r for r in thesrc.relationships(group_stix_id, 'uses', source_only=True) + if get_type_from_id(r.target_ref) in ['malware', 'tool'] + ] + + # get the technique stix ids that the malware, tools use + software_uses = thesrc.query([ + Filter('type', '=', 'relationship'), + Filter('relationship_type', '=', 'uses'), + Filter('source_ref', 'in', [r.source_ref for r in group_uses]) + ]) + + #get the techniques themselves + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('id', 'in', [r.target_ref for r in software_uses]) + ]) + +get_techniques_by_group_software(src, "intrusion-set--f047ee18-7985-4946-8bfb-4ed754d3a0dd") +``` + +### Working with deprecated and revoked objects + +Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either `x_mitre_deprecated` or `revoked`) noting their status. In the case of revoked objects, a relationship of type `revoked-by` is also created targeting the replacing object. + +Unlike other objects in the dataset, relationships cannot be revoked or deprecated. Relationships are considered deprecated/revoked if one of the objects it is attached to is revoked or deprecated. + +#### Removing revoked and deprecated objects + +Revoked and deprecated objects are kept in the knowledge base so that workflows relying on those objects are not +broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no +longer maintained by ATT&CK. + +We recommend _not_ using built-in STIX filters for removing revoked objects (e.g `Filter('revoked', '=', False)`). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See [issue #127](https://github.com/mitre/cti/issues/127) for more details. + +```python +from stix2 import Filter + +def remove_revoked_deprecated(stix_objects): + """Remove any revoked or deprecated objects from queries made to the data source""" + # Note we use .get() because the property may not be present in the JSON data. The default is False + # if the property is not set. + return list( + filter( + lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, + stix_objects + ) + ) + +mitigations = src.query([ Filter("type", "=", "course-of-action") ]) +mitigations = remove_revoked_deprecated(mitigations) +``` + +#### Getting a revoking object + +When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: + +```python +from stix2 import Filter + +def getRevokedBy(stix_id, thesrc): + relations = thesrc.relationships(stix_id, 'revoked-by', source_only=True) + revoked_by = thesrc.query([ + Filter('id', 'in', [r.target_ref for r in relations]), + Filter('revoked', '=', False) + ]) + if revoked_by is not None: + revoked_by = revoked_by[0] + + return revoked_by + +getRevokedBy("attack-pattern--c16e5409-ee53-4d79-afdc-4099dc9292df", src) +``` From c1e01883a153af13520791d0792c3a0f2965aa99 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 30 Jun 2023 16:53:09 -0400 Subject: [PATCH 004/159] Update stix-recipes.rst --- docs/stix_overview/stix-recipes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_overview/stix-recipes.rst b/docs/stix_overview/stix-recipes.rst index 23dff9b4..92bf90a0 100644 --- a/docs/stix_overview/stix-recipes.rst +++ b/docs/stix_overview/stix-recipes.rst @@ -1,4 +1,4 @@ -Getting Started +Stix Recipes =============== ## Python recipes From 759ee16737019f281e162901f61674ce2454ffe5 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 30 Jun 2023 17:03:14 -0400 Subject: [PATCH 005/159] Update stix-recipes.rst --- docs/stix_overview/stix-recipes.rst | 656 ---------------------------- 1 file changed, 656 deletions(-) diff --git a/docs/stix_overview/stix-recipes.rst b/docs/stix_overview/stix-recipes.rst index 92bf90a0..8e180453 100644 --- a/docs/stix_overview/stix-recipes.rst +++ b/docs/stix_overview/stix-recipes.rst @@ -11,659 +11,3 @@ This section utilizes the [stix2 python library](https://github.com/oasis-open/c The recipes in this section address how to query the dataset for a single object. -#### By STIX ID - -The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. - -```python -g0075 = src.get("intrusion-set--f40eb8ce-2a74-4e56-89a1-227021410142") -``` - -#### By ATT&CK ID - -The following recipe can be used to retrieve an object according to its ATT&CK ID: - -```python -from stix2 import Filter - -g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] -``` - -Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. - -```python -from stix2 import Filter - -t1134 = src.query([ - Filter("external_references.external_id", "=", "T1134"), - Filter("type", "=", "attack-pattern") -])[0] -``` - -The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). - -#### By name - -The following recipe retrieves an object according to its name: - -```python -from stix2 import Filter - -def get_technique_by_name(thesrc, name): - filt = [ - Filter('type', '=', 'attack-pattern'), - Filter('name', '=', name) - ] - return thesrc.query(filt) -# get the technique titled "System Information Discovery" -get_technique_by_name(src, 'System Information Discovery') -``` - -#### By alias - -The following methodology can be used to find the group corresponding to a given alias: - -```python -from stix2 import Filter - -def get_group_by_alias(thesrc, alias): - return thesrc.query([ - Filter('type', '=', 'intrusion-set'), - Filter('aliases', '=', alias) - ])[0] - -get_group_by_alias(src, 'Cozy Bear') -``` - -### Getting multiple objects - -The recipes in this section address how to query the dataset for multiple objects. - -⚠ When working with queries to return objects based on a set of characteristics, it is likely that you'll end up with a few objects which are no longer maintained by ATT&CK. These are objects marked as deprecated or revoked. We keep these outdated objects around so that workflows depending on them don't break, but we recommend you avoid using them when possible. Please see the section [Working with deprecated and revoked objects](#Working-with-deprecated-and-revoked-objects) for more information. - -#### Objects by type - -See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. - -```python -from stix2 import Filter - -# use the appropriate STIX type in the query according to the desired ATT&CK type -groups = src.query([ Filter("type", "=", "intrusion-set") ]) -``` - -##### Getting techniques or sub-techniques - -ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. - -```python -from stix2 import Filter - -def get_techniques_or_subtechniques(thesrc, include="both"): - """Filter Techniques or Sub-Techniques from ATT&CK Enterprise Domain. - include argument has three options: "techniques", "subtechniques", or "both" - depending on the intended behavior.""" - if include == "techniques": - query_results = thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('x_mitre_is_subtechnique', '=', False) - ]) - elif include == "subtechniques": - query_results = thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('x_mitre_is_subtechnique', '=', True) - ]) - elif include == "both": - query_results = thesrc.query([ - Filter('type', '=', 'attack-pattern') - ]) - else: - raise RuntimeError("Unknown option %s!" % include) - - return query_results - - -subtechniques = get_techniques_or_subtechniques(src, "subtechniques") -subtechniques = remove_revoked_deprecated(subtechniques) # see https://github.com/mitre/cti/blob/master/USAGE.md#removing-revoked-and-deprecated-objects -``` - -##### Getting software - -Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. - -```python -from itertools import chain -from stix2 import Filter - -def get_software(thesrc): - return list(chain.from_iterable( - thesrc.query(f) for f in [ - Filter("type", "=", "tool"), - Filter("type", "=", "malware") - ] - )) - -get_software(src) -``` - -#### Objects by content - -Sometimes it may be useful to query objects by the content of their description: - -```python -from stix2 import Filter - -def get_techniques_by_content(thesrc, content): - techniques = src.query([ Filter('type', '=', 'attack-pattern') ]) - return list(filter(lambda t: content.lower() in t.description.lower(), techniques)) - -# Get all techniques where the string LSASS appears in the description -get_techniques_by_content(src, 'LSASS') -``` - -#### Techniques by platform - -Techniques are associated with one or more platforms. You can query the techniques -under a specific platform with the following code: - -```python -from stix2 import Filter - -def get_techniques_by_platform(thesrc, platform): - return thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('x_mitre_platforms', '=', platform) - ]) - -# get techniques in the windows platform -get_techniques_by_platform(src, 'Windows') -``` - -#### Techniques by tactic - -Techniques are related to tactics by their kill_chain_phases property. -The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. - -```python -from stix2 import Filter - -def get_tactic_techniques(thesrc, tactic): - # double checking the kill chain is MITRE ATT&CK - # note: kill_chain_name is different for other domains: - # - enterprise: "mitre-attack" - # - mobile: "mitre-mobile-attack" - # - ics: "mitre-ics-attack" - return thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('kill_chain_phases.phase_name', '=', tactic), - Filter('kill_chain_phases.kill_chain_name', '=', 'mitre-attack'), - ]) - - -# use the x_mitre_shortname as argument -get_tactic_techniques(src, 'defense-evasion') -``` - -#### Tactics by matrix - -The tactics are individual objects (`x-mitre-tactic`), and their order in a matrix (`x-mitre-matrix`) is -found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches -the ordering of the tactics in that matrix. The following recipe returns a structured list of tactics within each matrix of the input DataStore. - -```python -from stix2 import Filter - -def getTacticsByMatrix(thesrc): - tactics = {} - matrix = thesrc.query([ - Filter('type', '=', 'x-mitre-matrix'), - ]) - - for i in range(len(matrix)): - tactics[matrix[i]['name']] = [] - for tactic_id in matrix[i]['tactic_refs']: - tactics[matrix[i]['name']].append(thesrc.get(tactic_id)) - - return tactics - -# get tactic layout -getTacticsByMatrix(src) -``` - -#### Objects created or modified since a given date - -Sometimes you may want to get a list of objects which have been created or modified after a certain time. - -```python -from stix2 import Filter - -def get_created_after(thesrc, timestamp): - filt = [ - Filter('created', '>', timestamp) - ] - return thesrc.query(filt) - -get_created_after(src, "2018-10-01T00:14:20.652Z") - - -def get_modified_after(thesrc, timestamp): - filt = [ - Filter('modified', '>', timestamp) - ] - return thesrc.query(filt) - -get_modified_after(src, "2018-10-01T00:14:20.652Z") -``` - -We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to [check the list of released versions of ATT&CK](https://github.com/mitre/cti/blob/master/USAGE.md#access-a-specific-version-of-attck). - -### Getting related objects - -A large part of working with ATT&CK revolves around parsing relationships between objects. It is useful -to track not only the related object but the relationship itself because a description is often -present to contextualize the nature of the relationship. The following recipes demonstrate -some common uses of relationships. - -#### Relationships microlibrary - -NOTE: The following code is intended to be used with the ATT&CK v12 release which includes Campaign Objects. -The examples are backwards-compatible for previous versions af ATT&CK that omit those objects. - -This microlibrary can be used to build a lookup table of stixID to related objects and relationships. -The argument to each accessor function is a STIX2 MemoryStore to build the relationship mappings from. - -```python -from pprint import pprint -from stix2 import MemoryStore, Filter - -# See section below on "Removing revoked and deprecated objects" -def remove_revoked_deprecated(stix_objects): - """Remove any revoked or deprecated objects from queries made to the data source""" - # Note we use .get() because the property may not be present in the JSON data. The default is False - # if the property is not set. - return list( - filter( - lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, - stix_objects - ) - ) - -def get_related(thesrc, src_type, rel_type, target_type, reverse=False): - """build relationship mappings - params: - thesrc: MemoryStore to build relationship lookups for - src_type: source type for the relationships, e.g "attack-pattern" - rel_type: relationship type for the relationships, e.g "uses" - target_type: target type for the relationship, e.g "intrusion-set" - reverse: build reverse mapping of target to source - """ - - relationships = thesrc.query([ - Filter('type', '=', 'relationship'), - Filter('relationship_type', '=', rel_type), - Filter('revoked', '=', False), - ]) - - # See section below on "Removing revoked and deprecated objects" - relationships = remove_revoked_deprecated(relationships) - - # stix_id => [ { relationship, related_object_id } for each related object ] - id_to_related = {} - - # build the dict - for relationship in relationships: - if src_type in relationship.source_ref and target_type in relationship.target_ref: - if (relationship.source_ref in id_to_related and not reverse) or (relationship.target_ref in id_to_related and reverse): - # append to existing entry - if not reverse: - id_to_related[relationship.source_ref].append({ - "relationship": relationship, - "id": relationship.target_ref - }) - else: - id_to_related[relationship.target_ref].append({ - "relationship": relationship, - "id": relationship.source_ref - }) - else: - # create a new entry - if not reverse: - id_to_related[relationship.source_ref] = [{ - "relationship": relationship, - "id": relationship.target_ref - }] - else: - id_to_related[relationship.target_ref] = [{ - "relationship": relationship, - "id": relationship.source_ref - }] - # all objects of relevant type - if not reverse: - targets = thesrc.query([ - Filter('type', '=', target_type), - Filter('revoked', '=', False) - ]) - else: - targets = thesrc.query([ - Filter('type', '=', src_type), - Filter('revoked', '=', False) - ]) - - # build lookup of stixID to stix object - id_to_target = {} - for target in targets: - id_to_target[target.id] = target - - # build final output mappings - output = {} - for stix_id in id_to_related: - value = [] - for related in id_to_related[stix_id]: - if not related["id"] in id_to_target: - continue # targeting a revoked object - value.append({ - "object": id_to_target[related["id"]], - "relationship": related["relationship"] - }) - output[stix_id] = value - return output - -# software:group -def software_used_by_groups(thesrc): - """returns group_id => {software, relationship} for each software used by the group and each software used by campaigns attributed to the group.""" - # get all software used by groups - tools_used_by_group = get_related(thesrc, "intrusion-set", "uses", "tool") - malware_used_by_group = get_related(thesrc, "intrusion-set", "uses", "malware") - software_used_by_group = {**tools_used_by_group, **malware_used_by_group} # group_id -> [{software, relationship}] - - # get groups attributing to campaigns and all software used by campaigns - software_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") - malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") - for id in malware_used_by_campaign: - if id in software_used_by_campaign: - software_used_by_campaign[id].extend(malware_used_by_campaign[id]) - else: - software_used_by_campaign[id] = malware_used_by_campaign[id] - campaigns_attributed_to_group = { - "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} - "software": software_used_by_campaign # campaign_id => {software, relationship} - } - - for group_id in campaigns_attributed_to_group["campaigns"]: - software_used_by_campaigns = [] - # check if attributed campaign is using software - for campaign in campaigns_attributed_to_group["campaigns"][group_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in campaigns_attributed_to_group["software"]: - software_used_by_campaigns.extend(campaigns_attributed_to_group["software"][campaign_id]) - - # update software used by group to include software used by a groups attributed campaign - if group_id in software_used_by_group: - software_used_by_group[group_id].extend(software_used_by_campaigns) - else: - software_used_by_group[group_id] = software_used_by_campaigns - return software_used_by_group - -def groups_using_software(thesrc): - """returns software_id => {group, relationship} for each group using the software and each software used by attributed campaigns.""" - # get all groups using software - groups_using_tool = get_related(thesrc, "intrusion-set", "uses", "tool", reverse=True) - groups_using_malware = get_related(thesrc, "intrusion-set", "uses", "malware", reverse=True) - groups_using_software = {**groups_using_tool, **groups_using_malware} # software_id => {group, relationship} - - # get campaigns attributed to groups and all campaigns using software - campaigns_using_software = get_related(thesrc, "campaign", "uses", "tool", reverse=True) - campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) - for id in campaigns_using_malware: - if id in campaigns_using_software: - campaigns_using_software[id].extend(campaigns_using_malware[id]) - else: - campaigns_using_software[id] = campaigns_using_malware[id] - groups_attributing_to_campaigns = { - "campaigns": campaigns_using_software,# software_id => {campaign, relationship} - "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} - } - - for software_id in groups_attributing_to_campaigns["campaigns"]: - groups_attributed_to_campaigns = [] - # check if campaign is attributed to group - for campaign in groups_attributing_to_campaigns["campaigns"][software_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in groups_attributing_to_campaigns["groups"]: - groups_attributed_to_campaigns.extend(groups_attributing_to_campaigns["groups"][campaign_id]) - - # update groups using software to include software used by a groups attributed campaign - if software_id in groups_using_software: - groups_using_software[software_id].extend(groups_attributed_to_campaigns) - else: - groups_using_software[software_id] = groups_attributed_to_campaigns - return groups_using_software - -# software:campaign -def software_used_by_campaigns(thesrc): - """returns campaign_id => {software, relationship} for each software used by the campaign.""" - tools_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") - malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") - return {**tools_used_by_campaign, **malware_used_by_campaign} - -def campaigns_using_software(thesrc): - """returns software_id => {campaign, relationship} for each campaign using the software.""" - campaigns_using_tool = get_related(thesrc, "campaign", "uses", "tool", reverse=True) - campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) - return {**campaigns_using_tool, **campaigns_using_malware} - -# campaign:group -def groups_attributing_to_campaign(thesrc): - """returns campaign_id => {group, relationship} for each group attributing to the campaign.""" - return get_related(thesrc, "campaign", "attributed-to", "intrusion-set") - -def campaigns_attributed_to_group(thesrc): - """returns group_id => {campaign, relationship} for each campaign attributed to the group.""" - return get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True) - -# technique:group -def techniques_used_by_groups(thesrc): - """returns group_id => {technique, relationship} for each technique used by the group and each - technique used by campaigns attributed to the group.""" - # get all techniques used by groups - techniques_used_by_groups = get_related(thesrc, "intrusion-set", "uses", "attack-pattern") # group_id => {technique, relationship} - - # get groups attributing to campaigns and all techniques used by campaigns - campaigns_attributed_to_group = { - "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} - "techniques": get_related(thesrc, "campaign", "uses", "attack-pattern") # campaign_id => {technique, relationship} - } - - for group_id in campaigns_attributed_to_group["campaigns"]: - techniques_used_by_campaigns = [] - # check if attributed campaign is using technique - for campaign in campaigns_attributed_to_group["campaigns"][group_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in campaigns_attributed_to_group["techniques"]: - techniques_used_by_campaigns.extend(campaigns_attributed_to_group["techniques"][campaign_id]) - - # update techniques used by groups to include techniques used by a groups attributed campaign - if group_id in techniques_used_by_groups: - techniques_used_by_groups[group_id].extend(techniques_used_by_campaigns) - else: - techniques_used_by_groups[group_id] = techniques_used_by_campaigns - return techniques_used_by_groups - -def groups_using_technique(thesrc): - """returns technique_id => {group, relationship} for each group using the technique and each campaign attributed to groups using the technique.""" - # get all groups using techniques - groups_using_techniques = get_related(thesrc, "intrusion-set", "uses", "attack-pattern", reverse=True) # technique_id => {group, relationship} - - # get campaigns attributed to groups and all campaigns using techniques - groups_attributing_to_campaigns = { - "campaigns": get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True), # technique_id => {campaign, relationship} - "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} - } - - for technique_id in groups_attributing_to_campaigns["campaigns"]: - campaigns_attributed_to_group = [] - # check if campaign is attributed to group - for campaign in groups_attributing_to_campaigns["campaigns"][technique_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in groups_attributing_to_campaigns["groups"]: - campaigns_attributed_to_group.extend(groups_attributing_to_campaigns["groups"][campaign_id]) - - # update groups using techniques to include techniques used by a groups attributed campaign - if technique_id in groups_using_techniques: - groups_using_techniques[technique_id].extend(campaigns_attributed_to_group) - else: - groups_using_techniques[technique_id] = campaigns_attributed_to_group - return groups_using_techniques - -# technique:campaign -def techniques_used_by_campaigns(thesrc): - """returns campaign_id => {technique, relationship} for each technique used by the campaign.""" - return get_related(thesrc, "campaign", "uses", "attack-pattern") - -def campaigns_using_technique(thesrc): - """returns technique_id => {campaign, relationship} for each campaign using the technique.""" - return get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True) - -# technique:software -def techniques_used_by_software(thesrc): - """return software_id => {technique, relationship} for each technique used by the software.""" - techniques_by_tool = get_related(thesrc, "tool", "uses", "attack-pattern") - techniques_by_malware = get_related(thesrc, "malware", "uses", "attack-pattern") - return {**techniques_by_tool, **techniques_by_malware} - -def software_using_technique(thesrc): - """return technique_id => {software, relationship} for each software using the technique.""" - tools_by_technique_id = get_related(thesrc, "tool", "uses", "attack-pattern", reverse=True) - malware_by_technique_id = get_related(thesrc, "malware", "uses", "attack-pattern", reverse=True) - return {**tools_by_technique_id, **malware_by_technique_id} - -# technique:mitigation -def mitigation_mitigates_techniques(thesrc): - """return mitigation_id => {technique, relationship} for each technique mitigated by the mitigation.""" - return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=False) - -def technique_mitigated_by_mitigations(thesrc): - """return technique_id => {mitigation, relationship} for each mitigation of the technique.""" - return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=True) - -# technique:sub-technique -def subtechniques_of(thesrc): - """return technique_id => {subtechnique, relationship} for each subtechnique of the technique.""" - return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern", reverse=True) - -def parent_technique_of(thesrc): - """return subtechnique_id => {technique, relationship} describing the parent technique of the subtechnique""" - return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern")[0] - -# technique:data-component -def datacomponent_detects_techniques(thesrc): - """return datacomponent_id => {technique, relationship} describing the detections of each data component""" - return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern") - -def technique_detected_by_datacomponents(thesrc): - """return technique_id => {datacomponent, relationship} describing the data components that can detect the technique""" - return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern", reverse=True) - -# Example usage: -src = MemoryStore() -src.load_from_file("path/to/enterprise-attack.json") - -group_id_to_software = software_used_by_groups(src) -pprint(group_id_to_software["intrusion-set--2a158b0a-7ef8-43cb-9985-bf34d1e12050"]) # G0019 -# [ -# { -# "object": Malware, # S0061 -# "relationship": Relationship # relationship between G0019 and S0061 -# }, -# { -# ... -# } -# ] -``` - -#### Getting techniques used by a group's software - -Because a group uses software, and software uses techniques, groups can be considered indirect users of techniques used by their software. -These techniques are oftentimes distinct from the techniques used directly by a group, although there are occasionally intersections in these two sets of techniques. - -The following recipe can be used to retrieve the techniques used by a group's software: - -```python -from stix2.utils import get_type_from_id -from stix2 import Filter - -def get_techniques_by_group_software(thesrc, group_stix_id): - # get the malware, tools that the group uses - group_uses = [ - r for r in thesrc.relationships(group_stix_id, 'uses', source_only=True) - if get_type_from_id(r.target_ref) in ['malware', 'tool'] - ] - - # get the technique stix ids that the malware, tools use - software_uses = thesrc.query([ - Filter('type', '=', 'relationship'), - Filter('relationship_type', '=', 'uses'), - Filter('source_ref', 'in', [r.source_ref for r in group_uses]) - ]) - - #get the techniques themselves - return thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('id', 'in', [r.target_ref for r in software_uses]) - ]) - -get_techniques_by_group_software(src, "intrusion-set--f047ee18-7985-4946-8bfb-4ed754d3a0dd") -``` - -### Working with deprecated and revoked objects - -Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either `x_mitre_deprecated` or `revoked`) noting their status. In the case of revoked objects, a relationship of type `revoked-by` is also created targeting the replacing object. - -Unlike other objects in the dataset, relationships cannot be revoked or deprecated. Relationships are considered deprecated/revoked if one of the objects it is attached to is revoked or deprecated. - -#### Removing revoked and deprecated objects - -Revoked and deprecated objects are kept in the knowledge base so that workflows relying on those objects are not -broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no -longer maintained by ATT&CK. - -We recommend _not_ using built-in STIX filters for removing revoked objects (e.g `Filter('revoked', '=', False)`). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See [issue #127](https://github.com/mitre/cti/issues/127) for more details. - -```python -from stix2 import Filter - -def remove_revoked_deprecated(stix_objects): - """Remove any revoked or deprecated objects from queries made to the data source""" - # Note we use .get() because the property may not be present in the JSON data. The default is False - # if the property is not set. - return list( - filter( - lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, - stix_objects - ) - ) - -mitigations = src.query([ Filter("type", "=", "course-of-action") ]) -mitigations = remove_revoked_deprecated(mitigations) -``` - -#### Getting a revoking object - -When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: - -```python -from stix2 import Filter - -def getRevokedBy(stix_id, thesrc): - relations = thesrc.relationships(stix_id, 'revoked-by', source_only=True) - revoked_by = thesrc.query([ - Filter('id', 'in', [r.target_ref for r in relations]), - Filter('revoked', '=', False) - ]) - if revoked_by is not None: - revoked_by = revoked_by[0] - - return revoked_by - -getRevokedBy("attack-pattern--c16e5409-ee53-4d79-afdc-4099dc9292df", src) -``` From bd3436e1bdc6afbfa1eb0e4ebaf4bbe1363d1024 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 30 Jun 2023 17:10:25 -0400 Subject: [PATCH 006/159] splitting up python recipe sections --- docs/stix_overview/attack_id.rst | 25 +++++++++++++++++++++++++ docs/stix_overview/by_alias.rst | 17 +++++++++++++++++ docs/stix_overview/by_name.rst | 20 ++++++++++++++++++++ docs/stix_overview/multiple_objects.rst | 7 +++++++ docs/stix_overview/objects_by_type.rst | 13 +++++++++++++ docs/stix_overview/stix_id.rst | 9 +++++++++ 6 files changed, 91 insertions(+) create mode 100644 docs/stix_overview/attack_id.rst create mode 100644 docs/stix_overview/by_alias.rst create mode 100644 docs/stix_overview/by_name.rst create mode 100644 docs/stix_overview/multiple_objects.rst create mode 100644 docs/stix_overview/objects_by_type.rst create mode 100644 docs/stix_overview/stix_id.rst diff --git a/docs/stix_overview/attack_id.rst b/docs/stix_overview/attack_id.rst new file mode 100644 index 00000000..ff13578c --- /dev/null +++ b/docs/stix_overview/attack_id.rst @@ -0,0 +1,25 @@ +By ATT&CK ID +=============== + +#### By ATT&CK ID + +The following recipe can be used to retrieve an object according to its ATT&CK ID: + +```python +from stix2 import Filter + +g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] +``` + +Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. + +```python +from stix2 import Filter + +t1134 = src.query([ + Filter("external_references.external_id", "=", "T1134"), + Filter("type", "=", "attack-pattern") +])[0] +``` + +The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). \ No newline at end of file diff --git a/docs/stix_overview/by_alias.rst b/docs/stix_overview/by_alias.rst new file mode 100644 index 00000000..b4670991 --- /dev/null +++ b/docs/stix_overview/by_alias.rst @@ -0,0 +1,17 @@ +By Alias +=============== +#### By alias + +The following methodology can be used to find the group corresponding to a given alias: + +```python +from stix2 import Filter + +def get_group_by_alias(thesrc, alias): + return thesrc.query([ + Filter('type', '=', 'intrusion-set'), + Filter('aliases', '=', alias) + ])[0] + +get_group_by_alias(src, 'Cozy Bear') +``` \ No newline at end of file diff --git a/docs/stix_overview/by_name.rst b/docs/stix_overview/by_name.rst new file mode 100644 index 00000000..48785f0e --- /dev/null +++ b/docs/stix_overview/by_name.rst @@ -0,0 +1,20 @@ +By Name +=============== + + +#### By name + +The following recipe retrieves an object according to its name: + +```python +from stix2 import Filter + +def get_technique_by_name(thesrc, name): + filt = [ + Filter('type', '=', 'attack-pattern'), + Filter('name', '=', name) + ] + return thesrc.query(filt) +# get the technique titled "System Information Discovery" +get_technique_by_name(src, 'System Information Discovery') +``` \ No newline at end of file diff --git a/docs/stix_overview/multiple_objects.rst b/docs/stix_overview/multiple_objects.rst new file mode 100644 index 00000000..0ee3abfc --- /dev/null +++ b/docs/stix_overview/multiple_objects.rst @@ -0,0 +1,7 @@ +Getting multiple objects +=============== +### Getting multiple objects + +The recipes in this following section address how to query the dataset for multiple objects. + +⚠ When working with queries to return objects based on a set of characteristics, it is likely that you'll end up with a few objects which are no longer maintained by ATT&CK. These are objects marked as deprecated or revoked. We keep these outdated objects around so that workflows depending on them don't break, but we recommend you avoid using them when possible. Please see the section [Working with deprecated and revoked objects](#Working-with-deprecated-and-revoked-objects) for more information. \ No newline at end of file diff --git a/docs/stix_overview/objects_by_type.rst b/docs/stix_overview/objects_by_type.rst new file mode 100644 index 00000000..d3754dcd --- /dev/null +++ b/docs/stix_overview/objects_by_type.rst @@ -0,0 +1,13 @@ +Objects by type +=============== + +#### Objects by type + +See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. + +```python +from stix2 import Filter + +# use the appropriate STIX type in the query according to the desired ATT&CK type +groups = src.query([ Filter("type", "=", "intrusion-set") ]) +``` \ No newline at end of file diff --git a/docs/stix_overview/stix_id.rst b/docs/stix_overview/stix_id.rst new file mode 100644 index 00000000..716e7dcc --- /dev/null +++ b/docs/stix_overview/stix_id.rst @@ -0,0 +1,9 @@ +By STIX ID +=============== +#### By STIX ID + +The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. + +```python +g0075 = src.get("intrusion-set--f40eb8ce-2a74-4e56-89a1-227021410142") +``` \ No newline at end of file From 8ca5d8c605521068603996b61559f433131f094c Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 30 Jun 2023 17:17:19 -0400 Subject: [PATCH 007/159] adding more page splits --- .../stix_overview/getting_related_objects.rst | 8 + docs/stix_overview/getting_software.rst | 21 ++ docs/stix_overview/objects_by_content.rst | 17 + docs/stix_overview/objects_since_date.rst | 28 ++ .../relationships_microlibrary.rst | 319 ++++++++++++++++++ docs/stix_overview/tactics_by_matrix.rst | 27 ++ docs/stix_overview/techniques_by_platform.rst | 21 ++ docs/stix_overview/techniques_by_tactic.rst | 27 ++ .../techniques_subtechniques.rst | 37 ++ 9 files changed, 505 insertions(+) create mode 100644 docs/stix_overview/getting_related_objects.rst create mode 100644 docs/stix_overview/getting_software.rst create mode 100644 docs/stix_overview/objects_by_content.rst create mode 100644 docs/stix_overview/objects_since_date.rst create mode 100644 docs/stix_overview/relationships_microlibrary.rst create mode 100644 docs/stix_overview/tactics_by_matrix.rst create mode 100644 docs/stix_overview/techniques_by_platform.rst create mode 100644 docs/stix_overview/techniques_by_tactic.rst create mode 100644 docs/stix_overview/techniques_subtechniques.rst diff --git a/docs/stix_overview/getting_related_objects.rst b/docs/stix_overview/getting_related_objects.rst new file mode 100644 index 00000000..b3a48a1f --- /dev/null +++ b/docs/stix_overview/getting_related_objects.rst @@ -0,0 +1,8 @@ +Getting related objects +=============== +### Getting related objects + +A large part of working with ATT&CK revolves around parsing relationships between objects. It is useful +to track not only the related object but the relationship itself because a description is often +present to contextualize the nature of the relationship. The following recipes demonstrate +some common uses of relationships. \ No newline at end of file diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_overview/getting_software.rst new file mode 100644 index 00000000..487dd971 --- /dev/null +++ b/docs/stix_overview/getting_software.rst @@ -0,0 +1,21 @@ +Getting software +=============== + +##### Getting software + +Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. + +```python +from itertools import chain +from stix2 import Filter + +def get_software(thesrc): + return list(chain.from_iterable( + thesrc.query(f) for f in [ + Filter("type", "=", "tool"), + Filter("type", "=", "malware") + ] + )) + +get_software(src) +``` \ No newline at end of file diff --git a/docs/stix_overview/objects_by_content.rst b/docs/stix_overview/objects_by_content.rst new file mode 100644 index 00000000..9b705ac6 --- /dev/null +++ b/docs/stix_overview/objects_by_content.rst @@ -0,0 +1,17 @@ +Objects by content +=============== + +#### Objects by content + +Sometimes it may be useful to query objects by the content of their description: + +```python +from stix2 import Filter + +def get_techniques_by_content(thesrc, content): + techniques = src.query([ Filter('type', '=', 'attack-pattern') ]) + return list(filter(lambda t: content.lower() in t.description.lower(), techniques)) + +# Get all techniques where the string LSASS appears in the description +get_techniques_by_content(src, 'LSASS') +``` \ No newline at end of file diff --git a/docs/stix_overview/objects_since_date.rst b/docs/stix_overview/objects_since_date.rst new file mode 100644 index 00000000..6e520389 --- /dev/null +++ b/docs/stix_overview/objects_since_date.rst @@ -0,0 +1,28 @@ +Objects created or modified since a given date +=============== +#### Objects created or modified since a given date + +Sometimes you may want to get a list of objects which have been created or modified after a certain time. + +```python +from stix2 import Filter + +def get_created_after(thesrc, timestamp): + filt = [ + Filter('created', '>', timestamp) + ] + return thesrc.query(filt) + +get_created_after(src, "2018-10-01T00:14:20.652Z") + + +def get_modified_after(thesrc, timestamp): + filt = [ + Filter('modified', '>', timestamp) + ] + return thesrc.query(filt) + +get_modified_after(src, "2018-10-01T00:14:20.652Z") +``` + +We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to [check the list of released versions of ATT&CK](https://github.com/mitre/cti/blob/master/USAGE.md#access-a-specific-version-of-attck). diff --git a/docs/stix_overview/relationships_microlibrary.rst b/docs/stix_overview/relationships_microlibrary.rst new file mode 100644 index 00000000..055c8786 --- /dev/null +++ b/docs/stix_overview/relationships_microlibrary.rst @@ -0,0 +1,319 @@ +Relationships microlibrary +=============== + +#### Relationships microlibrary + +NOTE: The following code is intended to be used with the ATT&CK v12 release which includes Campaign Objects. +The examples are backwards-compatible for previous versions af ATT&CK that omit those objects. + +This microlibrary can be used to build a lookup table of stixID to related objects and relationships. +The argument to each accessor function is a STIX2 MemoryStore to build the relationship mappings from. + +```python +from pprint import pprint +from stix2 import MemoryStore, Filter + +# See section below on "Removing revoked and deprecated objects" +def remove_revoked_deprecated(stix_objects): + """Remove any revoked or deprecated objects from queries made to the data source""" + # Note we use .get() because the property may not be present in the JSON data. The default is False + # if the property is not set. + return list( + filter( + lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, + stix_objects + ) + ) + +def get_related(thesrc, src_type, rel_type, target_type, reverse=False): + """build relationship mappings + params: + thesrc: MemoryStore to build relationship lookups for + src_type: source type for the relationships, e.g "attack-pattern" + rel_type: relationship type for the relationships, e.g "uses" + target_type: target type for the relationship, e.g "intrusion-set" + reverse: build reverse mapping of target to source + """ + + relationships = thesrc.query([ + Filter('type', '=', 'relationship'), + Filter('relationship_type', '=', rel_type), + Filter('revoked', '=', False), + ]) + + # See section below on "Removing revoked and deprecated objects" + relationships = remove_revoked_deprecated(relationships) + + # stix_id => [ { relationship, related_object_id } for each related object ] + id_to_related = {} + + # build the dict + for relationship in relationships: + if src_type in relationship.source_ref and target_type in relationship.target_ref: + if (relationship.source_ref in id_to_related and not reverse) or (relationship.target_ref in id_to_related and reverse): + # append to existing entry + if not reverse: + id_to_related[relationship.source_ref].append({ + "relationship": relationship, + "id": relationship.target_ref + }) + else: + id_to_related[relationship.target_ref].append({ + "relationship": relationship, + "id": relationship.source_ref + }) + else: + # create a new entry + if not reverse: + id_to_related[relationship.source_ref] = [{ + "relationship": relationship, + "id": relationship.target_ref + }] + else: + id_to_related[relationship.target_ref] = [{ + "relationship": relationship, + "id": relationship.source_ref + }] + # all objects of relevant type + if not reverse: + targets = thesrc.query([ + Filter('type', '=', target_type), + Filter('revoked', '=', False) + ]) + else: + targets = thesrc.query([ + Filter('type', '=', src_type), + Filter('revoked', '=', False) + ]) + + # build lookup of stixID to stix object + id_to_target = {} + for target in targets: + id_to_target[target.id] = target + + # build final output mappings + output = {} + for stix_id in id_to_related: + value = [] + for related in id_to_related[stix_id]: + if not related["id"] in id_to_target: + continue # targeting a revoked object + value.append({ + "object": id_to_target[related["id"]], + "relationship": related["relationship"] + }) + output[stix_id] = value + return output + +# software:group +def software_used_by_groups(thesrc): + """returns group_id => {software, relationship} for each software used by the group and each software used by campaigns attributed to the group.""" + # get all software used by groups + tools_used_by_group = get_related(thesrc, "intrusion-set", "uses", "tool") + malware_used_by_group = get_related(thesrc, "intrusion-set", "uses", "malware") + software_used_by_group = {**tools_used_by_group, **malware_used_by_group} # group_id -> [{software, relationship}] + + # get groups attributing to campaigns and all software used by campaigns + software_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") + malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") + for id in malware_used_by_campaign: + if id in software_used_by_campaign: + software_used_by_campaign[id].extend(malware_used_by_campaign[id]) + else: + software_used_by_campaign[id] = malware_used_by_campaign[id] + campaigns_attributed_to_group = { + "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} + "software": software_used_by_campaign # campaign_id => {software, relationship} + } + + for group_id in campaigns_attributed_to_group["campaigns"]: + software_used_by_campaigns = [] + # check if attributed campaign is using software + for campaign in campaigns_attributed_to_group["campaigns"][group_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in campaigns_attributed_to_group["software"]: + software_used_by_campaigns.extend(campaigns_attributed_to_group["software"][campaign_id]) + + # update software used by group to include software used by a groups attributed campaign + if group_id in software_used_by_group: + software_used_by_group[group_id].extend(software_used_by_campaigns) + else: + software_used_by_group[group_id] = software_used_by_campaigns + return software_used_by_group + +def groups_using_software(thesrc): + """returns software_id => {group, relationship} for each group using the software and each software used by attributed campaigns.""" + # get all groups using software + groups_using_tool = get_related(thesrc, "intrusion-set", "uses", "tool", reverse=True) + groups_using_malware = get_related(thesrc, "intrusion-set", "uses", "malware", reverse=True) + groups_using_software = {**groups_using_tool, **groups_using_malware} # software_id => {group, relationship} + + # get campaigns attributed to groups and all campaigns using software + campaigns_using_software = get_related(thesrc, "campaign", "uses", "tool", reverse=True) + campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) + for id in campaigns_using_malware: + if id in campaigns_using_software: + campaigns_using_software[id].extend(campaigns_using_malware[id]) + else: + campaigns_using_software[id] = campaigns_using_malware[id] + groups_attributing_to_campaigns = { + "campaigns": campaigns_using_software,# software_id => {campaign, relationship} + "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} + } + + for software_id in groups_attributing_to_campaigns["campaigns"]: + groups_attributed_to_campaigns = [] + # check if campaign is attributed to group + for campaign in groups_attributing_to_campaigns["campaigns"][software_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in groups_attributing_to_campaigns["groups"]: + groups_attributed_to_campaigns.extend(groups_attributing_to_campaigns["groups"][campaign_id]) + + # update groups using software to include software used by a groups attributed campaign + if software_id in groups_using_software: + groups_using_software[software_id].extend(groups_attributed_to_campaigns) + else: + groups_using_software[software_id] = groups_attributed_to_campaigns + return groups_using_software + +# software:campaign +def software_used_by_campaigns(thesrc): + """returns campaign_id => {software, relationship} for each software used by the campaign.""" + tools_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") + malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") + return {**tools_used_by_campaign, **malware_used_by_campaign} + +def campaigns_using_software(thesrc): + """returns software_id => {campaign, relationship} for each campaign using the software.""" + campaigns_using_tool = get_related(thesrc, "campaign", "uses", "tool", reverse=True) + campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) + return {**campaigns_using_tool, **campaigns_using_malware} + +# campaign:group +def groups_attributing_to_campaign(thesrc): + """returns campaign_id => {group, relationship} for each group attributing to the campaign.""" + return get_related(thesrc, "campaign", "attributed-to", "intrusion-set") + +def campaigns_attributed_to_group(thesrc): + """returns group_id => {campaign, relationship} for each campaign attributed to the group.""" + return get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True) + +# technique:group +def techniques_used_by_groups(thesrc): + """returns group_id => {technique, relationship} for each technique used by the group and each + technique used by campaigns attributed to the group.""" + # get all techniques used by groups + techniques_used_by_groups = get_related(thesrc, "intrusion-set", "uses", "attack-pattern") # group_id => {technique, relationship} + + # get groups attributing to campaigns and all techniques used by campaigns + campaigns_attributed_to_group = { + "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} + "techniques": get_related(thesrc, "campaign", "uses", "attack-pattern") # campaign_id => {technique, relationship} + } + + for group_id in campaigns_attributed_to_group["campaigns"]: + techniques_used_by_campaigns = [] + # check if attributed campaign is using technique + for campaign in campaigns_attributed_to_group["campaigns"][group_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in campaigns_attributed_to_group["techniques"]: + techniques_used_by_campaigns.extend(campaigns_attributed_to_group["techniques"][campaign_id]) + + # update techniques used by groups to include techniques used by a groups attributed campaign + if group_id in techniques_used_by_groups: + techniques_used_by_groups[group_id].extend(techniques_used_by_campaigns) + else: + techniques_used_by_groups[group_id] = techniques_used_by_campaigns + return techniques_used_by_groups + +def groups_using_technique(thesrc): + """returns technique_id => {group, relationship} for each group using the technique and each campaign attributed to groups using the technique.""" + # get all groups using techniques + groups_using_techniques = get_related(thesrc, "intrusion-set", "uses", "attack-pattern", reverse=True) # technique_id => {group, relationship} + + # get campaigns attributed to groups and all campaigns using techniques + groups_attributing_to_campaigns = { + "campaigns": get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True), # technique_id => {campaign, relationship} + "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} + } + + for technique_id in groups_attributing_to_campaigns["campaigns"]: + campaigns_attributed_to_group = [] + # check if campaign is attributed to group + for campaign in groups_attributing_to_campaigns["campaigns"][technique_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in groups_attributing_to_campaigns["groups"]: + campaigns_attributed_to_group.extend(groups_attributing_to_campaigns["groups"][campaign_id]) + + # update groups using techniques to include techniques used by a groups attributed campaign + if technique_id in groups_using_techniques: + groups_using_techniques[technique_id].extend(campaigns_attributed_to_group) + else: + groups_using_techniques[technique_id] = campaigns_attributed_to_group + return groups_using_techniques + +# technique:campaign +def techniques_used_by_campaigns(thesrc): + """returns campaign_id => {technique, relationship} for each technique used by the campaign.""" + return get_related(thesrc, "campaign", "uses", "attack-pattern") + +def campaigns_using_technique(thesrc): + """returns technique_id => {campaign, relationship} for each campaign using the technique.""" + return get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True) + +# technique:software +def techniques_used_by_software(thesrc): + """return software_id => {technique, relationship} for each technique used by the software.""" + techniques_by_tool = get_related(thesrc, "tool", "uses", "attack-pattern") + techniques_by_malware = get_related(thesrc, "malware", "uses", "attack-pattern") + return {**techniques_by_tool, **techniques_by_malware} + +def software_using_technique(thesrc): + """return technique_id => {software, relationship} for each software using the technique.""" + tools_by_technique_id = get_related(thesrc, "tool", "uses", "attack-pattern", reverse=True) + malware_by_technique_id = get_related(thesrc, "malware", "uses", "attack-pattern", reverse=True) + return {**tools_by_technique_id, **malware_by_technique_id} + +# technique:mitigation +def mitigation_mitigates_techniques(thesrc): + """return mitigation_id => {technique, relationship} for each technique mitigated by the mitigation.""" + return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=False) + +def technique_mitigated_by_mitigations(thesrc): + """return technique_id => {mitigation, relationship} for each mitigation of the technique.""" + return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=True) + +# technique:sub-technique +def subtechniques_of(thesrc): + """return technique_id => {subtechnique, relationship} for each subtechnique of the technique.""" + return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern", reverse=True) + +def parent_technique_of(thesrc): + """return subtechnique_id => {technique, relationship} describing the parent technique of the subtechnique""" + return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern")[0] + +# technique:data-component +def datacomponent_detects_techniques(thesrc): + """return datacomponent_id => {technique, relationship} describing the detections of each data component""" + return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern") + +def technique_detected_by_datacomponents(thesrc): + """return technique_id => {datacomponent, relationship} describing the data components that can detect the technique""" + return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern", reverse=True) + +# Example usage: +src = MemoryStore() +src.load_from_file("path/to/enterprise-attack.json") + +group_id_to_software = software_used_by_groups(src) +pprint(group_id_to_software["intrusion-set--2a158b0a-7ef8-43cb-9985-bf34d1e12050"]) # G0019 +# [ +# { +# "object": Malware, # S0061 +# "relationship": Relationship # relationship between G0019 and S0061 +# }, +# { +# ... +# } +# ] +``` \ No newline at end of file diff --git a/docs/stix_overview/tactics_by_matrix.rst b/docs/stix_overview/tactics_by_matrix.rst new file mode 100644 index 00000000..84e68ba5 --- /dev/null +++ b/docs/stix_overview/tactics_by_matrix.rst @@ -0,0 +1,27 @@ +Tactics by matrix +=============== +#### Tactics by matrix + +The tactics are individual objects (`x-mitre-tactic`), and their order in a matrix (`x-mitre-matrix`) is +found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches +the ordering of the tactics in that matrix. The following recipe returns a structured list of tactics within each matrix of the input DataStore. + +```python +from stix2 import Filter + +def getTacticsByMatrix(thesrc): + tactics = {} + matrix = thesrc.query([ + Filter('type', '=', 'x-mitre-matrix'), + ]) + + for i in range(len(matrix)): + tactics[matrix[i]['name']] = [] + for tactic_id in matrix[i]['tactic_refs']: + tactics[matrix[i]['name']].append(thesrc.get(tactic_id)) + + return tactics + +# get tactic layout +getTacticsByMatrix(src) +``` \ No newline at end of file diff --git a/docs/stix_overview/techniques_by_platform.rst b/docs/stix_overview/techniques_by_platform.rst new file mode 100644 index 00000000..ae2584bc --- /dev/null +++ b/docs/stix_overview/techniques_by_platform.rst @@ -0,0 +1,21 @@ +Techniques by platform +=============== + + +#### Techniques by platform + +Techniques are associated with one or more platforms. You can query the techniques +under a specific platform with the following code: + +```python +from stix2 import Filter + +def get_techniques_by_platform(thesrc, platform): + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_platforms', '=', platform) + ]) + +# get techniques in the windows platform +get_techniques_by_platform(src, 'Windows') +``` diff --git a/docs/stix_overview/techniques_by_tactic.rst b/docs/stix_overview/techniques_by_tactic.rst new file mode 100644 index 00000000..5d345dc2 --- /dev/null +++ b/docs/stix_overview/techniques_by_tactic.rst @@ -0,0 +1,27 @@ +Techniques by tactic +=============== + +#### Techniques by tactic + +Techniques are related to tactics by their kill_chain_phases property. +The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. + +```python +from stix2 import Filter + +def get_tactic_techniques(thesrc, tactic): + # double checking the kill chain is MITRE ATT&CK + # note: kill_chain_name is different for other domains: + # - enterprise: "mitre-attack" + # - mobile: "mitre-mobile-attack" + # - ics: "mitre-ics-attack" + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('kill_chain_phases.phase_name', '=', tactic), + Filter('kill_chain_phases.kill_chain_name', '=', 'mitre-attack'), + ]) + + +# use the x_mitre_shortname as argument +get_tactic_techniques(src, 'defense-evasion') +``` \ No newline at end of file diff --git a/docs/stix_overview/techniques_subtechniques.rst b/docs/stix_overview/techniques_subtechniques.rst new file mode 100644 index 00000000..b18aaa07 --- /dev/null +++ b/docs/stix_overview/techniques_subtechniques.rst @@ -0,0 +1,37 @@ +Getting techniques or sub-techniques +=============== + +##### Getting techniques or sub-techniques + +ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. + +```python +from stix2 import Filter + +def get_techniques_or_subtechniques(thesrc, include="both"): + """Filter Techniques or Sub-Techniques from ATT&CK Enterprise Domain. + include argument has three options: "techniques", "subtechniques", or "both" + depending on the intended behavior.""" + if include == "techniques": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_is_subtechnique', '=', False) + ]) + elif include == "subtechniques": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_is_subtechnique', '=', True) + ]) + elif include == "both": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern') + ]) + else: + raise RuntimeError("Unknown option %s!" % include) + + return query_results + + +subtechniques = get_techniques_or_subtechniques(src, "subtechniques") +subtechniques = remove_revoked_deprecated(subtechniques) # see https://github.com/mitre/cti/blob/master/USAGE.md#removing-revoked-and-deprecated-objects +``` \ No newline at end of file From f6e8891d35c48745e2c53e0070e7fb1a376022b2 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 30 Jun 2023 17:19:57 -0400 Subject: [PATCH 008/159] added final sections --- docs/stix_overview/deprecated_revoked.rst | 8 +++++ docs/stix_overview/getting_revoked_object.rst | 23 ++++++++++++ .../remove_revoked_deprecated.rst | 28 +++++++++++++++ docs/stix_overview/techniques_by_group_sw.rst | 35 +++++++++++++++++++ 4 files changed, 94 insertions(+) create mode 100644 docs/stix_overview/deprecated_revoked.rst create mode 100644 docs/stix_overview/getting_revoked_object.rst create mode 100644 docs/stix_overview/remove_revoked_deprecated.rst create mode 100644 docs/stix_overview/techniques_by_group_sw.rst diff --git a/docs/stix_overview/deprecated_revoked.rst b/docs/stix_overview/deprecated_revoked.rst new file mode 100644 index 00000000..2704c75e --- /dev/null +++ b/docs/stix_overview/deprecated_revoked.rst @@ -0,0 +1,8 @@ +Working with deprecated and revoked objects +=============== + +### Working with deprecated and revoked objects + +Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either `x_mitre_deprecated` or `revoked`) noting their status. In the case of revoked objects, a relationship of type `revoked-by` is also created targeting the replacing object. + +Unlike other objects in the dataset, relationships cannot be revoked or deprecated. Relationships are considered deprecated/revoked if one of the objects it is attached to is revoked or deprecated. \ No newline at end of file diff --git a/docs/stix_overview/getting_revoked_object.rst b/docs/stix_overview/getting_revoked_object.rst new file mode 100644 index 00000000..92dc48fa --- /dev/null +++ b/docs/stix_overview/getting_revoked_object.rst @@ -0,0 +1,23 @@ +Getting a revoking object +=============== + +#### Getting a revoking object + +When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: + +```python +from stix2 import Filter + +def getRevokedBy(stix_id, thesrc): + relations = thesrc.relationships(stix_id, 'revoked-by', source_only=True) + revoked_by = thesrc.query([ + Filter('id', 'in', [r.target_ref for r in relations]), + Filter('revoked', '=', False) + ]) + if revoked_by is not None: + revoked_by = revoked_by[0] + + return revoked_by + +getRevokedBy("attack-pattern--c16e5409-ee53-4d79-afdc-4099dc9292df", src) +``` \ No newline at end of file diff --git a/docs/stix_overview/remove_revoked_deprecated.rst b/docs/stix_overview/remove_revoked_deprecated.rst new file mode 100644 index 00000000..264c60ce --- /dev/null +++ b/docs/stix_overview/remove_revoked_deprecated.rst @@ -0,0 +1,28 @@ +Removing revoked and deprecated objects +=============== + +#### Removing revoked and deprecated objects + +Revoked and deprecated objects are kept in the knowledge base so that workflows relying on those objects are not +broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no +longer maintained by ATT&CK. + +We recommend _not_ using built-in STIX filters for removing revoked objects (e.g `Filter('revoked', '=', False)`). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See [issue #127](https://github.com/mitre/cti/issues/127) for more details. + +```python +from stix2 import Filter + +def remove_revoked_deprecated(stix_objects): + """Remove any revoked or deprecated objects from queries made to the data source""" + # Note we use .get() because the property may not be present in the JSON data. The default is False + # if the property is not set. + return list( + filter( + lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, + stix_objects + ) + ) + +mitigations = src.query([ Filter("type", "=", "course-of-action") ]) +mitigations = remove_revoked_deprecated(mitigations) +``` \ No newline at end of file diff --git a/docs/stix_overview/techniques_by_group_sw.rst b/docs/stix_overview/techniques_by_group_sw.rst new file mode 100644 index 00000000..27a21cc3 --- /dev/null +++ b/docs/stix_overview/techniques_by_group_sw.rst @@ -0,0 +1,35 @@ +Getting techniques used by a group's software +=============== +#### Getting techniques used by a group's software + +Because a group uses software, and software uses techniques, groups can be considered indirect users of techniques used by their software. +These techniques are oftentimes distinct from the techniques used directly by a group, although there are occasionally intersections in these two sets of techniques. + +The following recipe can be used to retrieve the techniques used by a group's software: + +```python +from stix2.utils import get_type_from_id +from stix2 import Filter + +def get_techniques_by_group_software(thesrc, group_stix_id): + # get the malware, tools that the group uses + group_uses = [ + r for r in thesrc.relationships(group_stix_id, 'uses', source_only=True) + if get_type_from_id(r.target_ref) in ['malware', 'tool'] + ] + + # get the technique stix ids that the malware, tools use + software_uses = thesrc.query([ + Filter('type', '=', 'relationship'), + Filter('relationship_type', '=', 'uses'), + Filter('source_ref', 'in', [r.source_ref for r in group_uses]) + ]) + + #get the techniques themselves + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('id', 'in', [r.target_ref for r in software_uses]) + ]) + +get_techniques_by_group_software(src, "intrusion-set--f047ee18-7985-4946-8bfb-4ed754d3a0dd") +``` \ No newline at end of file From 11a37af42e7bb427a992b3bdb62ee9632d6bcf22 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 3 Jul 2023 10:16:23 -0400 Subject: [PATCH 009/159] Update index.rst --- docs/index.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/index.rst b/docs/index.rst index 0220f7ca..d7727f65 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -30,6 +30,21 @@ other modules in this library under "Additional Modules". stix_overview/getting-started stix_overview/stix-recipes + stix_overview/attack_id + stix_overview/by_alias + stix_overview/by_name + stix_overview/deprecated_revoked + stix_overview/getting_related_objects + stix_overview/getting_revoked_objects + stix_overview/getting_software + stix_overview/multiple_objects + stix_overview/objects_by_content + stix_overview/objects_by_type + stix_overview/objects_since_date + stix_overview/relationships_microlibrary + stix_overview/removed_revoked_deprecated + + .. toctree:: :maxdepth: 1 From 8cdfb000cab8eec9bd35c2e87682fa9cb6a3f27c Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 3 Jul 2023 10:22:50 -0400 Subject: [PATCH 010/159] Update index.rst --- docs/index.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/index.rst b/docs/index.rst index d7727f65..790c7b7e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -43,6 +43,11 @@ other modules in this library under "Additional Modules". stix_overview/objects_since_date stix_overview/relationships_microlibrary stix_overview/removed_revoked_deprecated + stix_overview/stix_recipes + stix_overview/stix_id + stix_overview/tactics_by_matrix + stix_overview/techniques_by_group_sw + stix_overview/techniques_by_platform From 43c46de58df8e103a6cacee05bd76975cc045a4c Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 3 Jul 2023 10:27:22 -0400 Subject: [PATCH 011/159] Update index.rst --- docs/index.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 790c7b7e..a0f2044d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -41,13 +41,15 @@ other modules in this library under "Additional Modules". stix_overview/objects_by_content stix_overview/objects_by_type stix_overview/objects_since_date - stix_overview/relationships_microlibrary - stix_overview/removed_revoked_deprecated - stix_overview/stix_recipes stix_overview/stix_id stix_overview/tactics_by_matrix - stix_overview/techniques_by_group_sw stix_overview/techniques_by_platform + stix_overview/techniques_by_platform + stix_overview/techniques_by_tactic + stix_overview/techniques_subtechniques + stix_overview/relationships_microlibrary + stix_overview/techniques_by_group_sw + stix_overview/removed_revoked_deprecated From a7fb3cdb982e20ec28b32f2e14ba158e2d46148a Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 3 Jul 2023 10:36:28 -0400 Subject: [PATCH 012/159] Update index.rst --- docs/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index a0f2044d..932eee08 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -35,7 +35,6 @@ other modules in this library under "Additional Modules". stix_overview/by_name stix_overview/deprecated_revoked stix_overview/getting_related_objects - stix_overview/getting_revoked_objects stix_overview/getting_software stix_overview/multiple_objects stix_overview/objects_by_content @@ -49,7 +48,8 @@ other modules in this library under "Additional Modules". stix_overview/techniques_subtechniques stix_overview/relationships_microlibrary stix_overview/techniques_by_group_sw - stix_overview/removed_revoked_deprecated + stix_overview/remove_revoked_deprecated + stix_overview/getting_revoked_objects From ca3383a66b3c703d90da9e93984984ef868a985c Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:08:43 -0400 Subject: [PATCH 013/159] Update attack_id.rst --- docs/stix_overview/attack_id.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/stix_overview/attack_id.rst b/docs/stix_overview/attack_id.rst index ff13578c..1f7a8e72 100644 --- a/docs/stix_overview/attack_id.rst +++ b/docs/stix_overview/attack_id.rst @@ -5,21 +5,20 @@ By ATT&CK ID The following recipe can be used to retrieve an object according to its ATT&CK ID: -```python +.. code-block:: python from stix2 import Filter g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] -``` + Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. -```python +.. code-block:: python from stix2 import Filter t1134 = src.query([ Filter("external_references.external_id", "=", "T1134"), Filter("type", "=", "attack-pattern") ])[0] -``` -The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). \ No newline at end of file +The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). From 7bd6640d5d7c177f90060fd64d4a6842ef578c1d Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:08:55 -0400 Subject: [PATCH 014/159] Update by_alias.rst --- docs/stix_overview/by_alias.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/stix_overview/by_alias.rst b/docs/stix_overview/by_alias.rst index b4670991..049bb76a 100644 --- a/docs/stix_overview/by_alias.rst +++ b/docs/stix_overview/by_alias.rst @@ -4,7 +4,7 @@ By Alias The following methodology can be used to find the group corresponding to a given alias: -```python +.. code-block:: python from stix2 import Filter def get_group_by_alias(thesrc, alias): @@ -14,4 +14,3 @@ def get_group_by_alias(thesrc, alias): ])[0] get_group_by_alias(src, 'Cozy Bear') -``` \ No newline at end of file From 367b0de630e976f69b41dcf942d855ea6d87f393 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:09:04 -0400 Subject: [PATCH 015/159] Update by_name.rst --- docs/stix_overview/by_name.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/stix_overview/by_name.rst b/docs/stix_overview/by_name.rst index 48785f0e..2be1ac8a 100644 --- a/docs/stix_overview/by_name.rst +++ b/docs/stix_overview/by_name.rst @@ -6,7 +6,7 @@ By Name The following recipe retrieves an object according to its name: -```python +.. code-block:: python from stix2 import Filter def get_technique_by_name(thesrc, name): @@ -17,4 +17,3 @@ def get_technique_by_name(thesrc, name): return thesrc.query(filt) # get the technique titled "System Information Discovery" get_technique_by_name(src, 'System Information Discovery') -``` \ No newline at end of file From b103691d1407c8a3cf78a281463442764b2a8a74 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:10:49 -0400 Subject: [PATCH 016/159] Update getting-started.rst --- docs/stix_overview/getting-started.rst | 44 +++++++++++++------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/getting-started.rst index d63870a8..3cd87942 100644 --- a/docs/stix_overview/getting-started.rst +++ b/docs/stix_overview/getting-started.rst @@ -23,15 +23,15 @@ Before installing requirements, we recommend setting up a virtual environment: [stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: -```python +.. code-block:: python from stix2 import Filter -``` + However, if you are aiming to extend the ATT&CK dataset with new objects or implement complex workflows, you may need to use the `v20` specifier for some imports. This ensures that the objects use the STIX 2.0 API instead of the STIX 2.1 API. For example: -```python +.. code-block:: python from stix2.v20 import AttackPattern -``` + You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). @@ -41,9 +41,9 @@ You can see a full list of the classes which have versioned imports [here](https If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. -```python +.. code-block:: python from taxii2client.v20 import Collection -``` + ### Access local content @@ -58,22 +58,22 @@ Many users may opt to access the ATT&CK content via a local copy of the STIX dat Each domain in this repo is formatted according to the [STIX2 FileSystem spec](https://stix2.readthedocs.io/en/latest/guide/filesystem.html). Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: -```python +.. code-block:: python from stix2 import FileSystemSource src = FileSystemSource('./cti/enterprise-attack') -``` + #### Access via bundle If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: -```python +.. code-block:: python from stix2 import MemoryStore src = MemoryStore() src.load_from_file("enterprise-attack.json") -``` + ### Access live content @@ -94,7 +94,7 @@ Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII You can also get a list of available collection from the server directly: -```python +.. code-block:: python from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 server = Server("https://cti-taxii.mitre.org/taxii/") @@ -102,11 +102,11 @@ api_root = server.api_roots[0] # Print name and ID of all ATT&CK domains available as collections for collection in api_root.collections: print(collection.title.ljust(20) + collection.id) -``` + The following recipe demonstrates how to access the enterprise-attack data from the TAXII server. -```python +.. code-block:: python from stix2 import TAXIICollectionSource from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 @@ -118,7 +118,7 @@ collections = { collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/") src = TAXIICollectionSource(collection) -``` + For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). @@ -128,7 +128,7 @@ Users can alternatively access the data from MITRE/CTI using HTTP requests, and While typically the TAXII method is more desirable for "live" access, this method can be useful if you want to access data on a branch of the MITRE/CTI repo (the TAXII server only holds the master branch) or in the case of a TAXII server outage. -```python +.. code-block:: python import requests from stix2 import MemoryStore @@ -138,7 +138,7 @@ def get_data_from_branch(domain, branch="master"): return MemoryStore(stix_data=stix_json["objects"]) src = get_data_from_branch("enterprise-attack") -``` + ### Access a specific version of ATT&CK @@ -146,7 +146,7 @@ ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.co In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: -```python +.. code-block:: python import requests from stix2 import MemoryStore @@ -156,11 +156,11 @@ def get_data_from_version(domain, version): return MemoryStore(stix_data=stix_json["objects"]) src = get_data_from_version("enterprise-attack", "5.2") -``` + You can get a list of ATT&CK versions programmatically using the github API: -```python +.. code-block:: python import requests import re @@ -168,7 +168,7 @@ refToTag = re.compile(r"ATT&CK-v(.*)") tags = requests.get("https://api.github.com/repos/mitre/cti/git/refs/tags").json() versions = list(map(lambda tag: refToTag.search(tag["ref"]).groups()[0] , filter(lambda tag: "ATT&CK-v" in tag["ref"], tags))) # versions = ["1.0", "2.0", ...] -``` + ### Access multiple domains simultaneously @@ -177,11 +177,11 @@ with a single domain at a time. While oftentimes the hard separation of domains domains into a single DataStore. Use any of the methods above to acquire the individual datastores, and then use the following approach to combine them into a single CompositeDataSource: -```python +.. code-block:: python from stix2 import CompositeDataSource src = CompositeDataSource() src.add_data_sources([enterprise_attack_src, mobile_attack_src, ics_attack_src]) -``` + You can then use this CompositeDataSource just as you would the DataSource for an individual domain. From 7266602ec75d1f9866875ce5fe5aa9dca7cb9d2b Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:11:06 -0400 Subject: [PATCH 017/159] Update getting_revoked_object.rst --- docs/stix_overview/getting_revoked_object.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/stix_overview/getting_revoked_object.rst b/docs/stix_overview/getting_revoked_object.rst index 92dc48fa..d45ece23 100644 --- a/docs/stix_overview/getting_revoked_object.rst +++ b/docs/stix_overview/getting_revoked_object.rst @@ -5,7 +5,7 @@ Getting a revoking object When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: -```python +.. code-block:: python from stix2 import Filter def getRevokedBy(stix_id, thesrc): @@ -20,4 +20,3 @@ def getRevokedBy(stix_id, thesrc): return revoked_by getRevokedBy("attack-pattern--c16e5409-ee53-4d79-afdc-4099dc9292df", src) -``` \ No newline at end of file From 635083feaa4488806a88a3bd4211636e26bc18f8 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:11:16 -0400 Subject: [PATCH 018/159] Update getting_software.rst --- docs/stix_overview/getting_software.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_overview/getting_software.rst index 487dd971..607c4372 100644 --- a/docs/stix_overview/getting_software.rst +++ b/docs/stix_overview/getting_software.rst @@ -5,7 +5,7 @@ Getting software Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. -```python +.. code-block:: python from itertools import chain from stix2 import Filter @@ -18,4 +18,3 @@ def get_software(thesrc): )) get_software(src) -``` \ No newline at end of file From 7a22c09bdd06db961a1646b1ad5e24683777aef4 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:11:33 -0400 Subject: [PATCH 019/159] Update objects_by_content.rst --- docs/stix_overview/objects_by_content.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/stix_overview/objects_by_content.rst b/docs/stix_overview/objects_by_content.rst index 9b705ac6..d47c1f0a 100644 --- a/docs/stix_overview/objects_by_content.rst +++ b/docs/stix_overview/objects_by_content.rst @@ -5,7 +5,7 @@ Objects by content Sometimes it may be useful to query objects by the content of their description: -```python +.. code-block:: python from stix2 import Filter def get_techniques_by_content(thesrc, content): @@ -14,4 +14,3 @@ def get_techniques_by_content(thesrc, content): # Get all techniques where the string LSASS appears in the description get_techniques_by_content(src, 'LSASS') -``` \ No newline at end of file From 16fcc280efb4286ce3a8a0bdf867d51d64ab87e9 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:11:42 -0400 Subject: [PATCH 020/159] Update objects_by_type.rst --- docs/stix_overview/objects_by_type.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/stix_overview/objects_by_type.rst b/docs/stix_overview/objects_by_type.rst index d3754dcd..92360db2 100644 --- a/docs/stix_overview/objects_by_type.rst +++ b/docs/stix_overview/objects_by_type.rst @@ -5,9 +5,8 @@ Objects by type See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. -```python +.. code-block:: python from stix2 import Filter # use the appropriate STIX type in the query according to the desired ATT&CK type groups = src.query([ Filter("type", "=", "intrusion-set") ]) -``` \ No newline at end of file From 9a679508dc21c35e412b032178347be7269763e2 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:11:55 -0400 Subject: [PATCH 021/159] Update objects_since_date.rst --- docs/stix_overview/objects_since_date.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/stix_overview/objects_since_date.rst b/docs/stix_overview/objects_since_date.rst index 6e520389..7418bc2f 100644 --- a/docs/stix_overview/objects_since_date.rst +++ b/docs/stix_overview/objects_since_date.rst @@ -4,7 +4,7 @@ Objects created or modified since a given date Sometimes you may want to get a list of objects which have been created or modified after a certain time. -```python +.. code-block:: python from stix2 import Filter def get_created_after(thesrc, timestamp): @@ -23,6 +23,6 @@ def get_modified_after(thesrc, timestamp): return thesrc.query(filt) get_modified_after(src, "2018-10-01T00:14:20.652Z") -``` + We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to [check the list of released versions of ATT&CK](https://github.com/mitre/cti/blob/master/USAGE.md#access-a-specific-version-of-attck). From c930c2c4dd6139fd20514d9b06276a4bee2b9f07 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 5 Jul 2023 13:17:22 -0400 Subject: [PATCH 022/159] restyling code blocks --- docs/stix_overview/attack_id.rst | 14 +- docs/stix_overview/by_alias.rst | 16 +- docs/stix_overview/by_name.rst | 18 +- docs/stix_overview/getting-started.rst | 90 +-- docs/stix_overview/getting_revoked_object.rst | 22 +- docs/stix_overview/getting_software.rst | 18 +- docs/stix_overview/objects_by_content.rst | 12 +- docs/stix_overview/objects_by_type.rst | 6 +- docs/stix_overview/objects_since_date.rst | 28 +- .../relationships_microlibrary.rst | 605 +++++++++--------- .../remove_revoked_deprecated.rst | 27 +- docs/stix_overview/stix_id.rst | 5 +- docs/stix_overview/tactics_by_matrix.rst | 33 +- docs/stix_overview/techniques_by_group_sw.rst | 43 +- docs/stix_overview/techniques_by_platform.rst | 20 +- docs/stix_overview/techniques_by_tactic.rst | 31 +- .../techniques_subtechniques.rst | 51 +- 17 files changed, 516 insertions(+), 523 deletions(-) diff --git a/docs/stix_overview/attack_id.rst b/docs/stix_overview/attack_id.rst index 1f7a8e72..414ee1b1 100644 --- a/docs/stix_overview/attack_id.rst +++ b/docs/stix_overview/attack_id.rst @@ -6,19 +6,19 @@ By ATT&CK ID The following recipe can be used to retrieve an object according to its ATT&CK ID: .. code-block:: python -from stix2 import Filter + from stix2 import Filter -g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] + g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. .. code-block:: python -from stix2 import Filter + from stix2 import Filter -t1134 = src.query([ - Filter("external_references.external_id", "=", "T1134"), - Filter("type", "=", "attack-pattern") -])[0] + t1134 = src.query([ + Filter("external_references.external_id", "=", "T1134"), + Filter("type", "=", "attack-pattern") + ])[0] The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). diff --git a/docs/stix_overview/by_alias.rst b/docs/stix_overview/by_alias.rst index 049bb76a..c2abf03a 100644 --- a/docs/stix_overview/by_alias.rst +++ b/docs/stix_overview/by_alias.rst @@ -5,12 +5,12 @@ By Alias The following methodology can be used to find the group corresponding to a given alias: .. code-block:: python -from stix2 import Filter + from stix2 import Filter -def get_group_by_alias(thesrc, alias): - return thesrc.query([ - Filter('type', '=', 'intrusion-set'), - Filter('aliases', '=', alias) - ])[0] - -get_group_by_alias(src, 'Cozy Bear') + def get_group_by_alias(thesrc, alias): + return thesrc.query([ + Filter('type', '=', 'intrusion-set'), + Filter('aliases', '=', alias) + ])[0] + + get_group_by_alias(src, 'Cozy Bear') diff --git a/docs/stix_overview/by_name.rst b/docs/stix_overview/by_name.rst index 2be1ac8a..8a3b793c 100644 --- a/docs/stix_overview/by_name.rst +++ b/docs/stix_overview/by_name.rst @@ -7,13 +7,13 @@ By Name The following recipe retrieves an object according to its name: .. code-block:: python -from stix2 import Filter + from stix2 import Filter -def get_technique_by_name(thesrc, name): - filt = [ - Filter('type', '=', 'attack-pattern'), - Filter('name', '=', name) - ] - return thesrc.query(filt) -# get the technique titled "System Information Discovery" -get_technique_by_name(src, 'System Information Discovery') + def get_technique_by_name(thesrc, name): + filt = [ + Filter('type', '=', 'attack-pattern'), + Filter('name', '=', name) + ] + return thesrc.query(filt) + # get the technique titled "System Information Discovery" + get_technique_by_name(src, 'System Information Discovery') diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/getting-started.rst index 3cd87942..168f7eaa 100644 --- a/docs/stix_overview/getting-started.rst +++ b/docs/stix_overview/getting-started.rst @@ -24,13 +24,13 @@ Before installing requirements, we recommend setting up a virtual environment: [stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: .. code-block:: python -from stix2 import Filter + from stix2 import Filter However, if you are aiming to extend the ATT&CK dataset with new objects or implement complex workflows, you may need to use the `v20` specifier for some imports. This ensures that the objects use the STIX 2.0 API instead of the STIX 2.1 API. For example: .. code-block:: python -from stix2.v20 import AttackPattern + from stix2.v20 import AttackPattern You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). @@ -42,7 +42,7 @@ You can see a full list of the classes which have versioned imports [here](https If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. .. code-block:: python -from taxii2client.v20 import Collection + from taxii2client.v20 import Collection ### Access local content @@ -59,7 +59,7 @@ Each domain in this repo is formatted according to the [STIX2 FileSystem spec](h Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: .. code-block:: python -from stix2 import FileSystemSource + from stix2 import FileSystemSource src = FileSystemSource('./cti/enterprise-attack') @@ -69,10 +69,10 @@ src = FileSystemSource('./cti/enterprise-attack') If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: .. code-block:: python -from stix2 import MemoryStore + from stix2 import MemoryStore -src = MemoryStore() -src.load_from_file("enterprise-attack.json") + src = MemoryStore() + src.load_from_file("enterprise-attack.json") ### Access live content @@ -95,29 +95,29 @@ Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII You can also get a list of available collection from the server directly: .. code-block:: python -from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 + from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 -server = Server("https://cti-taxii.mitre.org/taxii/") -api_root = server.api_roots[0] -# Print name and ID of all ATT&CK domains available as collections -for collection in api_root.collections: - print(collection.title.ljust(20) + collection.id) + server = Server("https://cti-taxii.mitre.org/taxii/") + api_root = server.api_roots[0] + # Print name and ID of all ATT&CK domains available as collections + for collection in api_root.collections: + print(collection.title.ljust(20) + collection.id) The following recipe demonstrates how to access the enterprise-attack data from the TAXII server. .. code-block:: python -from stix2 import TAXIICollectionSource -from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 + from stix2 import TAXIICollectionSource + from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 -collections = { - "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e", - "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b", - "ics-attack": "02c3ef24-9cd4-48f3-a99f-b74ce24f1d34" -} + collections = { + "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e", + "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b", + "ics-attack": "02c3ef24-9cd4-48f3-a99f-b74ce24f1d34" + } -collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/") -src = TAXIICollectionSource(collection) + collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/") + src = TAXIICollectionSource(collection) For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). @@ -129,15 +129,15 @@ While typically the TAXII method is more desirable for "live" access, this metho access data on a branch of the MITRE/CTI repo (the TAXII server only holds the master branch) or in the case of a TAXII server outage. .. code-block:: python -import requests -from stix2 import MemoryStore + import requests + from stix2 import MemoryStore -def get_data_from_branch(domain, branch="master"): - """get the ATT&CK STIX data from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'. Branch should typically be master.""" - stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/{branch}/{domain}/{domain}.json").json() - return MemoryStore(stix_data=stix_json["objects"]) + def get_data_from_branch(domain, branch="master"): + """get the ATT&CK STIX data from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'. Branch should typically be master.""" + stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/{branch}/{domain}/{domain}.json").json() + return MemoryStore(stix_data=stix_json["objects"]) -src = get_data_from_branch("enterprise-attack") + src = get_data_from_branch("enterprise-attack") ### Access a specific version of ATT&CK @@ -147,27 +147,27 @@ ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.co In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: .. code-block:: python -import requests -from stix2 import MemoryStore + import requests + from stix2 import MemoryStore -def get_data_from_version(domain, version): - """get the ATT&CK STIX data for the given version from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'.""" - stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v{version}/{domain}/{domain}.json").json() - return MemoryStore(stix_data=stix_json["objects"]) + def get_data_from_version(domain, version): + """get the ATT&CK STIX data for the given version from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'.""" + stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v{version}/{domain}/{domain}.json").json() + return MemoryStore(stix_data=stix_json["objects"]) -src = get_data_from_version("enterprise-attack", "5.2") + src = get_data_from_version("enterprise-attack", "5.2") You can get a list of ATT&CK versions programmatically using the github API: .. code-block:: python -import requests -import re + import requests + import re -refToTag = re.compile(r"ATT&CK-v(.*)") -tags = requests.get("https://api.github.com/repos/mitre/cti/git/refs/tags").json() -versions = list(map(lambda tag: refToTag.search(tag["ref"]).groups()[0] , filter(lambda tag: "ATT&CK-v" in tag["ref"], tags))) -# versions = ["1.0", "2.0", ...] + refToTag = re.compile(r"ATT&CK-v(.*)") + tags = requests.get("https://api.github.com/repos/mitre/cti/git/refs/tags").json() + versions = list(map(lambda tag: refToTag.search(tag["ref"]).groups()[0] , filter(lambda tag: "ATT&CK-v" in tag["ref"], tags))) + # versions = ["1.0", "2.0", ...] ### Access multiple domains simultaneously @@ -178,10 +178,10 @@ domains into a single DataStore. Use any of the methods above to acquire the ind a single CompositeDataSource: .. code-block:: python -from stix2 import CompositeDataSource + from stix2 import CompositeDataSource -src = CompositeDataSource() -src.add_data_sources([enterprise_attack_src, mobile_attack_src, ics_attack_src]) + src = CompositeDataSource() + src.add_data_sources([enterprise_attack_src, mobile_attack_src, ics_attack_src]) You can then use this CompositeDataSource just as you would the DataSource for an individual domain. diff --git a/docs/stix_overview/getting_revoked_object.rst b/docs/stix_overview/getting_revoked_object.rst index d45ece23..71e60b69 100644 --- a/docs/stix_overview/getting_revoked_object.rst +++ b/docs/stix_overview/getting_revoked_object.rst @@ -6,17 +6,17 @@ Getting a revoking object When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: .. code-block:: python -from stix2 import Filter + from stix2 import Filter -def getRevokedBy(stix_id, thesrc): - relations = thesrc.relationships(stix_id, 'revoked-by', source_only=True) - revoked_by = thesrc.query([ - Filter('id', 'in', [r.target_ref for r in relations]), - Filter('revoked', '=', False) - ]) - if revoked_by is not None: - revoked_by = revoked_by[0] + def getRevokedBy(stix_id, thesrc): + relations = thesrc.relationships(stix_id, 'revoked-by', source_only=True) + revoked_by = thesrc.query([ + Filter('id', 'in', [r.target_ref for r in relations]), + Filter('revoked', '=', False) + ]) + if revoked_by is not None: + revoked_by = revoked_by[0] - return revoked_by + return revoked_by -getRevokedBy("attack-pattern--c16e5409-ee53-4d79-afdc-4099dc9292df", src) + getRevokedBy("attack-pattern--c16e5409-ee53-4d79-afdc-4099dc9292df", src) diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_overview/getting_software.rst index 607c4372..932fc3b5 100644 --- a/docs/stix_overview/getting_software.rst +++ b/docs/stix_overview/getting_software.rst @@ -7,14 +7,14 @@ Because software are the union of two STIX types (`tool` and `malware`), the pro .. code-block:: python from itertools import chain -from stix2 import Filter + from stix2 import Filter -def get_software(thesrc): - return list(chain.from_iterable( - thesrc.query(f) for f in [ - Filter("type", "=", "tool"), - Filter("type", "=", "malware") - ] - )) + def get_software(thesrc): + return list(chain.from_iterable( + thesrc.query(f) for f in [ + Filter("type", "=", "tool"), + Filter("type", "=", "malware") + ] + )) -get_software(src) + get_software(src) diff --git a/docs/stix_overview/objects_by_content.rst b/docs/stix_overview/objects_by_content.rst index d47c1f0a..ace27a91 100644 --- a/docs/stix_overview/objects_by_content.rst +++ b/docs/stix_overview/objects_by_content.rst @@ -6,11 +6,11 @@ Objects by content Sometimes it may be useful to query objects by the content of their description: .. code-block:: python -from stix2 import Filter + from stix2 import Filter -def get_techniques_by_content(thesrc, content): - techniques = src.query([ Filter('type', '=', 'attack-pattern') ]) - return list(filter(lambda t: content.lower() in t.description.lower(), techniques)) + def get_techniques_by_content(thesrc, content): + techniques = src.query([ Filter('type', '=', 'attack-pattern') ]) + return list(filter(lambda t: content.lower() in t.description.lower(), techniques)) -# Get all techniques where the string LSASS appears in the description -get_techniques_by_content(src, 'LSASS') + # Get all techniques where the string LSASS appears in the description + get_techniques_by_content(src, 'LSASS') diff --git a/docs/stix_overview/objects_by_type.rst b/docs/stix_overview/objects_by_type.rst index 92360db2..efed2f38 100644 --- a/docs/stix_overview/objects_by_type.rst +++ b/docs/stix_overview/objects_by_type.rst @@ -6,7 +6,7 @@ Objects by type See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. .. code-block:: python -from stix2 import Filter + from stix2 import Filter -# use the appropriate STIX type in the query according to the desired ATT&CK type -groups = src.query([ Filter("type", "=", "intrusion-set") ]) + # use the appropriate STIX type in the query according to the desired ATT&CK type + groups = src.query([ Filter("type", "=", "intrusion-set") ]) diff --git a/docs/stix_overview/objects_since_date.rst b/docs/stix_overview/objects_since_date.rst index 7418bc2f..df2a23c4 100644 --- a/docs/stix_overview/objects_since_date.rst +++ b/docs/stix_overview/objects_since_date.rst @@ -5,24 +5,24 @@ Objects created or modified since a given date Sometimes you may want to get a list of objects which have been created or modified after a certain time. .. code-block:: python -from stix2 import Filter + from stix2 import Filter -def get_created_after(thesrc, timestamp): - filt = [ - Filter('created', '>', timestamp) - ] - return thesrc.query(filt) + def get_created_after(thesrc, timestamp): + filt = [ + Filter('created', '>', timestamp) + ] + return thesrc.query(filt) -get_created_after(src, "2018-10-01T00:14:20.652Z") + get_created_after(src, "2018-10-01T00:14:20.652Z") -def get_modified_after(thesrc, timestamp): - filt = [ - Filter('modified', '>', timestamp) - ] - return thesrc.query(filt) - -get_modified_after(src, "2018-10-01T00:14:20.652Z") + def get_modified_after(thesrc, timestamp): + filt = [ + Filter('modified', '>', timestamp) + ] + return thesrc.query(filt) + + get_modified_after(src, "2018-10-01T00:14:20.652Z") We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to [check the list of released versions of ATT&CK](https://github.com/mitre/cti/blob/master/USAGE.md#access-a-specific-version-of-attck). diff --git a/docs/stix_overview/relationships_microlibrary.rst b/docs/stix_overview/relationships_microlibrary.rst index 055c8786..bf296cb6 100644 --- a/docs/stix_overview/relationships_microlibrary.rst +++ b/docs/stix_overview/relationships_microlibrary.rst @@ -9,311 +9,310 @@ The examples are backwards-compatible for previous versions af ATT&CK that omit This microlibrary can be used to build a lookup table of stixID to related objects and relationships. The argument to each accessor function is a STIX2 MemoryStore to build the relationship mappings from. -```python -from pprint import pprint -from stix2 import MemoryStore, Filter - -# See section below on "Removing revoked and deprecated objects" -def remove_revoked_deprecated(stix_objects): - """Remove any revoked or deprecated objects from queries made to the data source""" - # Note we use .get() because the property may not be present in the JSON data. The default is False - # if the property is not set. - return list( - filter( - lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, - stix_objects - ) - ) - -def get_related(thesrc, src_type, rel_type, target_type, reverse=False): - """build relationship mappings - params: - thesrc: MemoryStore to build relationship lookups for - src_type: source type for the relationships, e.g "attack-pattern" - rel_type: relationship type for the relationships, e.g "uses" - target_type: target type for the relationship, e.g "intrusion-set" - reverse: build reverse mapping of target to source - """ - - relationships = thesrc.query([ - Filter('type', '=', 'relationship'), - Filter('relationship_type', '=', rel_type), - Filter('revoked', '=', False), - ]) +.. code-block:: python + from pprint import pprint + from stix2 import MemoryStore, Filter # See section below on "Removing revoked and deprecated objects" - relationships = remove_revoked_deprecated(relationships) - - # stix_id => [ { relationship, related_object_id } for each related object ] - id_to_related = {} - - # build the dict - for relationship in relationships: - if src_type in relationship.source_ref and target_type in relationship.target_ref: - if (relationship.source_ref in id_to_related and not reverse) or (relationship.target_ref in id_to_related and reverse): - # append to existing entry - if not reverse: - id_to_related[relationship.source_ref].append({ - "relationship": relationship, - "id": relationship.target_ref - }) - else: - id_to_related[relationship.target_ref].append({ - "relationship": relationship, - "id": relationship.source_ref - }) - else: - # create a new entry - if not reverse: - id_to_related[relationship.source_ref] = [{ - "relationship": relationship, - "id": relationship.target_ref - }] - else: - id_to_related[relationship.target_ref] = [{ - "relationship": relationship, - "id": relationship.source_ref - }] - # all objects of relevant type - if not reverse: - targets = thesrc.query([ - Filter('type', '=', target_type), - Filter('revoked', '=', False) - ]) - else: - targets = thesrc.query([ - Filter('type', '=', src_type), - Filter('revoked', '=', False) + def remove_revoked_deprecated(stix_objects): + """Remove any revoked or deprecated objects from queries made to the data source""" + # Note we use .get() because the property may not be present in the JSON data. The default is False + # if the property is not set. + return list( + filter( + lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, + stix_objects + ) + ) + + def get_related(thesrc, src_type, rel_type, target_type, reverse=False): + """build relationship mappings + params: + thesrc: MemoryStore to build relationship lookups for + src_type: source type for the relationships, e.g "attack-pattern" + rel_type: relationship type for the relationships, e.g "uses" + target_type: target type for the relationship, e.g "intrusion-set" + reverse: build reverse mapping of target to source + """ + + relationships = thesrc.query([ + Filter('type', '=', 'relationship'), + Filter('relationship_type', '=', rel_type), + Filter('revoked', '=', False), ]) - # build lookup of stixID to stix object - id_to_target = {} - for target in targets: - id_to_target[target.id] = target - - # build final output mappings - output = {} - for stix_id in id_to_related: - value = [] - for related in id_to_related[stix_id]: - if not related["id"] in id_to_target: - continue # targeting a revoked object - value.append({ - "object": id_to_target[related["id"]], - "relationship": related["relationship"] - }) - output[stix_id] = value - return output - -# software:group -def software_used_by_groups(thesrc): - """returns group_id => {software, relationship} for each software used by the group and each software used by campaigns attributed to the group.""" - # get all software used by groups - tools_used_by_group = get_related(thesrc, "intrusion-set", "uses", "tool") - malware_used_by_group = get_related(thesrc, "intrusion-set", "uses", "malware") - software_used_by_group = {**tools_used_by_group, **malware_used_by_group} # group_id -> [{software, relationship}] - - # get groups attributing to campaigns and all software used by campaigns - software_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") - malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") - for id in malware_used_by_campaign: - if id in software_used_by_campaign: - software_used_by_campaign[id].extend(malware_used_by_campaign[id]) - else: - software_used_by_campaign[id] = malware_used_by_campaign[id] - campaigns_attributed_to_group = { - "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} - "software": software_used_by_campaign # campaign_id => {software, relationship} - } - - for group_id in campaigns_attributed_to_group["campaigns"]: - software_used_by_campaigns = [] - # check if attributed campaign is using software - for campaign in campaigns_attributed_to_group["campaigns"][group_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in campaigns_attributed_to_group["software"]: - software_used_by_campaigns.extend(campaigns_attributed_to_group["software"][campaign_id]) - - # update software used by group to include software used by a groups attributed campaign - if group_id in software_used_by_group: - software_used_by_group[group_id].extend(software_used_by_campaigns) - else: - software_used_by_group[group_id] = software_used_by_campaigns - return software_used_by_group - -def groups_using_software(thesrc): - """returns software_id => {group, relationship} for each group using the software and each software used by attributed campaigns.""" - # get all groups using software - groups_using_tool = get_related(thesrc, "intrusion-set", "uses", "tool", reverse=True) - groups_using_malware = get_related(thesrc, "intrusion-set", "uses", "malware", reverse=True) - groups_using_software = {**groups_using_tool, **groups_using_malware} # software_id => {group, relationship} - - # get campaigns attributed to groups and all campaigns using software - campaigns_using_software = get_related(thesrc, "campaign", "uses", "tool", reverse=True) - campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) - for id in campaigns_using_malware: - if id in campaigns_using_software: - campaigns_using_software[id].extend(campaigns_using_malware[id]) - else: - campaigns_using_software[id] = campaigns_using_malware[id] - groups_attributing_to_campaigns = { - "campaigns": campaigns_using_software,# software_id => {campaign, relationship} - "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} - } - - for software_id in groups_attributing_to_campaigns["campaigns"]: - groups_attributed_to_campaigns = [] - # check if campaign is attributed to group - for campaign in groups_attributing_to_campaigns["campaigns"][software_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in groups_attributing_to_campaigns["groups"]: - groups_attributed_to_campaigns.extend(groups_attributing_to_campaigns["groups"][campaign_id]) - - # update groups using software to include software used by a groups attributed campaign - if software_id in groups_using_software: - groups_using_software[software_id].extend(groups_attributed_to_campaigns) - else: - groups_using_software[software_id] = groups_attributed_to_campaigns - return groups_using_software - -# software:campaign -def software_used_by_campaigns(thesrc): - """returns campaign_id => {software, relationship} for each software used by the campaign.""" - tools_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") - malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") - return {**tools_used_by_campaign, **malware_used_by_campaign} - -def campaigns_using_software(thesrc): - """returns software_id => {campaign, relationship} for each campaign using the software.""" - campaigns_using_tool = get_related(thesrc, "campaign", "uses", "tool", reverse=True) - campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) - return {**campaigns_using_tool, **campaigns_using_malware} - -# campaign:group -def groups_attributing_to_campaign(thesrc): - """returns campaign_id => {group, relationship} for each group attributing to the campaign.""" - return get_related(thesrc, "campaign", "attributed-to", "intrusion-set") - -def campaigns_attributed_to_group(thesrc): - """returns group_id => {campaign, relationship} for each campaign attributed to the group.""" - return get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True) - -# technique:group -def techniques_used_by_groups(thesrc): - """returns group_id => {technique, relationship} for each technique used by the group and each - technique used by campaigns attributed to the group.""" - # get all techniques used by groups - techniques_used_by_groups = get_related(thesrc, "intrusion-set", "uses", "attack-pattern") # group_id => {technique, relationship} - - # get groups attributing to campaigns and all techniques used by campaigns - campaigns_attributed_to_group = { - "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} - "techniques": get_related(thesrc, "campaign", "uses", "attack-pattern") # campaign_id => {technique, relationship} - } - - for group_id in campaigns_attributed_to_group["campaigns"]: - techniques_used_by_campaigns = [] - # check if attributed campaign is using technique - for campaign in campaigns_attributed_to_group["campaigns"][group_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in campaigns_attributed_to_group["techniques"]: - techniques_used_by_campaigns.extend(campaigns_attributed_to_group["techniques"][campaign_id]) - - # update techniques used by groups to include techniques used by a groups attributed campaign - if group_id in techniques_used_by_groups: - techniques_used_by_groups[group_id].extend(techniques_used_by_campaigns) - else: - techniques_used_by_groups[group_id] = techniques_used_by_campaigns - return techniques_used_by_groups - -def groups_using_technique(thesrc): - """returns technique_id => {group, relationship} for each group using the technique and each campaign attributed to groups using the technique.""" - # get all groups using techniques - groups_using_techniques = get_related(thesrc, "intrusion-set", "uses", "attack-pattern", reverse=True) # technique_id => {group, relationship} - - # get campaigns attributed to groups and all campaigns using techniques - groups_attributing_to_campaigns = { - "campaigns": get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True), # technique_id => {campaign, relationship} - "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} - } - - for technique_id in groups_attributing_to_campaigns["campaigns"]: - campaigns_attributed_to_group = [] - # check if campaign is attributed to group - for campaign in groups_attributing_to_campaigns["campaigns"][technique_id]: - campaign_id = campaign["object"]["id"] - if campaign_id in groups_attributing_to_campaigns["groups"]: - campaigns_attributed_to_group.extend(groups_attributing_to_campaigns["groups"][campaign_id]) - - # update groups using techniques to include techniques used by a groups attributed campaign - if technique_id in groups_using_techniques: - groups_using_techniques[technique_id].extend(campaigns_attributed_to_group) + # See section below on "Removing revoked and deprecated objects" + relationships = remove_revoked_deprecated(relationships) + + # stix_id => [ { relationship, related_object_id } for each related object ] + id_to_related = {} + + # build the dict + for relationship in relationships: + if src_type in relationship.source_ref and target_type in relationship.target_ref: + if (relationship.source_ref in id_to_related and not reverse) or (relationship.target_ref in id_to_related and reverse): + # append to existing entry + if not reverse: + id_to_related[relationship.source_ref].append({ + "relationship": relationship, + "id": relationship.target_ref + }) + else: + id_to_related[relationship.target_ref].append({ + "relationship": relationship, + "id": relationship.source_ref + }) + else: + # create a new entry + if not reverse: + id_to_related[relationship.source_ref] = [{ + "relationship": relationship, + "id": relationship.target_ref + }] + else: + id_to_related[relationship.target_ref] = [{ + "relationship": relationship, + "id": relationship.source_ref + }] + # all objects of relevant type + if not reverse: + targets = thesrc.query([ + Filter('type', '=', target_type), + Filter('revoked', '=', False) + ]) else: - groups_using_techniques[technique_id] = campaigns_attributed_to_group - return groups_using_techniques - -# technique:campaign -def techniques_used_by_campaigns(thesrc): - """returns campaign_id => {technique, relationship} for each technique used by the campaign.""" - return get_related(thesrc, "campaign", "uses", "attack-pattern") - -def campaigns_using_technique(thesrc): - """returns technique_id => {campaign, relationship} for each campaign using the technique.""" - return get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True) - -# technique:software -def techniques_used_by_software(thesrc): - """return software_id => {technique, relationship} for each technique used by the software.""" - techniques_by_tool = get_related(thesrc, "tool", "uses", "attack-pattern") - techniques_by_malware = get_related(thesrc, "malware", "uses", "attack-pattern") - return {**techniques_by_tool, **techniques_by_malware} - -def software_using_technique(thesrc): - """return technique_id => {software, relationship} for each software using the technique.""" - tools_by_technique_id = get_related(thesrc, "tool", "uses", "attack-pattern", reverse=True) - malware_by_technique_id = get_related(thesrc, "malware", "uses", "attack-pattern", reverse=True) - return {**tools_by_technique_id, **malware_by_technique_id} - -# technique:mitigation -def mitigation_mitigates_techniques(thesrc): - """return mitigation_id => {technique, relationship} for each technique mitigated by the mitigation.""" - return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=False) - -def technique_mitigated_by_mitigations(thesrc): - """return technique_id => {mitigation, relationship} for each mitigation of the technique.""" - return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=True) - -# technique:sub-technique -def subtechniques_of(thesrc): - """return technique_id => {subtechnique, relationship} for each subtechnique of the technique.""" - return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern", reverse=True) - -def parent_technique_of(thesrc): - """return subtechnique_id => {technique, relationship} describing the parent technique of the subtechnique""" - return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern")[0] - -# technique:data-component -def datacomponent_detects_techniques(thesrc): - """return datacomponent_id => {technique, relationship} describing the detections of each data component""" - return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern") - -def technique_detected_by_datacomponents(thesrc): - """return technique_id => {datacomponent, relationship} describing the data components that can detect the technique""" - return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern", reverse=True) - -# Example usage: -src = MemoryStore() -src.load_from_file("path/to/enterprise-attack.json") - -group_id_to_software = software_used_by_groups(src) -pprint(group_id_to_software["intrusion-set--2a158b0a-7ef8-43cb-9985-bf34d1e12050"]) # G0019 -# [ -# { -# "object": Malware, # S0061 -# "relationship": Relationship # relationship between G0019 and S0061 -# }, -# { -# ... -# } -# ] -``` \ No newline at end of file + targets = thesrc.query([ + Filter('type', '=', src_type), + Filter('revoked', '=', False) + ]) + + # build lookup of stixID to stix object + id_to_target = {} + for target in targets: + id_to_target[target.id] = target + + # build final output mappings + output = {} + for stix_id in id_to_related: + value = [] + for related in id_to_related[stix_id]: + if not related["id"] in id_to_target: + continue # targeting a revoked object + value.append({ + "object": id_to_target[related["id"]], + "relationship": related["relationship"] + }) + output[stix_id] = value + return output + + # software:group + def software_used_by_groups(thesrc): + """returns group_id => {software, relationship} for each software used by the group and each software used by campaigns attributed to the group.""" + # get all software used by groups + tools_used_by_group = get_related(thesrc, "intrusion-set", "uses", "tool") + malware_used_by_group = get_related(thesrc, "intrusion-set", "uses", "malware") + software_used_by_group = {**tools_used_by_group, **malware_used_by_group} # group_id -> [{software, relationship}] + + # get groups attributing to campaigns and all software used by campaigns + software_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") + malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") + for id in malware_used_by_campaign: + if id in software_used_by_campaign: + software_used_by_campaign[id].extend(malware_used_by_campaign[id]) + else: + software_used_by_campaign[id] = malware_used_by_campaign[id] + campaigns_attributed_to_group = { + "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} + "software": software_used_by_campaign # campaign_id => {software, relationship} + } + + for group_id in campaigns_attributed_to_group["campaigns"]: + software_used_by_campaigns = [] + # check if attributed campaign is using software + for campaign in campaigns_attributed_to_group["campaigns"][group_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in campaigns_attributed_to_group["software"]: + software_used_by_campaigns.extend(campaigns_attributed_to_group["software"][campaign_id]) + + # update software used by group to include software used by a groups attributed campaign + if group_id in software_used_by_group: + software_used_by_group[group_id].extend(software_used_by_campaigns) + else: + software_used_by_group[group_id] = software_used_by_campaigns + return software_used_by_group + + def groups_using_software(thesrc): + """returns software_id => {group, relationship} for each group using the software and each software used by attributed campaigns.""" + # get all groups using software + groups_using_tool = get_related(thesrc, "intrusion-set", "uses", "tool", reverse=True) + groups_using_malware = get_related(thesrc, "intrusion-set", "uses", "malware", reverse=True) + groups_using_software = {**groups_using_tool, **groups_using_malware} # software_id => {group, relationship} + + # get campaigns attributed to groups and all campaigns using software + campaigns_using_software = get_related(thesrc, "campaign", "uses", "tool", reverse=True) + campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) + for id in campaigns_using_malware: + if id in campaigns_using_software: + campaigns_using_software[id].extend(campaigns_using_malware[id]) + else: + campaigns_using_software[id] = campaigns_using_malware[id] + groups_attributing_to_campaigns = { + "campaigns": campaigns_using_software,# software_id => {campaign, relationship} + "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} + } + + for software_id in groups_attributing_to_campaigns["campaigns"]: + groups_attributed_to_campaigns = [] + # check if campaign is attributed to group + for campaign in groups_attributing_to_campaigns["campaigns"][software_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in groups_attributing_to_campaigns["groups"]: + groups_attributed_to_campaigns.extend(groups_attributing_to_campaigns["groups"][campaign_id]) + + # update groups using software to include software used by a groups attributed campaign + if software_id in groups_using_software: + groups_using_software[software_id].extend(groups_attributed_to_campaigns) + else: + groups_using_software[software_id] = groups_attributed_to_campaigns + return groups_using_software + + # software:campaign + def software_used_by_campaigns(thesrc): + """returns campaign_id => {software, relationship} for each software used by the campaign.""" + tools_used_by_campaign = get_related(thesrc, "campaign", "uses", "tool") + malware_used_by_campaign = get_related(thesrc, "campaign", "uses", "malware") + return {**tools_used_by_campaign, **malware_used_by_campaign} + + def campaigns_using_software(thesrc): + """returns software_id => {campaign, relationship} for each campaign using the software.""" + campaigns_using_tool = get_related(thesrc, "campaign", "uses", "tool", reverse=True) + campaigns_using_malware = get_related(thesrc, "campaign", "uses", "malware", reverse=True) + return {**campaigns_using_tool, **campaigns_using_malware} + + # campaign:group + def groups_attributing_to_campaign(thesrc): + """returns campaign_id => {group, relationship} for each group attributing to the campaign.""" + return get_related(thesrc, "campaign", "attributed-to", "intrusion-set") + + def campaigns_attributed_to_group(thesrc): + """returns group_id => {campaign, relationship} for each campaign attributed to the group.""" + return get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True) + + # technique:group + def techniques_used_by_groups(thesrc): + """returns group_id => {technique, relationship} for each technique used by the group and each + technique used by campaigns attributed to the group.""" + # get all techniques used by groups + techniques_used_by_groups = get_related(thesrc, "intrusion-set", "uses", "attack-pattern") # group_id => {technique, relationship} + + # get groups attributing to campaigns and all techniques used by campaigns + campaigns_attributed_to_group = { + "campaigns": get_related(thesrc, "campaign", "attributed-to", "intrusion-set", reverse=True), # group_id => {campaign, relationship} + "techniques": get_related(thesrc, "campaign", "uses", "attack-pattern") # campaign_id => {technique, relationship} + } + + for group_id in campaigns_attributed_to_group["campaigns"]: + techniques_used_by_campaigns = [] + # check if attributed campaign is using technique + for campaign in campaigns_attributed_to_group["campaigns"][group_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in campaigns_attributed_to_group["techniques"]: + techniques_used_by_campaigns.extend(campaigns_attributed_to_group["techniques"][campaign_id]) + + # update techniques used by groups to include techniques used by a groups attributed campaign + if group_id in techniques_used_by_groups: + techniques_used_by_groups[group_id].extend(techniques_used_by_campaigns) + else: + techniques_used_by_groups[group_id] = techniques_used_by_campaigns + return techniques_used_by_groups + + def groups_using_technique(thesrc): + """returns technique_id => {group, relationship} for each group using the technique and each campaign attributed to groups using the technique.""" + # get all groups using techniques + groups_using_techniques = get_related(thesrc, "intrusion-set", "uses", "attack-pattern", reverse=True) # technique_id => {group, relationship} + + # get campaigns attributed to groups and all campaigns using techniques + groups_attributing_to_campaigns = { + "campaigns": get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True), # technique_id => {campaign, relationship} + "groups": get_related(thesrc, "campaign", "attributed-to", "intrusion-set") # campaign_id => {group, relationship} + } + + for technique_id in groups_attributing_to_campaigns["campaigns"]: + campaigns_attributed_to_group = [] + # check if campaign is attributed to group + for campaign in groups_attributing_to_campaigns["campaigns"][technique_id]: + campaign_id = campaign["object"]["id"] + if campaign_id in groups_attributing_to_campaigns["groups"]: + campaigns_attributed_to_group.extend(groups_attributing_to_campaigns["groups"][campaign_id]) + + # update groups using techniques to include techniques used by a groups attributed campaign + if technique_id in groups_using_techniques: + groups_using_techniques[technique_id].extend(campaigns_attributed_to_group) + else: + groups_using_techniques[technique_id] = campaigns_attributed_to_group + return groups_using_techniques + + # technique:campaign + def techniques_used_by_campaigns(thesrc): + """returns campaign_id => {technique, relationship} for each technique used by the campaign.""" + return get_related(thesrc, "campaign", "uses", "attack-pattern") + + def campaigns_using_technique(thesrc): + """returns technique_id => {campaign, relationship} for each campaign using the technique.""" + return get_related(thesrc, "campaign", "uses", "attack-pattern", reverse=True) + + # technique:software + def techniques_used_by_software(thesrc): + """return software_id => {technique, relationship} for each technique used by the software.""" + techniques_by_tool = get_related(thesrc, "tool", "uses", "attack-pattern") + techniques_by_malware = get_related(thesrc, "malware", "uses", "attack-pattern") + return {**techniques_by_tool, **techniques_by_malware} + + def software_using_technique(thesrc): + """return technique_id => {software, relationship} for each software using the technique.""" + tools_by_technique_id = get_related(thesrc, "tool", "uses", "attack-pattern", reverse=True) + malware_by_technique_id = get_related(thesrc, "malware", "uses", "attack-pattern", reverse=True) + return {**tools_by_technique_id, **malware_by_technique_id} + + # technique:mitigation + def mitigation_mitigates_techniques(thesrc): + """return mitigation_id => {technique, relationship} for each technique mitigated by the mitigation.""" + return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=False) + + def technique_mitigated_by_mitigations(thesrc): + """return technique_id => {mitigation, relationship} for each mitigation of the technique.""" + return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=True) + + # technique:sub-technique + def subtechniques_of(thesrc): + """return technique_id => {subtechnique, relationship} for each subtechnique of the technique.""" + return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern", reverse=True) + + def parent_technique_of(thesrc): + """return subtechnique_id => {technique, relationship} describing the parent technique of the subtechnique""" + return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern")[0] + + # technique:data-component + def datacomponent_detects_techniques(thesrc): + """return datacomponent_id => {technique, relationship} describing the detections of each data component""" + return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern") + + def technique_detected_by_datacomponents(thesrc): + """return technique_id => {datacomponent, relationship} describing the data components that can detect the technique""" + return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern", reverse=True) + + # Example usage: + src = MemoryStore() + src.load_from_file("path/to/enterprise-attack.json") + + group_id_to_software = software_used_by_groups(src) + pprint(group_id_to_software["intrusion-set--2a158b0a-7ef8-43cb-9985-bf34d1e12050"]) # G0019 + # [ + # { + # "object": Malware, # S0061 + # "relationship": Relationship # relationship between G0019 and S0061 + # }, + # { + # ... + # } + # ] diff --git a/docs/stix_overview/remove_revoked_deprecated.rst b/docs/stix_overview/remove_revoked_deprecated.rst index 264c60ce..5f3b6b9b 100644 --- a/docs/stix_overview/remove_revoked_deprecated.rst +++ b/docs/stix_overview/remove_revoked_deprecated.rst @@ -9,20 +9,19 @@ longer maintained by ATT&CK. We recommend _not_ using built-in STIX filters for removing revoked objects (e.g `Filter('revoked', '=', False)`). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See [issue #127](https://github.com/mitre/cti/issues/127) for more details. -```python -from stix2 import Filter +.. code-block:: python + from stix2 import Filter -def remove_revoked_deprecated(stix_objects): - """Remove any revoked or deprecated objects from queries made to the data source""" - # Note we use .get() because the property may not be present in the JSON data. The default is False - # if the property is not set. - return list( - filter( - lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, - stix_objects + def remove_revoked_deprecated(stix_objects): + """Remove any revoked or deprecated objects from queries made to the data source""" + # Note we use .get() because the property may not be present in the JSON data. The default is False + # if the property is not set. + return list( + filter( + lambda x: x.get("x_mitre_deprecated", False) is False and x.get("revoked", False) is False, + stix_objects + ) ) - ) -mitigations = src.query([ Filter("type", "=", "course-of-action") ]) -mitigations = remove_revoked_deprecated(mitigations) -``` \ No newline at end of file + mitigations = src.query([ Filter("type", "=", "course-of-action") ]) + mitigations = remove_revoked_deprecated(mitigations) diff --git a/docs/stix_overview/stix_id.rst b/docs/stix_overview/stix_id.rst index 716e7dcc..030f65b1 100644 --- a/docs/stix_overview/stix_id.rst +++ b/docs/stix_overview/stix_id.rst @@ -4,6 +4,5 @@ By STIX ID The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. -```python -g0075 = src.get("intrusion-set--f40eb8ce-2a74-4e56-89a1-227021410142") -``` \ No newline at end of file +.. code-block:: python + g0075 = src.get("intrusion-set--f40eb8ce-2a74-4e56-89a1-227021410142") diff --git a/docs/stix_overview/tactics_by_matrix.rst b/docs/stix_overview/tactics_by_matrix.rst index 84e68ba5..8f10fa8f 100644 --- a/docs/stix_overview/tactics_by_matrix.rst +++ b/docs/stix_overview/tactics_by_matrix.rst @@ -6,22 +6,21 @@ The tactics are individual objects (`x-mitre-tactic`), and their order in a matr found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches the ordering of the tactics in that matrix. The following recipe returns a structured list of tactics within each matrix of the input DataStore. -```python -from stix2 import Filter +.. code-block:: python + from stix2 import Filter -def getTacticsByMatrix(thesrc): - tactics = {} - matrix = thesrc.query([ - Filter('type', '=', 'x-mitre-matrix'), - ]) - - for i in range(len(matrix)): - tactics[matrix[i]['name']] = [] - for tactic_id in matrix[i]['tactic_refs']: - tactics[matrix[i]['name']].append(thesrc.get(tactic_id)) - - return tactics + def getTacticsByMatrix(thesrc): + tactics = {} + matrix = thesrc.query([ + Filter('type', '=', 'x-mitre-matrix'), + ]) + + for i in range(len(matrix)): + tactics[matrix[i]['name']] = [] + for tactic_id in matrix[i]['tactic_refs']: + tactics[matrix[i]['name']].append(thesrc.get(tactic_id)) + + return tactics -# get tactic layout -getTacticsByMatrix(src) -``` \ No newline at end of file + # get tactic layout + getTacticsByMatrix(src) diff --git a/docs/stix_overview/techniques_by_group_sw.rst b/docs/stix_overview/techniques_by_group_sw.rst index 27a21cc3..cc357e37 100644 --- a/docs/stix_overview/techniques_by_group_sw.rst +++ b/docs/stix_overview/techniques_by_group_sw.rst @@ -7,29 +7,28 @@ These techniques are oftentimes distinct from the techniques used directly by a The following recipe can be used to retrieve the techniques used by a group's software: -```python -from stix2.utils import get_type_from_id -from stix2 import Filter +.. code-block:: python + from stix2.utils import get_type_from_id + from stix2 import Filter -def get_techniques_by_group_software(thesrc, group_stix_id): - # get the malware, tools that the group uses - group_uses = [ - r for r in thesrc.relationships(group_stix_id, 'uses', source_only=True) - if get_type_from_id(r.target_ref) in ['malware', 'tool'] - ] + def get_techniques_by_group_software(thesrc, group_stix_id): + # get the malware, tools that the group uses + group_uses = [ + r for r in thesrc.relationships(group_stix_id, 'uses', source_only=True) + if get_type_from_id(r.target_ref) in ['malware', 'tool'] + ] - # get the technique stix ids that the malware, tools use - software_uses = thesrc.query([ - Filter('type', '=', 'relationship'), - Filter('relationship_type', '=', 'uses'), - Filter('source_ref', 'in', [r.source_ref for r in group_uses]) - ]) + # get the technique stix ids that the malware, tools use + software_uses = thesrc.query([ + Filter('type', '=', 'relationship'), + Filter('relationship_type', '=', 'uses'), + Filter('source_ref', 'in', [r.source_ref for r in group_uses]) + ]) - #get the techniques themselves - return thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('id', 'in', [r.target_ref for r in software_uses]) - ]) + #get the techniques themselves + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('id', 'in', [r.target_ref for r in software_uses]) + ]) -get_techniques_by_group_software(src, "intrusion-set--f047ee18-7985-4946-8bfb-4ed754d3a0dd") -``` \ No newline at end of file + get_techniques_by_group_software(src, "intrusion-set--f047ee18-7985-4946-8bfb-4ed754d3a0dd") diff --git a/docs/stix_overview/techniques_by_platform.rst b/docs/stix_overview/techniques_by_platform.rst index ae2584bc..e333e5ed 100644 --- a/docs/stix_overview/techniques_by_platform.rst +++ b/docs/stix_overview/techniques_by_platform.rst @@ -7,15 +7,15 @@ Techniques by platform Techniques are associated with one or more platforms. You can query the techniques under a specific platform with the following code: -```python -from stix2 import Filter +.. code-block:: python + from stix2 import Filter -def get_techniques_by_platform(thesrc, platform): - return thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('x_mitre_platforms', '=', platform) - ]) + def get_techniques_by_platform(thesrc, platform): + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_platforms', '=', platform) + ]) + + # get techniques in the windows platform + get_techniques_by_platform(src, 'Windows') -# get techniques in the windows platform -get_techniques_by_platform(src, 'Windows') -``` diff --git a/docs/stix_overview/techniques_by_tactic.rst b/docs/stix_overview/techniques_by_tactic.rst index 5d345dc2..3b3d40bf 100644 --- a/docs/stix_overview/techniques_by_tactic.rst +++ b/docs/stix_overview/techniques_by_tactic.rst @@ -6,22 +6,21 @@ Techniques by tactic Techniques are related to tactics by their kill_chain_phases property. The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. -```python -from stix2 import Filter +.. code-block:: python + from stix2 import Filter -def get_tactic_techniques(thesrc, tactic): - # double checking the kill chain is MITRE ATT&CK - # note: kill_chain_name is different for other domains: - # - enterprise: "mitre-attack" - # - mobile: "mitre-mobile-attack" - # - ics: "mitre-ics-attack" - return thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('kill_chain_phases.phase_name', '=', tactic), - Filter('kill_chain_phases.kill_chain_name', '=', 'mitre-attack'), - ]) + def get_tactic_techniques(thesrc, tactic): + # double checking the kill chain is MITRE ATT&CK + # note: kill_chain_name is different for other domains: + # - enterprise: "mitre-attack" + # - mobile: "mitre-mobile-attack" + # - ics: "mitre-ics-attack" + return thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('kill_chain_phases.phase_name', '=', tactic), + Filter('kill_chain_phases.kill_chain_name', '=', 'mitre-attack'), + ]) -# use the x_mitre_shortname as argument -get_tactic_techniques(src, 'defense-evasion') -``` \ No newline at end of file + # use the x_mitre_shortname as argument + get_tactic_techniques(src, 'defense-evasion') diff --git a/docs/stix_overview/techniques_subtechniques.rst b/docs/stix_overview/techniques_subtechniques.rst index b18aaa07..5b5cb875 100644 --- a/docs/stix_overview/techniques_subtechniques.rst +++ b/docs/stix_overview/techniques_subtechniques.rst @@ -5,33 +5,32 @@ Getting techniques or sub-techniques ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. -```python -from stix2 import Filter +.. code-block:: python + from stix2 import Filter -def get_techniques_or_subtechniques(thesrc, include="both"): - """Filter Techniques or Sub-Techniques from ATT&CK Enterprise Domain. - include argument has three options: "techniques", "subtechniques", or "both" - depending on the intended behavior.""" - if include == "techniques": - query_results = thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('x_mitre_is_subtechnique', '=', False) - ]) - elif include == "subtechniques": - query_results = thesrc.query([ - Filter('type', '=', 'attack-pattern'), - Filter('x_mitre_is_subtechnique', '=', True) - ]) - elif include == "both": - query_results = thesrc.query([ - Filter('type', '=', 'attack-pattern') - ]) - else: - raise RuntimeError("Unknown option %s!" % include) + def get_techniques_or_subtechniques(thesrc, include="both"): + """Filter Techniques or Sub-Techniques from ATT&CK Enterprise Domain. + include argument has three options: "techniques", "subtechniques", or "both" + depending on the intended behavior.""" + if include == "techniques": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_is_subtechnique', '=', False) + ]) + elif include == "subtechniques": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern'), + Filter('x_mitre_is_subtechnique', '=', True) + ]) + elif include == "both": + query_results = thesrc.query([ + Filter('type', '=', 'attack-pattern') + ]) + else: + raise RuntimeError("Unknown option %s!" % include) - return query_results + return query_results -subtechniques = get_techniques_or_subtechniques(src, "subtechniques") -subtechniques = remove_revoked_deprecated(subtechniques) # see https://github.com/mitre/cti/blob/master/USAGE.md#removing-revoked-and-deprecated-objects -``` \ No newline at end of file + subtechniques = get_techniques_or_subtechniques(src, "subtechniques") + subtechniques = remove_revoked_deprecated(subtechniques) # see https://github.com/mitre/cti/blob/master/USAGE.md#removing-revoked-and-deprecated-objects From a5ff0f20034cd9412f335640657f510cd960916b Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Wed, 5 Jul 2023 13:35:48 -0400 Subject: [PATCH 023/159] Update getting_software.rst --- docs/stix_overview/getting_software.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_overview/getting_software.rst index 932fc3b5..6b9a9121 100644 --- a/docs/stix_overview/getting_software.rst +++ b/docs/stix_overview/getting_software.rst @@ -6,7 +6,7 @@ Getting software Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. .. code-block:: python -from itertools import chain + from itertools import chain from stix2 import Filter def get_software(thesrc): From 9c015f187cdfb7a67bf8bcddaf3586426d340460 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 5 Jul 2023 13:37:50 -0400 Subject: [PATCH 024/159] trying to make python code visible --- docs/stix_overview/getting_revoked_object.rst | 1 + docs/stix_overview/getting_software.rst | 1 + docs/stix_overview/objects_by_content.rst | 1 + docs/stix_overview/objects_since_date.rst | 1 + docs/stix_overview/relationships_microlibrary.rst | 1 + docs/stix_overview/remove_revoked_deprecated.rst | 1 + docs/stix_overview/tactics_by_matrix.rst | 1 + docs/stix_overview/techniques_by_group_sw.rst | 1 + docs/stix_overview/techniques_by_platform.rst | 1 + docs/stix_overview/techniques_by_tactic.rst | 1 + 10 files changed, 10 insertions(+) diff --git a/docs/stix_overview/getting_revoked_object.rst b/docs/stix_overview/getting_revoked_object.rst index 71e60b69..97c5f38e 100644 --- a/docs/stix_overview/getting_revoked_object.rst +++ b/docs/stix_overview/getting_revoked_object.rst @@ -6,6 +6,7 @@ Getting a revoking object When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: .. code-block:: python + from stix2 import Filter def getRevokedBy(stix_id, thesrc): diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_overview/getting_software.rst index 6b9a9121..a5750a3f 100644 --- a/docs/stix_overview/getting_software.rst +++ b/docs/stix_overview/getting_software.rst @@ -6,6 +6,7 @@ Getting software Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. .. code-block:: python + from itertools import chain from stix2 import Filter diff --git a/docs/stix_overview/objects_by_content.rst b/docs/stix_overview/objects_by_content.rst index ace27a91..8de61d1b 100644 --- a/docs/stix_overview/objects_by_content.rst +++ b/docs/stix_overview/objects_by_content.rst @@ -6,6 +6,7 @@ Objects by content Sometimes it may be useful to query objects by the content of their description: .. code-block:: python + from stix2 import Filter def get_techniques_by_content(thesrc, content): diff --git a/docs/stix_overview/objects_since_date.rst b/docs/stix_overview/objects_since_date.rst index df2a23c4..751e3a44 100644 --- a/docs/stix_overview/objects_since_date.rst +++ b/docs/stix_overview/objects_since_date.rst @@ -5,6 +5,7 @@ Objects created or modified since a given date Sometimes you may want to get a list of objects which have been created or modified after a certain time. .. code-block:: python + from stix2 import Filter def get_created_after(thesrc, timestamp): diff --git a/docs/stix_overview/relationships_microlibrary.rst b/docs/stix_overview/relationships_microlibrary.rst index bf296cb6..6fa2c4ea 100644 --- a/docs/stix_overview/relationships_microlibrary.rst +++ b/docs/stix_overview/relationships_microlibrary.rst @@ -10,6 +10,7 @@ This microlibrary can be used to build a lookup table of stixID to related objec The argument to each accessor function is a STIX2 MemoryStore to build the relationship mappings from. .. code-block:: python + from pprint import pprint from stix2 import MemoryStore, Filter diff --git a/docs/stix_overview/remove_revoked_deprecated.rst b/docs/stix_overview/remove_revoked_deprecated.rst index 5f3b6b9b..4bd83530 100644 --- a/docs/stix_overview/remove_revoked_deprecated.rst +++ b/docs/stix_overview/remove_revoked_deprecated.rst @@ -10,6 +10,7 @@ longer maintained by ATT&CK. We recommend _not_ using built-in STIX filters for removing revoked objects (e.g `Filter('revoked', '=', False)`). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See [issue #127](https://github.com/mitre/cti/issues/127) for more details. .. code-block:: python + from stix2 import Filter def remove_revoked_deprecated(stix_objects): diff --git a/docs/stix_overview/tactics_by_matrix.rst b/docs/stix_overview/tactics_by_matrix.rst index 8f10fa8f..95a3578c 100644 --- a/docs/stix_overview/tactics_by_matrix.rst +++ b/docs/stix_overview/tactics_by_matrix.rst @@ -7,6 +7,7 @@ found within the `tactic_refs` property in a matrix. The order of the tactics in the ordering of the tactics in that matrix. The following recipe returns a structured list of tactics within each matrix of the input DataStore. .. code-block:: python + from stix2 import Filter def getTacticsByMatrix(thesrc): diff --git a/docs/stix_overview/techniques_by_group_sw.rst b/docs/stix_overview/techniques_by_group_sw.rst index cc357e37..7c0693d7 100644 --- a/docs/stix_overview/techniques_by_group_sw.rst +++ b/docs/stix_overview/techniques_by_group_sw.rst @@ -8,6 +8,7 @@ These techniques are oftentimes distinct from the techniques used directly by a The following recipe can be used to retrieve the techniques used by a group's software: .. code-block:: python + from stix2.utils import get_type_from_id from stix2 import Filter diff --git a/docs/stix_overview/techniques_by_platform.rst b/docs/stix_overview/techniques_by_platform.rst index e333e5ed..122e8c6c 100644 --- a/docs/stix_overview/techniques_by_platform.rst +++ b/docs/stix_overview/techniques_by_platform.rst @@ -8,6 +8,7 @@ Techniques are associated with one or more platforms. You can query the techniqu under a specific platform with the following code: .. code-block:: python + from stix2 import Filter def get_techniques_by_platform(thesrc, platform): diff --git a/docs/stix_overview/techniques_by_tactic.rst b/docs/stix_overview/techniques_by_tactic.rst index 3b3d40bf..af87a6ac 100644 --- a/docs/stix_overview/techniques_by_tactic.rst +++ b/docs/stix_overview/techniques_by_tactic.rst @@ -7,6 +7,7 @@ Techniques are related to tactics by their kill_chain_phases property. The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. .. code-block:: python + from stix2 import Filter def get_tactic_techniques(thesrc, tactic): From 32ea070201255c96eb4c8292e25b3ca0b4fe3fb2 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 5 Jul 2023 13:39:55 -0400 Subject: [PATCH 025/159] trying to make python code visible part 2 --- docs/stix_overview/attack_id.rst | 2 ++ docs/stix_overview/by_alias.rst | 1 + docs/stix_overview/by_name.rst | 1 + docs/stix_overview/getting-started.rst | 11 +++++++++++ docs/stix_overview/objects_by_type.rst | 1 + docs/stix_overview/stix_id.rst | 1 + docs/stix_overview/techniques_subtechniques.rst | 1 + 7 files changed, 18 insertions(+) diff --git a/docs/stix_overview/attack_id.rst b/docs/stix_overview/attack_id.rst index 414ee1b1..5474fe4f 100644 --- a/docs/stix_overview/attack_id.rst +++ b/docs/stix_overview/attack_id.rst @@ -6,6 +6,7 @@ By ATT&CK ID The following recipe can be used to retrieve an object according to its ATT&CK ID: .. code-block:: python + from stix2 import Filter g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] @@ -14,6 +15,7 @@ The following recipe can be used to retrieve an object according to its ATT&CK I Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. .. code-block:: python + from stix2 import Filter t1134 = src.query([ diff --git a/docs/stix_overview/by_alias.rst b/docs/stix_overview/by_alias.rst index c2abf03a..a0c3bc4f 100644 --- a/docs/stix_overview/by_alias.rst +++ b/docs/stix_overview/by_alias.rst @@ -5,6 +5,7 @@ By Alias The following methodology can be used to find the group corresponding to a given alias: .. code-block:: python + from stix2 import Filter def get_group_by_alias(thesrc, alias): diff --git a/docs/stix_overview/by_name.rst b/docs/stix_overview/by_name.rst index 8a3b793c..ae3a992c 100644 --- a/docs/stix_overview/by_name.rst +++ b/docs/stix_overview/by_name.rst @@ -7,6 +7,7 @@ By Name The following recipe retrieves an object according to its name: .. code-block:: python + from stix2 import Filter def get_technique_by_name(thesrc, name): diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/getting-started.rst index 168f7eaa..4cd00653 100644 --- a/docs/stix_overview/getting-started.rst +++ b/docs/stix_overview/getting-started.rst @@ -24,12 +24,14 @@ Before installing requirements, we recommend setting up a virtual environment: [stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: .. code-block:: python + from stix2 import Filter However, if you are aiming to extend the ATT&CK dataset with new objects or implement complex workflows, you may need to use the `v20` specifier for some imports. This ensures that the objects use the STIX 2.0 API instead of the STIX 2.1 API. For example: .. code-block:: python + from stix2.v20 import AttackPattern @@ -42,6 +44,7 @@ You can see a full list of the classes which have versioned imports [here](https If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. .. code-block:: python + from taxii2client.v20 import Collection @@ -59,6 +62,7 @@ Each domain in this repo is formatted according to the [STIX2 FileSystem spec](h Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: .. code-block:: python + from stix2 import FileSystemSource src = FileSystemSource('./cti/enterprise-attack') @@ -69,6 +73,7 @@ src = FileSystemSource('./cti/enterprise-attack') If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: .. code-block:: python + from stix2 import MemoryStore src = MemoryStore() @@ -95,6 +100,7 @@ Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII You can also get a list of available collection from the server directly: .. code-block:: python + from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 server = Server("https://cti-taxii.mitre.org/taxii/") @@ -107,6 +113,7 @@ You can also get a list of available collection from the server directly: The following recipe demonstrates how to access the enterprise-attack data from the TAXII server. .. code-block:: python + from stix2 import TAXIICollectionSource from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 @@ -129,6 +136,7 @@ While typically the TAXII method is more desirable for "live" access, this metho access data on a branch of the MITRE/CTI repo (the TAXII server only holds the master branch) or in the case of a TAXII server outage. .. code-block:: python + import requests from stix2 import MemoryStore @@ -147,6 +155,7 @@ ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.co In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: .. code-block:: python + import requests from stix2 import MemoryStore @@ -161,6 +170,7 @@ In addition to checking out the repo under the tag for a given version or downlo You can get a list of ATT&CK versions programmatically using the github API: .. code-block:: python + import requests import re @@ -178,6 +188,7 @@ domains into a single DataStore. Use any of the methods above to acquire the ind a single CompositeDataSource: .. code-block:: python + from stix2 import CompositeDataSource src = CompositeDataSource() diff --git a/docs/stix_overview/objects_by_type.rst b/docs/stix_overview/objects_by_type.rst index efed2f38..efe75fcb 100644 --- a/docs/stix_overview/objects_by_type.rst +++ b/docs/stix_overview/objects_by_type.rst @@ -6,6 +6,7 @@ Objects by type See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. .. code-block:: python + from stix2 import Filter # use the appropriate STIX type in the query according to the desired ATT&CK type diff --git a/docs/stix_overview/stix_id.rst b/docs/stix_overview/stix_id.rst index 030f65b1..80d4ed1a 100644 --- a/docs/stix_overview/stix_id.rst +++ b/docs/stix_overview/stix_id.rst @@ -5,4 +5,5 @@ By STIX ID The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. .. code-block:: python + g0075 = src.get("intrusion-set--f40eb8ce-2a74-4e56-89a1-227021410142") diff --git a/docs/stix_overview/techniques_subtechniques.rst b/docs/stix_overview/techniques_subtechniques.rst index 5b5cb875..4aa18d8c 100644 --- a/docs/stix_overview/techniques_subtechniques.rst +++ b/docs/stix_overview/techniques_subtechniques.rst @@ -6,6 +6,7 @@ Getting techniques or sub-techniques ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. .. code-block:: python + from stix2 import Filter def get_techniques_or_subtechniques(thesrc, include="both"): From 9617aecee3307f9567ea867af191ce039079354e Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 5 Jul 2023 16:19:23 -0400 Subject: [PATCH 026/159] added bolding to headers --- docs/stix_overview/attack_id.rst | 2 +- docs/stix_overview/by_alias.rst | 2 +- docs/stix_overview/by_name.rst | 2 +- docs/stix_overview/deprecated_revoked.rst | 2 +- docs/stix_overview/getting-started.rst | 22 +++++++++---------- .../stix_overview/getting_related_objects.rst | 2 +- docs/stix_overview/getting_revoked_object.rst | 2 +- docs/stix_overview/getting_software.rst | 2 +- docs/stix_overview/multiple_objects.rst | 2 +- docs/stix_overview/objects_by_content.rst | 2 +- docs/stix_overview/objects_by_type.rst | 2 +- docs/stix_overview/objects_since_date.rst | 2 +- .../relationships_microlibrary.rst | 2 +- .../remove_revoked_deprecated.rst | 2 +- docs/stix_overview/stix-recipes.rst | 4 ++-- docs/stix_overview/stix_id.rst | 3 ++- docs/stix_overview/tactics_by_matrix.rst | 2 +- docs/stix_overview/techniques_by_group_sw.rst | 2 +- docs/stix_overview/techniques_by_platform.rst | 2 +- docs/stix_overview/techniques_by_tactic.rst | 2 +- .../techniques_subtechniques.rst | 2 +- 21 files changed, 33 insertions(+), 32 deletions(-) diff --git a/docs/stix_overview/attack_id.rst b/docs/stix_overview/attack_id.rst index 5474fe4f..904cd14a 100644 --- a/docs/stix_overview/attack_id.rst +++ b/docs/stix_overview/attack_id.rst @@ -1,7 +1,7 @@ By ATT&CK ID =============== -#### By ATT&CK ID +**By ATT&CK ID** The following recipe can be used to retrieve an object according to its ATT&CK ID: diff --git a/docs/stix_overview/by_alias.rst b/docs/stix_overview/by_alias.rst index a0c3bc4f..d1f0e23c 100644 --- a/docs/stix_overview/by_alias.rst +++ b/docs/stix_overview/by_alias.rst @@ -1,6 +1,6 @@ By Alias =============== -#### By alias +**By alias** The following methodology can be used to find the group corresponding to a given alias: diff --git a/docs/stix_overview/by_name.rst b/docs/stix_overview/by_name.rst index ae3a992c..90750aed 100644 --- a/docs/stix_overview/by_name.rst +++ b/docs/stix_overview/by_name.rst @@ -2,7 +2,7 @@ By Name =============== -#### By name +**By name** The following recipe retrieves an object according to its name: diff --git a/docs/stix_overview/deprecated_revoked.rst b/docs/stix_overview/deprecated_revoked.rst index 2704c75e..8c2f9fcd 100644 --- a/docs/stix_overview/deprecated_revoked.rst +++ b/docs/stix_overview/deprecated_revoked.rst @@ -1,7 +1,7 @@ Working with deprecated and revoked objects =============== -### Working with deprecated and revoked objects +**Working with deprecated and revoked objects** Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either `x_mitre_deprecated` or `revoked`) noting their status. In the case of revoked objects, a relationship of type `revoked-by` is also created targeting the replacing object. diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/getting-started.rst index 4cd00653..72805360 100644 --- a/docs/stix_overview/getting-started.rst +++ b/docs/stix_overview/getting-started.rst @@ -1,14 +1,14 @@ Getting Started =============== -## Accessing ATT&CK data in python +**Accessing ATT&CK data in python** There are several ways to acquire the ATT&CK data in Python. All of them will provide an object implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. -### Requirements and imports +**Requirements and imports** Before installing requirements, we recommend setting up a virtual environment: @@ -19,7 +19,7 @@ Before installing requirements, we recommend setting up a virtual environment: - macOS and Linux: `source env/bin/activate` - Windows: `env/Scripts/activate.bat` -#### stix2 +**stix2** [stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: @@ -37,7 +37,7 @@ However, if you are aiming to extend the ATT&CK dataset with new objects or impl You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). -#### taxii2client +**taxii2client** [taxii2-client can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-taxii-client#installation). The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of `taxii2client` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. @@ -48,7 +48,7 @@ If the TAXII Client is getting a 406 Response, make sure you are running the lat from taxii2client.v20 import Collection -### Access local content +**Access local content** Many users may opt to access the ATT&CK content via a local copy of the STIX data on this repo. This can be advantageous for several reasons: @@ -68,7 +68,7 @@ Therefore you can use a `FileSystemSource` to load a domain, for example to load src = FileSystemSource('./cti/enterprise-attack') -#### Access via bundle +**Access via bundle** If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: @@ -80,14 +80,14 @@ If you instead prefer to download just the domain bundle, e.g [enterprise-attack src.load_from_file("enterprise-attack.json") -### Access live content +**Access live content** Some users may instead prefer to access "live" ATT&CK content over the internet. This is advantageous for several reasons: - Always stays up to date with the evolving ATT&CK catalog - Doesn't require an initial download of the ATT&CK content, generally requires less setup -#### Access from the ATT&CK TAXII server +**Access from the ATT&CK TAXII server** Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII, the ATT&CK domains are represented as collections with static IDs: @@ -129,7 +129,7 @@ The following recipe demonstrates how to access the enterprise-attack data from For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). -#### Access from Github via requests +**Access from Github via requests** Users can alternatively access the data from MITRE/CTI using HTTP requests, and load the resulting content into a MemoryStore. While typically the TAXII method is more desirable for "live" access, this method can be useful if you want to @@ -148,7 +148,7 @@ access data on a branch of the MITRE/CTI repo (the TAXII server only holds the m src = get_data_from_branch("enterprise-attack") -### Access a specific version of ATT&CK +**Access a specific version of ATT&CK** ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.com/mitre/cti/tags). Tags prefixed with `ATT&CK-v` correspond to ATT&CK versions and tags prefixed with `CAPEC-v` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. @@ -180,7 +180,7 @@ You can get a list of ATT&CK versions programmatically using the github API: # versions = ["1.0", "2.0", ...] -### Access multiple domains simultaneously +**Access multiple domains simultaneously** Because ATT&CK is stored in multiple domains (as of this writing, enterprise-attack, mobile-attack and ics-attack), the above methodologies will only allow you to work with a single domain at a time. While oftentimes the hard separation of domains is advantageous, occasionally it is useful to combine diff --git a/docs/stix_overview/getting_related_objects.rst b/docs/stix_overview/getting_related_objects.rst index b3a48a1f..2ea4f3c1 100644 --- a/docs/stix_overview/getting_related_objects.rst +++ b/docs/stix_overview/getting_related_objects.rst @@ -1,6 +1,6 @@ Getting related objects =============== -### Getting related objects +**Getting related objects** A large part of working with ATT&CK revolves around parsing relationships between objects. It is useful to track not only the related object but the relationship itself because a description is often diff --git a/docs/stix_overview/getting_revoked_object.rst b/docs/stix_overview/getting_revoked_object.rst index 97c5f38e..2bfdc4ac 100644 --- a/docs/stix_overview/getting_revoked_object.rst +++ b/docs/stix_overview/getting_revoked_object.rst @@ -1,7 +1,7 @@ Getting a revoking object =============== -#### Getting a revoking object +**Getting a revoking object** When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_overview/getting_software.rst index a5750a3f..2d5cda6e 100644 --- a/docs/stix_overview/getting_software.rst +++ b/docs/stix_overview/getting_software.rst @@ -1,7 +1,7 @@ Getting software =============== -##### Getting software +**Getting software** Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. diff --git a/docs/stix_overview/multiple_objects.rst b/docs/stix_overview/multiple_objects.rst index 0ee3abfc..36aff9a8 100644 --- a/docs/stix_overview/multiple_objects.rst +++ b/docs/stix_overview/multiple_objects.rst @@ -1,6 +1,6 @@ Getting multiple objects =============== -### Getting multiple objects +**Getting multiple objects** The recipes in this following section address how to query the dataset for multiple objects. diff --git a/docs/stix_overview/objects_by_content.rst b/docs/stix_overview/objects_by_content.rst index 8de61d1b..d71939b3 100644 --- a/docs/stix_overview/objects_by_content.rst +++ b/docs/stix_overview/objects_by_content.rst @@ -1,7 +1,7 @@ Objects by content =============== -#### Objects by content +**Objects by content** Sometimes it may be useful to query objects by the content of their description: diff --git a/docs/stix_overview/objects_by_type.rst b/docs/stix_overview/objects_by_type.rst index efe75fcb..d5d5c500 100644 --- a/docs/stix_overview/objects_by_type.rst +++ b/docs/stix_overview/objects_by_type.rst @@ -1,7 +1,7 @@ Objects by type =============== -#### Objects by type +**Objects by type** See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. diff --git a/docs/stix_overview/objects_since_date.rst b/docs/stix_overview/objects_since_date.rst index 751e3a44..4fa16938 100644 --- a/docs/stix_overview/objects_since_date.rst +++ b/docs/stix_overview/objects_since_date.rst @@ -1,6 +1,6 @@ Objects created or modified since a given date =============== -#### Objects created or modified since a given date +**Objects created or modified since a given date** Sometimes you may want to get a list of objects which have been created or modified after a certain time. diff --git a/docs/stix_overview/relationships_microlibrary.rst b/docs/stix_overview/relationships_microlibrary.rst index 6fa2c4ea..83668d20 100644 --- a/docs/stix_overview/relationships_microlibrary.rst +++ b/docs/stix_overview/relationships_microlibrary.rst @@ -1,7 +1,7 @@ Relationships microlibrary =============== -#### Relationships microlibrary +**Relationships microlibrary** NOTE: The following code is intended to be used with the ATT&CK v12 release which includes Campaign Objects. The examples are backwards-compatible for previous versions af ATT&CK that omit those objects. diff --git a/docs/stix_overview/remove_revoked_deprecated.rst b/docs/stix_overview/remove_revoked_deprecated.rst index 4bd83530..1c0b55a8 100644 --- a/docs/stix_overview/remove_revoked_deprecated.rst +++ b/docs/stix_overview/remove_revoked_deprecated.rst @@ -1,7 +1,7 @@ Removing revoked and deprecated objects =============== -#### Removing revoked and deprecated objects +**Removing revoked and deprecated objects** Revoked and deprecated objects are kept in the knowledge base so that workflows relying on those objects are not broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no diff --git a/docs/stix_overview/stix-recipes.rst b/docs/stix_overview/stix-recipes.rst index 8e180453..ef6a4cb2 100644 --- a/docs/stix_overview/stix-recipes.rst +++ b/docs/stix_overview/stix-recipes.rst @@ -1,13 +1,13 @@ Stix Recipes =============== -## Python recipes +**Python recipes** Below are example python recipes which can be used to work with ATT&CK data. They assume the existence of an object implementing the DataStore API. Any of the methods outlined in the [Accessing ATT&CK data in python](#accessing-ATTCK-Data-in-Python) section should provide an object implementing this API. This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. See also the section on [Requirements and imports](#requirements-and-imports). -### Getting an object +**Getting an object** The recipes in this section address how to query the dataset for a single object. diff --git a/docs/stix_overview/stix_id.rst b/docs/stix_overview/stix_id.rst index 80d4ed1a..ab79f78d 100644 --- a/docs/stix_overview/stix_id.rst +++ b/docs/stix_overview/stix_id.rst @@ -1,6 +1,7 @@ By STIX ID =============== -#### By STIX ID + +**By STIX ID** The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. diff --git a/docs/stix_overview/tactics_by_matrix.rst b/docs/stix_overview/tactics_by_matrix.rst index 95a3578c..469662e1 100644 --- a/docs/stix_overview/tactics_by_matrix.rst +++ b/docs/stix_overview/tactics_by_matrix.rst @@ -1,6 +1,6 @@ Tactics by matrix =============== -#### Tactics by matrix +**Tactics by matrix** The tactics are individual objects (`x-mitre-tactic`), and their order in a matrix (`x-mitre-matrix`) is found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches diff --git a/docs/stix_overview/techniques_by_group_sw.rst b/docs/stix_overview/techniques_by_group_sw.rst index 7c0693d7..e1def1a5 100644 --- a/docs/stix_overview/techniques_by_group_sw.rst +++ b/docs/stix_overview/techniques_by_group_sw.rst @@ -1,6 +1,6 @@ Getting techniques used by a group's software =============== -#### Getting techniques used by a group's software +**Getting techniques used by a group's software** Because a group uses software, and software uses techniques, groups can be considered indirect users of techniques used by their software. These techniques are oftentimes distinct from the techniques used directly by a group, although there are occasionally intersections in these two sets of techniques. diff --git a/docs/stix_overview/techniques_by_platform.rst b/docs/stix_overview/techniques_by_platform.rst index 122e8c6c..5dd0573a 100644 --- a/docs/stix_overview/techniques_by_platform.rst +++ b/docs/stix_overview/techniques_by_platform.rst @@ -2,7 +2,7 @@ Techniques by platform =============== -#### Techniques by platform +**Techniques by platform** Techniques are associated with one or more platforms. You can query the techniques under a specific platform with the following code: diff --git a/docs/stix_overview/techniques_by_tactic.rst b/docs/stix_overview/techniques_by_tactic.rst index af87a6ac..43a47d66 100644 --- a/docs/stix_overview/techniques_by_tactic.rst +++ b/docs/stix_overview/techniques_by_tactic.rst @@ -1,7 +1,7 @@ Techniques by tactic =============== -#### Techniques by tactic +**Techniques by tactic** Techniques are related to tactics by their kill_chain_phases property. The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. diff --git a/docs/stix_overview/techniques_subtechniques.rst b/docs/stix_overview/techniques_subtechniques.rst index 4aa18d8c..ba7681c8 100644 --- a/docs/stix_overview/techniques_subtechniques.rst +++ b/docs/stix_overview/techniques_subtechniques.rst @@ -1,7 +1,7 @@ Getting techniques or sub-techniques =============== -##### Getting techniques or sub-techniques +**Getting techniques or sub-techniques** ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. From 018a9e1f55d1d1862294f39aa1390f1cb2d606ea Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Thu, 6 Jul 2023 14:51:19 -0400 Subject: [PATCH 027/159] Update and rename getting-started.rst to access-attack.rst --- docs/stix_overview/{getting-started.rst => access-attack.rst} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename docs/stix_overview/{getting-started.rst => access-attack.rst} (99%) diff --git a/docs/stix_overview/getting-started.rst b/docs/stix_overview/access-attack.rst similarity index 99% rename from docs/stix_overview/getting-started.rst rename to docs/stix_overview/access-attack.rst index 72805360..cc1088ea 100644 --- a/docs/stix_overview/getting-started.rst +++ b/docs/stix_overview/access-attack.rst @@ -1,4 +1,4 @@ -Getting Started +Accessing ATT&CK data in python =============== **Accessing ATT&CK data in python** From 387065d0e2356167517d62cd0c4f387235c6dbd0 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Thu, 6 Jul 2023 14:51:49 -0400 Subject: [PATCH 028/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 932eee08..936b4c50 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -28,7 +28,7 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: STIX Overview - stix_overview/getting-started + stix_overview/access-attack stix_overview/stix-recipes stix_overview/attack_id stix_overview/by_alias From 24d5ef23709b99a2ef1f134ccd21a70f6fd0e00b Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Thu, 6 Jul 2023 16:23:19 -0400 Subject: [PATCH 029/159] Update README.md --- README.md | 33 +-------------------------------- 1 file changed, 1 insertion(+), 32 deletions(-) diff --git a/README.md b/README.md index f566eaa9..4b1cd4ce 100644 --- a/README.md +++ b/README.md @@ -33,38 +33,7 @@ More detailed information and examples about the specific usage of the additiona ## Related MITRE Work -### CTI - -[Cyber Threat Intelligence repository](https://github.com/mitre/cti) of the ATT&CK catalog expressed in STIX 2.0 JSON. -This repository also contains [our USAGE document](https://github.com/mitre/cti/blob/master/USAGE.md) which includes -additional examples of accessing and parsing our dataset in Python. - -### ATT&CK - -ATT&CK® is a curated knowledge base and model for cyber adversary behavior, reflecting the various phases of -an adversary’s lifecycle, and the platforms they are known to target. -ATT&CK is useful for understanding security risk against known adversary behavior, -for planning security improvements, and verifying defenses work as expected. - - - -### STIX - -Structured Threat Information Expression (STIX™) is a language and serialization format used to exchange cyber threat intelligence (CTI). - -STIX enables organizations to share CTI with one another in a consistent and machine-readable manner, -allowing security communities to better understand what computer-based attacks they are most likely to -see and to anticipate and/or respond to those attacks faster and more effectively. - -STIX is designed to improve many capabilities, such as collaborative threat analysis, automated threat exchange, automated detection and response, and more. - - - -### ATT&CK scripts - -One-off scripts and code examples you can use as inspiration for how to work with ATT&CK programmatically. Many of the functionalities found in the mitreattack-python package were originally posted on attack-scripts. - - +Go to [this link](https://mitreattack-python.readthedocs.io/en/latest/related_work.html) for related MITRE work. ## Contributing From 25f728345dd260ccfd061895e5dada429f9853a8 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 6 Jul 2023 16:55:15 -0400 Subject: [PATCH 030/159] changed folder name and removed secondary titles --- .../attack_id.rst | 2 - .../by_alias.rst | 1 - .../by_name.rst | 2 - .../deprecated_revoked.rst | 2 - docs/stix_primer/getting-started.rst | 198 ++++++++++++++++++ .../getting_related_objects.rst | 1 - .../getting_revoked_object.rst | 2 - .../getting_software.rst | 2 - .../multiple_objects.rst | 1 - .../objects_by_content.rst | 2 - .../objects_by_type.rst | 2 - .../objects_since_date.rst | 1 - .../relationships_microlibrary.rst | 2 - .../remove_revoked_deprecated.rst | 2 - .../stix-recipes.rst | 0 .../stix_id.rst | 2 - .../tactics_by_matrix.rst | 1 - .../techniques_by_group_sw.rst | 1 - .../techniques_by_platform.rst | 3 - .../techniques_by_tactic.rst | 2 - .../techniques_subtechniques.rst | 2 - 21 files changed, 198 insertions(+), 33 deletions(-) rename docs/{stix_overview => stix_primer}/attack_id.rst (98%) rename docs/{stix_overview => stix_primer}/by_alias.rst (96%) rename docs/{stix_overview => stix_primer}/by_name.rst (97%) rename docs/{stix_overview => stix_primer}/deprecated_revoked.rst (93%) create mode 100644 docs/stix_primer/getting-started.rst rename docs/{stix_overview => stix_primer}/getting_related_objects.rst (92%) rename docs/{stix_overview => stix_primer}/getting_revoked_object.rst (96%) rename docs/{stix_overview => stix_primer}/getting_software.rst (95%) rename docs/{stix_overview => stix_primer}/multiple_objects.rst (95%) rename docs/{stix_overview => stix_primer}/objects_by_content.rst (95%) rename docs/{stix_overview => stix_primer}/objects_by_type.rst (94%) rename docs/{stix_overview => stix_primer}/objects_since_date.rst (95%) rename docs/{stix_overview => stix_primer}/relationships_microlibrary.rst (99%) rename docs/{stix_overview => stix_primer}/remove_revoked_deprecated.rst (96%) rename docs/{stix_overview => stix_primer}/stix-recipes.rst (100%) rename docs/{stix_overview => stix_primer}/stix_id.rst (95%) rename docs/{stix_overview => stix_primer}/tactics_by_matrix.rst (97%) rename docs/{stix_overview => stix_primer}/techniques_by_group_sw.rst (96%) rename docs/{stix_overview => stix_primer}/techniques_by_platform.rst (94%) rename docs/{stix_overview => stix_primer}/techniques_by_tactic.rst (97%) rename docs/{stix_overview => stix_primer}/techniques_subtechniques.rst (97%) diff --git a/docs/stix_overview/attack_id.rst b/docs/stix_primer/attack_id.rst similarity index 98% rename from docs/stix_overview/attack_id.rst rename to docs/stix_primer/attack_id.rst index 904cd14a..4a449c7c 100644 --- a/docs/stix_overview/attack_id.rst +++ b/docs/stix_primer/attack_id.rst @@ -1,8 +1,6 @@ By ATT&CK ID =============== -**By ATT&CK ID** - The following recipe can be used to retrieve an object according to its ATT&CK ID: .. code-block:: python diff --git a/docs/stix_overview/by_alias.rst b/docs/stix_primer/by_alias.rst similarity index 96% rename from docs/stix_overview/by_alias.rst rename to docs/stix_primer/by_alias.rst index d1f0e23c..8b01bc4d 100644 --- a/docs/stix_overview/by_alias.rst +++ b/docs/stix_primer/by_alias.rst @@ -1,6 +1,5 @@ By Alias =============== -**By alias** The following methodology can be used to find the group corresponding to a given alias: diff --git a/docs/stix_overview/by_name.rst b/docs/stix_primer/by_name.rst similarity index 97% rename from docs/stix_overview/by_name.rst rename to docs/stix_primer/by_name.rst index 90750aed..63bde04a 100644 --- a/docs/stix_overview/by_name.rst +++ b/docs/stix_primer/by_name.rst @@ -2,8 +2,6 @@ By Name =============== -**By name** - The following recipe retrieves an object according to its name: .. code-block:: python diff --git a/docs/stix_overview/deprecated_revoked.rst b/docs/stix_primer/deprecated_revoked.rst similarity index 93% rename from docs/stix_overview/deprecated_revoked.rst rename to docs/stix_primer/deprecated_revoked.rst index 8c2f9fcd..875d19e0 100644 --- a/docs/stix_overview/deprecated_revoked.rst +++ b/docs/stix_primer/deprecated_revoked.rst @@ -1,8 +1,6 @@ Working with deprecated and revoked objects =============== -**Working with deprecated and revoked objects** - Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either `x_mitre_deprecated` or `revoked`) noting their status. In the case of revoked objects, a relationship of type `revoked-by` is also created targeting the replacing object. Unlike other objects in the dataset, relationships cannot be revoked or deprecated. Relationships are considered deprecated/revoked if one of the objects it is attached to is revoked or deprecated. \ No newline at end of file diff --git a/docs/stix_primer/getting-started.rst b/docs/stix_primer/getting-started.rst new file mode 100644 index 00000000..72805360 --- /dev/null +++ b/docs/stix_primer/getting-started.rst @@ -0,0 +1,198 @@ +Getting Started +=============== + +**Accessing ATT&CK data in python** + +There are several ways to acquire the ATT&CK data in Python. All of them will provide an object +implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. + +This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. + +**Requirements and imports** + +Before installing requirements, we recommend setting up a virtual environment: + +1. Create virtual environment: + - macOS and Linux: `python3 -m venv env` + - Windows: `py -m venv env` +2. Activate the virtual environment: + - macOS and Linux: `source env/bin/activate` + - Windows: `env/Scripts/activate.bat` + +**stix2** + +[stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: + +.. code-block:: python + + from stix2 import Filter + + +However, if you are aiming to extend the ATT&CK dataset with new objects or implement complex workflows, you may need to use the `v20` specifier for some imports. This ensures that the objects use the STIX 2.0 API instead of the STIX 2.1 API. For example: + +.. code-block:: python + + from stix2.v20 import AttackPattern + + +You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). + +**taxii2client** + +[taxii2-client can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-taxii-client#installation). The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of `taxii2client` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. + +If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. + +.. code-block:: python + + from taxii2client.v20 import Collection + + +**Access local content** + +Many users may opt to access the ATT&CK content via a local copy of the STIX data on this repo. This can be advantageous for several reasons: + +- Doesn't require internet access after the initial download +- User can modify the ATT&CK content if desired +- Downloaded copy is static, so updates to the ATT&CK catalog won't cause bugs in automated workflows. User can still manually update by cloning a fresh version of the data + +#### Access via FileSystemSource + +Each domain in this repo is formatted according to the [STIX2 FileSystem spec](https://stix2.readthedocs.io/en/latest/guide/filesystem.html). +Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: + +.. code-block:: python + + from stix2 import FileSystemSource + +src = FileSystemSource('./cti/enterprise-attack') + + +**Access via bundle** + +If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: + +.. code-block:: python + + from stix2 import MemoryStore + + src = MemoryStore() + src.load_from_file("enterprise-attack.json") + + +**Access live content** + +Some users may instead prefer to access "live" ATT&CK content over the internet. This is advantageous for several reasons: + +- Always stays up to date with the evolving ATT&CK catalog +- Doesn't require an initial download of the ATT&CK content, generally requires less setup + +**Access from the ATT&CK TAXII server** + +Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII, the ATT&CK domains are represented as collections with static IDs: + +| domain | collection ID | +|:-------|:--------------| +| `enterprise-attack` | `95ecc380-afe9-11e4-9b6c-751b66dd541e` | +| `mobile-attack` | `2f669986-b40b-4423-b720-4396ca6a462b` | +| `ics-attack` | `02c3ef24-9cd4-48f3-a99f-b74ce24f1d34` | + +You can also get a list of available collection from the server directly: + +.. code-block:: python + + from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 + + server = Server("https://cti-taxii.mitre.org/taxii/") + api_root = server.api_roots[0] + # Print name and ID of all ATT&CK domains available as collections + for collection in api_root.collections: + print(collection.title.ljust(20) + collection.id) + + +The following recipe demonstrates how to access the enterprise-attack data from the TAXII server. + +.. code-block:: python + + from stix2 import TAXIICollectionSource + from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 + + collections = { + "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e", + "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b", + "ics-attack": "02c3ef24-9cd4-48f3-a99f-b74ce24f1d34" + } + + collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/") + src = TAXIICollectionSource(collection) + + +For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). + +**Access from Github via requests** + +Users can alternatively access the data from MITRE/CTI using HTTP requests, and load the resulting content into a MemoryStore. +While typically the TAXII method is more desirable for "live" access, this method can be useful if you want to +access data on a branch of the MITRE/CTI repo (the TAXII server only holds the master branch) or in the case of a TAXII server outage. + +.. code-block:: python + + import requests + from stix2 import MemoryStore + + def get_data_from_branch(domain, branch="master"): + """get the ATT&CK STIX data from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'. Branch should typically be master.""" + stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/{branch}/{domain}/{domain}.json").json() + return MemoryStore(stix_data=stix_json["objects"]) + + src = get_data_from_branch("enterprise-attack") + + +**Access a specific version of ATT&CK** + +ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.com/mitre/cti/tags). Tags prefixed with `ATT&CK-v` correspond to ATT&CK versions and tags prefixed with `CAPEC-v` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. + +In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: + +.. code-block:: python + + import requests + from stix2 import MemoryStore + + def get_data_from_version(domain, version): + """get the ATT&CK STIX data for the given version from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'.""" + stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v{version}/{domain}/{domain}.json").json() + return MemoryStore(stix_data=stix_json["objects"]) + + src = get_data_from_version("enterprise-attack", "5.2") + + +You can get a list of ATT&CK versions programmatically using the github API: + +.. code-block:: python + + import requests + import re + + refToTag = re.compile(r"ATT&CK-v(.*)") + tags = requests.get("https://api.github.com/repos/mitre/cti/git/refs/tags").json() + versions = list(map(lambda tag: refToTag.search(tag["ref"]).groups()[0] , filter(lambda tag: "ATT&CK-v" in tag["ref"], tags))) + # versions = ["1.0", "2.0", ...] + + +**Access multiple domains simultaneously** + +Because ATT&CK is stored in multiple domains (as of this writing, enterprise-attack, mobile-attack and ics-attack), the above methodologies will only allow you to work +with a single domain at a time. While oftentimes the hard separation of domains is advantageous, occasionally it is useful to combine +domains into a single DataStore. Use any of the methods above to acquire the individual datastores, and then use the following approach to combine them into +a single CompositeDataSource: + +.. code-block:: python + + from stix2 import CompositeDataSource + + src = CompositeDataSource() + src.add_data_sources([enterprise_attack_src, mobile_attack_src, ics_attack_src]) + + +You can then use this CompositeDataSource just as you would the DataSource for an individual domain. diff --git a/docs/stix_overview/getting_related_objects.rst b/docs/stix_primer/getting_related_objects.rst similarity index 92% rename from docs/stix_overview/getting_related_objects.rst rename to docs/stix_primer/getting_related_objects.rst index 2ea4f3c1..8ca9edef 100644 --- a/docs/stix_overview/getting_related_objects.rst +++ b/docs/stix_primer/getting_related_objects.rst @@ -1,6 +1,5 @@ Getting related objects =============== -**Getting related objects** A large part of working with ATT&CK revolves around parsing relationships between objects. It is useful to track not only the related object but the relationship itself because a description is often diff --git a/docs/stix_overview/getting_revoked_object.rst b/docs/stix_primer/getting_revoked_object.rst similarity index 96% rename from docs/stix_overview/getting_revoked_object.rst rename to docs/stix_primer/getting_revoked_object.rst index 2bfdc4ac..62e92873 100644 --- a/docs/stix_overview/getting_revoked_object.rst +++ b/docs/stix_primer/getting_revoked_object.rst @@ -1,8 +1,6 @@ Getting a revoking object =============== -**Getting a revoking object** - When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: .. code-block:: python diff --git a/docs/stix_overview/getting_software.rst b/docs/stix_primer/getting_software.rst similarity index 95% rename from docs/stix_overview/getting_software.rst rename to docs/stix_primer/getting_software.rst index 2d5cda6e..6308db32 100644 --- a/docs/stix_overview/getting_software.rst +++ b/docs/stix_primer/getting_software.rst @@ -1,8 +1,6 @@ Getting software =============== -**Getting software** - Because software are the union of two STIX types (`tool` and `malware`), the process for accessing software is slightly more complicated. .. code-block:: python diff --git a/docs/stix_overview/multiple_objects.rst b/docs/stix_primer/multiple_objects.rst similarity index 95% rename from docs/stix_overview/multiple_objects.rst rename to docs/stix_primer/multiple_objects.rst index 36aff9a8..77dfa923 100644 --- a/docs/stix_overview/multiple_objects.rst +++ b/docs/stix_primer/multiple_objects.rst @@ -1,6 +1,5 @@ Getting multiple objects =============== -**Getting multiple objects** The recipes in this following section address how to query the dataset for multiple objects. diff --git a/docs/stix_overview/objects_by_content.rst b/docs/stix_primer/objects_by_content.rst similarity index 95% rename from docs/stix_overview/objects_by_content.rst rename to docs/stix_primer/objects_by_content.rst index d71939b3..ccbb25a6 100644 --- a/docs/stix_overview/objects_by_content.rst +++ b/docs/stix_primer/objects_by_content.rst @@ -1,8 +1,6 @@ Objects by content =============== -**Objects by content** - Sometimes it may be useful to query objects by the content of their description: .. code-block:: python diff --git a/docs/stix_overview/objects_by_type.rst b/docs/stix_primer/objects_by_type.rst similarity index 94% rename from docs/stix_overview/objects_by_type.rst rename to docs/stix_primer/objects_by_type.rst index d5d5c500..339f69f5 100644 --- a/docs/stix_overview/objects_by_type.rst +++ b/docs/stix_primer/objects_by_type.rst @@ -1,8 +1,6 @@ Objects by type =============== -**Objects by type** - See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. .. code-block:: python diff --git a/docs/stix_overview/objects_since_date.rst b/docs/stix_primer/objects_since_date.rst similarity index 95% rename from docs/stix_overview/objects_since_date.rst rename to docs/stix_primer/objects_since_date.rst index 4fa16938..49454bb8 100644 --- a/docs/stix_overview/objects_since_date.rst +++ b/docs/stix_primer/objects_since_date.rst @@ -1,6 +1,5 @@ Objects created or modified since a given date =============== -**Objects created or modified since a given date** Sometimes you may want to get a list of objects which have been created or modified after a certain time. diff --git a/docs/stix_overview/relationships_microlibrary.rst b/docs/stix_primer/relationships_microlibrary.rst similarity index 99% rename from docs/stix_overview/relationships_microlibrary.rst rename to docs/stix_primer/relationships_microlibrary.rst index 83668d20..d3817bf8 100644 --- a/docs/stix_overview/relationships_microlibrary.rst +++ b/docs/stix_primer/relationships_microlibrary.rst @@ -1,8 +1,6 @@ Relationships microlibrary =============== -**Relationships microlibrary** - NOTE: The following code is intended to be used with the ATT&CK v12 release which includes Campaign Objects. The examples are backwards-compatible for previous versions af ATT&CK that omit those objects. diff --git a/docs/stix_overview/remove_revoked_deprecated.rst b/docs/stix_primer/remove_revoked_deprecated.rst similarity index 96% rename from docs/stix_overview/remove_revoked_deprecated.rst rename to docs/stix_primer/remove_revoked_deprecated.rst index 1c0b55a8..0de3e7d7 100644 --- a/docs/stix_overview/remove_revoked_deprecated.rst +++ b/docs/stix_primer/remove_revoked_deprecated.rst @@ -1,8 +1,6 @@ Removing revoked and deprecated objects =============== -**Removing revoked and deprecated objects** - Revoked and deprecated objects are kept in the knowledge base so that workflows relying on those objects are not broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no longer maintained by ATT&CK. diff --git a/docs/stix_overview/stix-recipes.rst b/docs/stix_primer/stix-recipes.rst similarity index 100% rename from docs/stix_overview/stix-recipes.rst rename to docs/stix_primer/stix-recipes.rst diff --git a/docs/stix_overview/stix_id.rst b/docs/stix_primer/stix_id.rst similarity index 95% rename from docs/stix_overview/stix_id.rst rename to docs/stix_primer/stix_id.rst index ab79f78d..60ee9a6e 100644 --- a/docs/stix_overview/stix_id.rst +++ b/docs/stix_primer/stix_id.rst @@ -1,8 +1,6 @@ By STIX ID =============== -**By STIX ID** - The following recipe can be used to retrieve an object according to its STIX ID. This is typically the preferred way to retrieve objects when working with ATT&CK data because STIX IDs are guaranteed to be unique. .. code-block:: python diff --git a/docs/stix_overview/tactics_by_matrix.rst b/docs/stix_primer/tactics_by_matrix.rst similarity index 97% rename from docs/stix_overview/tactics_by_matrix.rst rename to docs/stix_primer/tactics_by_matrix.rst index 469662e1..d30bb285 100644 --- a/docs/stix_overview/tactics_by_matrix.rst +++ b/docs/stix_primer/tactics_by_matrix.rst @@ -1,6 +1,5 @@ Tactics by matrix =============== -**Tactics by matrix** The tactics are individual objects (`x-mitre-tactic`), and their order in a matrix (`x-mitre-matrix`) is found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches diff --git a/docs/stix_overview/techniques_by_group_sw.rst b/docs/stix_primer/techniques_by_group_sw.rst similarity index 96% rename from docs/stix_overview/techniques_by_group_sw.rst rename to docs/stix_primer/techniques_by_group_sw.rst index e1def1a5..42cfd3a7 100644 --- a/docs/stix_overview/techniques_by_group_sw.rst +++ b/docs/stix_primer/techniques_by_group_sw.rst @@ -1,6 +1,5 @@ Getting techniques used by a group's software =============== -**Getting techniques used by a group's software** Because a group uses software, and software uses techniques, groups can be considered indirect users of techniques used by their software. These techniques are oftentimes distinct from the techniques used directly by a group, although there are occasionally intersections in these two sets of techniques. diff --git a/docs/stix_overview/techniques_by_platform.rst b/docs/stix_primer/techniques_by_platform.rst similarity index 94% rename from docs/stix_overview/techniques_by_platform.rst rename to docs/stix_primer/techniques_by_platform.rst index 5dd0573a..2bb48be7 100644 --- a/docs/stix_overview/techniques_by_platform.rst +++ b/docs/stix_primer/techniques_by_platform.rst @@ -1,9 +1,6 @@ Techniques by platform =============== - -**Techniques by platform** - Techniques are associated with one or more platforms. You can query the techniques under a specific platform with the following code: diff --git a/docs/stix_overview/techniques_by_tactic.rst b/docs/stix_primer/techniques_by_tactic.rst similarity index 97% rename from docs/stix_overview/techniques_by_tactic.rst rename to docs/stix_primer/techniques_by_tactic.rst index 43a47d66..a09761ac 100644 --- a/docs/stix_overview/techniques_by_tactic.rst +++ b/docs/stix_primer/techniques_by_tactic.rst @@ -1,8 +1,6 @@ Techniques by tactic =============== -**Techniques by tactic** - Techniques are related to tactics by their kill_chain_phases property. The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. diff --git a/docs/stix_overview/techniques_subtechniques.rst b/docs/stix_primer/techniques_subtechniques.rst similarity index 97% rename from docs/stix_overview/techniques_subtechniques.rst rename to docs/stix_primer/techniques_subtechniques.rst index ba7681c8..96ae2d73 100644 --- a/docs/stix_overview/techniques_subtechniques.rst +++ b/docs/stix_primer/techniques_subtechniques.rst @@ -1,8 +1,6 @@ Getting techniques or sub-techniques =============== -**Getting techniques or sub-techniques** - ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. .. code-block:: python From 107111d6ceddc7cbc2bd4a9545eb743d4cfa8925 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 11:54:22 -0400 Subject: [PATCH 031/159] removing extra folder --- .../access-attack.rst | 0 docs/stix_primer/getting-started.rst | 198 ------------------ 2 files changed, 198 deletions(-) rename docs/{stix_overview => stix_primer}/access-attack.rst (100%) delete mode 100644 docs/stix_primer/getting-started.rst diff --git a/docs/stix_overview/access-attack.rst b/docs/stix_primer/access-attack.rst similarity index 100% rename from docs/stix_overview/access-attack.rst rename to docs/stix_primer/access-attack.rst diff --git a/docs/stix_primer/getting-started.rst b/docs/stix_primer/getting-started.rst deleted file mode 100644 index 72805360..00000000 --- a/docs/stix_primer/getting-started.rst +++ /dev/null @@ -1,198 +0,0 @@ -Getting Started -=============== - -**Accessing ATT&CK data in python** - -There are several ways to acquire the ATT&CK data in Python. All of them will provide an object -implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. - -This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. - -**Requirements and imports** - -Before installing requirements, we recommend setting up a virtual environment: - -1. Create virtual environment: - - macOS and Linux: `python3 -m venv env` - - Windows: `py -m venv env` -2. Activate the virtual environment: - - macOS and Linux: `source env/bin/activate` - - Windows: `env/Scripts/activate.bat` - -**stix2** - -[stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: - -.. code-block:: python - - from stix2 import Filter - - -However, if you are aiming to extend the ATT&CK dataset with new objects or implement complex workflows, you may need to use the `v20` specifier for some imports. This ensures that the objects use the STIX 2.0 API instead of the STIX 2.1 API. For example: - -.. code-block:: python - - from stix2.v20 import AttackPattern - - -You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). - -**taxii2client** - -[taxii2-client can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-taxii-client#installation). The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of `taxii2client` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. - -If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. - -.. code-block:: python - - from taxii2client.v20 import Collection - - -**Access local content** - -Many users may opt to access the ATT&CK content via a local copy of the STIX data on this repo. This can be advantageous for several reasons: - -- Doesn't require internet access after the initial download -- User can modify the ATT&CK content if desired -- Downloaded copy is static, so updates to the ATT&CK catalog won't cause bugs in automated workflows. User can still manually update by cloning a fresh version of the data - -#### Access via FileSystemSource - -Each domain in this repo is formatted according to the [STIX2 FileSystem spec](https://stix2.readthedocs.io/en/latest/guide/filesystem.html). -Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: - -.. code-block:: python - - from stix2 import FileSystemSource - -src = FileSystemSource('./cti/enterprise-attack') - - -**Access via bundle** - -If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: - -.. code-block:: python - - from stix2 import MemoryStore - - src = MemoryStore() - src.load_from_file("enterprise-attack.json") - - -**Access live content** - -Some users may instead prefer to access "live" ATT&CK content over the internet. This is advantageous for several reasons: - -- Always stays up to date with the evolving ATT&CK catalog -- Doesn't require an initial download of the ATT&CK content, generally requires less setup - -**Access from the ATT&CK TAXII server** - -Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII, the ATT&CK domains are represented as collections with static IDs: - -| domain | collection ID | -|:-------|:--------------| -| `enterprise-attack` | `95ecc380-afe9-11e4-9b6c-751b66dd541e` | -| `mobile-attack` | `2f669986-b40b-4423-b720-4396ca6a462b` | -| `ics-attack` | `02c3ef24-9cd4-48f3-a99f-b74ce24f1d34` | - -You can also get a list of available collection from the server directly: - -.. code-block:: python - - from taxii2client.v20 import Server # only specify v20 if your installed version is >= 2.0.0 - - server = Server("https://cti-taxii.mitre.org/taxii/") - api_root = server.api_roots[0] - # Print name and ID of all ATT&CK domains available as collections - for collection in api_root.collections: - print(collection.title.ljust(20) + collection.id) - - -The following recipe demonstrates how to access the enterprise-attack data from the TAXII server. - -.. code-block:: python - - from stix2 import TAXIICollectionSource - from taxii2client.v20 import Collection # only specify v20 if your installed version is >= 2.0.0 - - collections = { - "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e", - "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b", - "ics-attack": "02c3ef24-9cd4-48f3-a99f-b74ce24f1d34" - } - - collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/") - src = TAXIICollectionSource(collection) - - -For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). - -**Access from Github via requests** - -Users can alternatively access the data from MITRE/CTI using HTTP requests, and load the resulting content into a MemoryStore. -While typically the TAXII method is more desirable for "live" access, this method can be useful if you want to -access data on a branch of the MITRE/CTI repo (the TAXII server only holds the master branch) or in the case of a TAXII server outage. - -.. code-block:: python - - import requests - from stix2 import MemoryStore - - def get_data_from_branch(domain, branch="master"): - """get the ATT&CK STIX data from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'. Branch should typically be master.""" - stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/{branch}/{domain}/{domain}.json").json() - return MemoryStore(stix_data=stix_json["objects"]) - - src = get_data_from_branch("enterprise-attack") - - -**Access a specific version of ATT&CK** - -ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.com/mitre/cti/tags). Tags prefixed with `ATT&CK-v` correspond to ATT&CK versions and tags prefixed with `CAPEC-v` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. - -In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: - -.. code-block:: python - - import requests - from stix2 import MemoryStore - - def get_data_from_version(domain, version): - """get the ATT&CK STIX data for the given version from MITRE/CTI. Domain should be 'enterprise-attack', 'mobile-attack' or 'ics-attack'.""" - stix_json = requests.get(f"https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v{version}/{domain}/{domain}.json").json() - return MemoryStore(stix_data=stix_json["objects"]) - - src = get_data_from_version("enterprise-attack", "5.2") - - -You can get a list of ATT&CK versions programmatically using the github API: - -.. code-block:: python - - import requests - import re - - refToTag = re.compile(r"ATT&CK-v(.*)") - tags = requests.get("https://api.github.com/repos/mitre/cti/git/refs/tags").json() - versions = list(map(lambda tag: refToTag.search(tag["ref"]).groups()[0] , filter(lambda tag: "ATT&CK-v" in tag["ref"], tags))) - # versions = ["1.0", "2.0", ...] - - -**Access multiple domains simultaneously** - -Because ATT&CK is stored in multiple domains (as of this writing, enterprise-attack, mobile-attack and ics-attack), the above methodologies will only allow you to work -with a single domain at a time. While oftentimes the hard separation of domains is advantageous, occasionally it is useful to combine -domains into a single DataStore. Use any of the methods above to acquire the individual datastores, and then use the following approach to combine them into -a single CompositeDataSource: - -.. code-block:: python - - from stix2 import CompositeDataSource - - src = CompositeDataSource() - src.add_data_sources([enterprise_attack_src, mobile_attack_src, ics_attack_src]) - - -You can then use this CompositeDataSource just as you would the DataSource for an individual domain. From 0a47812d756004bb9b819043c571c678ee059c18 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 11:55:06 -0400 Subject: [PATCH 032/159] Update index.rst --- docs/index.rst | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 936b4c50..416f27d9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -29,27 +29,6 @@ other modules in this library under "Additional Modules". :caption: STIX Overview stix_overview/access-attack - stix_overview/stix-recipes - stix_overview/attack_id - stix_overview/by_alias - stix_overview/by_name - stix_overview/deprecated_revoked - stix_overview/getting_related_objects - stix_overview/getting_software - stix_overview/multiple_objects - stix_overview/objects_by_content - stix_overview/objects_by_type - stix_overview/objects_since_date - stix_overview/stix_id - stix_overview/tactics_by_matrix - stix_overview/techniques_by_platform - stix_overview/techniques_by_platform - stix_overview/techniques_by_tactic - stix_overview/techniques_subtechniques - stix_overview/relationships_microlibrary - stix_overview/techniques_by_group_sw - stix_overview/remove_revoked_deprecated - stix_overview/getting_revoked_objects From bff1a554f56be7db62eb2def15f6c6c5b743a108 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 12:39:00 -0400 Subject: [PATCH 033/159] Update access-attack.rst --- docs/stix_primer/access-attack.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index cc1088ea..80f7ffd1 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -196,3 +196,9 @@ a single CompositeDataSource: You can then use this CompositeDataSource just as you would the DataSource for an individual domain. + + +.. toctree:: + :maxdepth: 1 + :caption: Table of Contents + From 1f4d6f3a3eb700eac1edc438fbeb446ab6ff5b00 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 12:55:17 -0400 Subject: [PATCH 034/159] Update access-attack.rst --- docs/stix_primer/access-attack.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index 80f7ffd1..038eba40 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -201,4 +201,26 @@ You can then use this CompositeDataSource just as you would the DataSource for a .. toctree:: :maxdepth: 1 :caption: Table of Contents + + stix-recipes + attack_id + by_alias + by_name + deprecated_revoked + getting_related_objects + getting_software + multiple_objects + objects_by_content + objects_by_type + objects_since_date + stix_id + tactics_by_matrix + techniques_by_platform + techniques_by_platform + techniques_by_tactic + techniques_subtechniques + relationships_microlibrary + techniques_by_group_sw + remove_revoked_deprecated + getting_revoked_objects From 5fd4b4940f308e621a898f5519c85a6d37c5725b Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 13:06:02 -0400 Subject: [PATCH 035/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 416f27d9..6c375c6f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -28,7 +28,7 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: STIX Overview - stix_overview/access-attack + stix_primer/access-attack From 645b0ebdc8c7e9921c0aee3c42499827fb4d6c25 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 13:26:08 -0400 Subject: [PATCH 036/159] added getting started doc --- docs/getting_started.rst | 65 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 docs/getting_started.rst diff --git a/docs/getting_started.rst b/docs/getting_started.rst new file mode 100644 index 00000000..cab010af --- /dev/null +++ b/docs/getting_started.rst @@ -0,0 +1,65 @@ + +**mitreattack-python** + +This repository contains a library of Python tools and utilities for working with ATT&CK data. For more information, +see the [full documentation](https://mitreattack-python.readthedocs.io/) on ReadTheDocs. + +**Install** + +To use this package, install the mitreattack-python library with [pip](https://pip.pypa.io/en/stable/): + +```shell +pip install mitreattack-python +``` + +Note: the library requires [python3](https://www.python.org/). + +**MitreAttackData Library** + +The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides +the ability to query the dataset for objects and their related objects. This is the main content of mitreattack-python; +you can read more about other modules in this library under "Additional Modules". + +**Additional Modules** + +More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. + +| module | description | documentation | +|:------------|:------------|:--------------| +| [navlayers](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/navlayers) | A collection of utilities for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md).| +| [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel) | A collection of utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| +| [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| +| [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/diffStix/README.md).| + + +**Related MITRE Work** + +Go to [this link](https://mitreattack-python.readthedocs.io/en/latest/related_work.html) for related MITRE work. + + +**Contributing** + +To contribute to this project, either through a bug report, feature request, or merge request, +please see the [Contributors Guide](https://github.com/mitre-attack/mitreattack-python/blob/master/docs/CONTRIBUTING.md). + +**Notice** + +Copyright 2023 The MITRE Corporation + +Approved for Public Release; Distribution Unlimited. Case Number 19-0486. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +This project makes use of ATT&CK® + +[ATT&CK Terms of Use](https://attack.mitre.org/resources/terms-of-use/) From e035018ed143f6ceae1e180dfd3805298968d0cc Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 13:48:32 -0400 Subject: [PATCH 037/159] added section docs --- docs/attacktoexcel.rst | 108 +++++++++ docs/collections.rst | 94 ++++++++ docs/diffstix.rst | 119 ++++++++++ docs/navlayers.rst | 521 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 842 insertions(+) create mode 100644 docs/attacktoexcel.rst create mode 100644 docs/collections.rst create mode 100644 docs/diffstix.rst create mode 100644 docs/navlayers.rst diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst new file mode 100644 index 00000000..7aa0f722 --- /dev/null +++ b/docs/attacktoexcel.rst @@ -0,0 +1,108 @@ +# ATT&CK To Excel + +This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. +It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. + +## Usage + +### Command Line + +Print full usage instructions: + +```shell +python3 attackToExcel.py -h +``` + +Example execution: + +```shell +python3 attackToExcel.py +``` + +Build a excel files corresponding to a specific domain and version of ATT&CK: + +```shell +python3 attackToExcel -domain mobile-attack -version v5.0 +``` + +### Module + +Example execution targeting a specific domain and version: + +```python +import mitreattack.attackToExcel.attackToExcel as attackToExcel + +attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") +``` + +## Interfaces + +### attackToExcel + +attackToExcel provides the means by which to convert/extract the ATT&CK STIX data to Excel spreadsheets. A brief +overview of the available methods follows. + +| method name | arguments | usage | +|:------------|:----------|:------| +|get_stix_data|`domain`: the domain of ATT&CK to fetch data from
`version`: optional parameter indicating which version to fetch data from (such as "v8.1"). If omitted retrieves the most recent version of ATT&CK.
`remote`: optional parameter that provides a URL of a remote ATT&CK Workbench instance to grab data from.| Retrieves the ATT&CK STIX data for the specified version and returns it as a MemoryStore object| +|build_dataframes| `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to| Builds a Pandas DataFrame collection as a dictionary, with keys for each type, based on the ATT&CK data provided| +|write_excel| `dataframes`: pandas DataFrame dictionary (generated by build_dataframes)
`domain`: domain of ATT&CK that `dataframes` corresponds to
`version`: optional parameter indicating which version of ATT&CK is in use
`output_dir`: optional parameter specifying output directory| Writes out DataFrame based ATT&CK data to excel files| +|export| `domain`: the domain of ATT&CK to download
`version`: optional parameter specifying which version of ATT&CK to download
`output_dir`: optional parameter specifying output directory| Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets | + +### stixToDf + +stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for +processing. A brief overview of these methods follows. + +| method name | arguments | usage | +|:------------|:----------|:------| +|techniquesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX techniques from the provided data and returns corresponding Pandas DataFrames.| +|tacticsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX tactics from the provided data and returns corresponding Pandas DataFrames.| +|softwareToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX software from the provided data and returns corresponding Pandas DataFrames.| +|groupsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX groups from the provided data and returns corresponding Pandas DataFrames.| +|mitigationsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX mitigations from the provided data and returns corresponding Pandas DataFrames.| +|relationshipsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX relationships from the provided data and returns corresponding Pandas DataFrames.| +|matricesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]`| + +## Spreadsheet format + +The Excel representation of the ATT&CK dataset includes both master spreadsheets, +containing all object types, and individual spreadsheets for each object type. +The individual type spreadsheets break out relationships (e.g procedure examples connecting groups to techniques) +into separate sheets by relationship type, while the master spreadsheet includes all relationship types in a single sheet. +Otherwise, the representation is identical. + +A citations sheet can be used to look up the in-text citations which appear in some fields. +For domains that include multiple matrices, such as Mobile ATT&CK, each matrix gets its own named sheet. +Unlike the STIX dataset, objects that have been revoked or deprecated are not included in the spreadsheets. + +## Accessing the Pandas DataFrames + +Internally, attackToExcel stores the parsed STIX data as [Pandas](https://pandas.pydata.org/) DataFrames. +These can be retrieved for use in data analysis. + +Example of accessing [Pandas](https://pandas.pydata.org/) DataFrames: + +```python +import mitreattack.attackToExcel.attackToExcel as attackToExcel +import mitreattack.attackToExcel.stixToDf as stixToDf + +# download and parse ATT&CK STIX data +attackdata = attackToExcel.get_stix_data("enterprise-attack") +techniques_data = stixToDf.techniquesToDf(attackdata, "enterprise-attack") + +# show T1102 and sub-techniques of T1102 +techniques_df = techniques_data["techniques"] +print(techniques_df[techniques_df["ID"].str.contains("T1102")]["name"]) +# 512 Web Service +# 38 Web Service: Bidirectional Communication +# 121 Web Service: Dead Drop Resolver +# 323 Web Service: One-Way Communication +# Name: name, dtype: object + +# show citation data for LOLBAS Wmic reference +citations_df = techniques_data["citations"] +print(citations_df[citations_df["reference"].str.contains("LOLBAS Wmic")]) +# reference citation url +# 1010 LOLBAS Wmic LOLBAS. (n.d.). Wmic.exe. Retrieved July 31, 2... https://lolbas-project.github.io/lolbas/Binari... +``` \ No newline at end of file diff --git a/docs/collections.rst b/docs/collections.rst new file mode 100644 index 00000000..3fd2ec44 --- /dev/null +++ b/docs/collections.rst @@ -0,0 +1,94 @@ +# collections + +This folder contains modules and scripts for working with ATT&CK collections. +Collections are sets of ATT&CK STIX objects, grouped for user convienence. +For more information about ATT&CK collections, see the corresponding +[ATT&CK documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). + +## Collections Scripts + +| script | description | +|:-------|:------------| +|[index_to_markdown](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/index_to_markdown.py)| Provides a means by which to convert a [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| +|[collection_to_index](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/collection_to_index.py)| Provides a means by which to convert a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) into a easy-to-share [index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes). More information can be found in the corresponding [section](#collection_to_index.py) below.| +|[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| + +### index_to_markdown.py + +`index_to_markdown.py` provides the `IndexToMarkdown` class, which provides a way to transform an existing +[collection index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) +into a markdown file for easy of use and reference. +The `IndexToMarkdown` class is very simple, and provides a single method, `index_to_markdown`, +which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown. +An example of how to use the class, and method, can be found below. + +#### Example Usage + +```python +import json +from mitreattack.collections import IndexToMarkdown + +with open('collection_index.json', 'r') as input_file: + with open('collection_index.md', 'w') as output_file: + input_index = json.load(input_file) + generated_md = IndexToMarkdown.index_to_markdown(input_index) # Convert index to markdown + output_file.write(generated_md) +print(generated_md) +``` + +### collection_to_index.py + +`collection_to_index.py` provides the `CollectionToIndex` class, which proves a means by which to summarize existing +[collections](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) +into a single [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) file. +The `CollectionToIndex` class contains the `generate_index` function, which when provided with a name, description, root url (pointing to where the raw collections are stored), +and a list of either files, folders, or already loaded bundles in the form of dictionaries, will create a summarizing index. + +#### Example Usage + +```python +import json +from mitreattack.collections import CollectionToIndex + +output_indexA = CollectionToIndex.generate_index(name='example', description='example index', + root_url='www.example.com', + files=['/path/to/collection1.json', '/path/to/collection2.json'], + folders=None, sets=None) +output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index', + root_url='www.example.com', + files=None, folders=['/path/to/folder/with/collections'], sets=None) +with open('path/to/bundle/bundleC.json', 'r') as f: + data = json.load(f) +output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index', + root_url='www.example.com', + files=None, folders=None, sets=[data]) +print(output_indexA) +print(output_indexB) +print(output_indexC) +``` + +### stix_to_collection.py + +`stix_to_collection.py` provides the `STIXToCollection` class, which proves a means by which to convert +existing stix bundles into ones containing a +[collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) object. +The `STIXToCollection` class contains the `stix_to_collection` function, which when provided with a starter bundle, +a name, a version, and an optional description, will output a modified bundle that contains a summary collection object. + +#### Example Usage + +```python +import json +from mitreattack.collections import STIXToCollection + +with open('path/to/bundle/bundle2_0.json', 'r') as f: + data = json.load(f) +output_bundleA = STIXToCollection.stix_to_collection(bundle=data, name='collectionA', version='9.1', description="demo bundle (2.0)") + +with open('path/to/bundle/bundle2_1.json', 'r') as f: + data = json.load(f) +output_bundleB = STIXToCollection.stix_to_collection(bundle=data, name='collectionB', version='9.0', description="demo bundle (2.1)") + +print(output_bundleA) +print(output_bundleB) +``` \ No newline at end of file diff --git a/docs/diffstix.rst b/docs/diffstix.rst new file mode 100644 index 00000000..e5324882 --- /dev/null +++ b/docs/diffstix.rst @@ -0,0 +1,119 @@ +# Diff Stix + +This folder contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers +reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. +Run `diff_stix -h` for full usage instructions. + +## Usage + +### Command Line + +Print full usage instructions: + +```shell +# You must run `pip install mitreattack-python` in order to access the diff_stix command +diff_stix --help +usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] + [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] + +Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. + +options: + -h, --help show this help message and exit + --old OLD Directory to load old STIX data from. + --new NEW Directory to load new STIX data from. + --domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...] + Which domains to report on. Choices (and defaults) are enterprise-attack, mobile-attack, ics-attack + --markdown-file MARKDOWN_FILE + Create a markdown file reporting changes. + --html-file HTML_FILE + Create HTML page from markdown content. + --html-file-detailed HTML_FILE_DETAILED + Create an HTML file reporting detailed changes. + --json-file JSON_FILE + Create a JSON file reporting changes. + --layers [LAYERS ...] + Create layer files showing changes in each domain expected order of filenames is 'enterprise', 'mobile', 'ics', 'pre attack'. If values are unspecified, defaults to output/January_2023_Updates_Enterprise.json, + output/January_2023_Updates_Mobile.json, output/January_2023_Updates_ICS.json, output/January_2023_Updates_Pre.json + --site_prefix SITE_PREFIX + Prefix links in markdown output, e.g. [prefix]/techniques/T1484 + --unchanged Show objects without changes in the markdown output + --use-mitre-cti Use content from the MITRE CTI repo for the -old data + --show-key Add a key explaining the change types to the markdown + --contributors Show new contributors between releases + --no-contributors Do not show new contributors between releases + -v, --verbose Print status messages +``` + +Example execution: + +```shell +diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ +``` + +## Changelog JSON format + +The changelog helper script has the option to output a JSON file with detailed differences between ATT&CK releases. +This is the overall structure you can expect to find in the file. +A brief explanation of key pieces can be found below. + +```JSON +{ + "enterprise-attack": { + "techniques": { + "additions": [], + "major_version_changes": [], + "minor_version_changes": [], + "other_version_changes": [], + "patches": [], + "revocations": [], + "deprecations": [], + "deletions": [], + }, + "software": {}, + "groups": {}, + "campaigns": {}, + "mitigations": {}, + "datasources": {}, + "datacomponents": {} + }, + "mobile-attack": {}, + "ics-attack": {}, + "new-contributors": [ + "Contributor A", + "Contributor B", + "Contributor C" + ] +} +``` + +* The top-level objects include information about specific domains as well as `new-contributors`, which are only found in the newer ATT&CK release. +* For domain changes, they are broken down by object type, e.g. `techniques` or `mitigations`. +* The following table helps break down the change types that are currently tracked. + +| Field | Type | Description | +|-------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------| +| `additions` | array[object] | ATT&CK objects which are only present in the new STIX data. | +| `major_version_changes` | array[object] | ATT&CK objects that have a major version change. (e.g. 1.0 → 2.0). | +| `minor_version_changes` | array[object] | ATT&CK objects that have a minor version change. (e.g. 1.0 → 1.1). | +| `other_version_changes` | array[object] | ATT&CK objects that have a version change of any other kind. (e.g. 1.0 → 1.3). These are unintended, but can be found in previous releases. | +| `patches` | array[object] | ATT&CK objects that have been patched while keeping the version the same. | +| `revocations` | array[object] | ATT&CK objects which are revoked by a different object. | +| `deprecations` | array[object] | ATT&CK objects which are deprecated and no longer in use, and not replaced. | +| `deletions` | array[object] | ATT&CK objects which are no longer found in the STIX data. This should almost never happen. | + +### Changed Objects + +The bulk of the changelog file consists of lists of JSON objects. +If you are familiar with reading the STIX format, they may look famliar, yet a little "off". +That is because there are a few fields that have been added in some cases depending on what section they appear in. +For example, objects that are brand new do not have `previous_version` available to them. +The following table lists the extra fields that can be found in objects in the changelog. + +| Field | Required | Type | Description | +|----------------------------|----------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `changelog_mitigations` | false | object | Three lists for `shared`, `new`, and `dropped` for Mitigations that are related to a Technique between versions. | +| `changelog_detections` | false | object | Three lists for `shared`, `new`, and `dropped` for Detections that are related to a Technique between versions. || `description_change_table` | false | string | HTML rendering of a table that displays the differences between descriptions for an ATT&CK object. | +| `detailed_diff` | false | string | A python DeepDiff object that has been JSON serialized which represents STIX changes for an ATT&CK object between releases. | +| `previous_version` | false | string | If the object existed in the previous release, then it denotes the version the object was in the previous release. | +| `version_change` | false | string | If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' | \ No newline at end of file diff --git a/docs/navlayers.rst b/docs/navlayers.rst new file mode 100644 index 00000000..ae507a80 --- /dev/null +++ b/docs/navlayers.rst @@ -0,0 +1,521 @@ +# navlayers + +This folder contains modules and scripts for working with ATT&CK Navigator layers. +ATT&CK Navigator Layers are a set of annotations overlaid on top of the ATT&CK Matrix. +For more about ATT&CK Navigator layers, visit the ATT&CK Navigator repository. +The core module allows users to load, validate, manipulate, and save ATT&CK layers. +A brief overview of the components can be found below. +All scripts adhere to the MITRE ATT&CK Navigator Layer file format, +[version 4.3](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_3.md), +but will accept legacy [version 3.0](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv3.md) +and version 4.X layers, upgrading them to version 4.3. + +## Core Modules + +| script | description | +|:-------|:------------| +| [filter](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/filter.py) | Implements a basic [filter object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#filter-object-properties). | +| [gradient](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/gradient.py) | Implements a basic [gradient object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#gradient-object-properties). | +| [layer](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/layer.py) | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | +| [layout](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/layout.py) | Implements a basic [layout object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#layout-object-properties). | +| [legenditem](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/legenditem.py) | Implements a basic [legenditem object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#legenditem-object-properties). | +| [metadata](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/metadata.py) | Implements a basic [metadata object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#metadata-object-properties). | +| [technique](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/technique.py) | Implements a basic [technique object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#technique-object-properties). | +| [versions](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/versions.py) | Implements a basic [versions object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#versions-object-properties).| + +### Manipulator Scripts + +| script | description | +|:-------|:------------| +| [layerops](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/manipulators/layerops.py) | Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. | + +### Exporter Scripts + +| script | description | +|:-------|:------------| +| [to_excel](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_excel.py) | Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. | +| [to_svg](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_svg.py) | Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export.| + +### Generator Scripts + +| script | description | +|:-------|:------------| +| [overview_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/overview_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | +| [usage_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/usage_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | +| [sum_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/sum_generator.py)| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | + +### Utility Modules + +| script | description | +|:-------|:------------| +| [excel_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/excel_templates.py) | Provides a means by which to convert a matrix into a clean excel matrix template. | +| [matrix_gen](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/matrix_gen.py) | Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). | +| [svg_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_templates.py) | Provides a means by which to convert a layer file into a marked up svg file. | +| [svg_objects](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_objects.py) | Provides raw templates and supporting functionality for generating svg objects. | + +### Command Line Tools + +| script | description | +|:-------|:------------| +| [layerExporter_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerExporter_cli.py) | A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. | +| [layerGenerator_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerGenerator_cli.py) | A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. | + +## Layer + +The `Layer` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. +It is the primary interface through which other Layer-related classes defined in the core module should be used. +The Layer class API and a usage example are below. +The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. + +| method [x = Layer()]| description | +|:-------|:------------| +| `x.from_str(_input_)` | Loads an ATT&CK layer from a string representation of a json layer. | +| `x.from_dict(_input_)` | Loads an ATT&CK layer from a dictionary. | +| `x.from_file(_filepath_)` | Loads an ATT&CK layer from a file location specified by the _filepath_. | +| `x.to_file(_filepath_)` | Saves the current state of the loaded ATT&CK layer to a json file denoted by the _filepath_. | +| `x.to_dict()` | Returns a representation of the current ATT&CK layer object as a dictionary. | +| `x.to_str()` | Returns a representation of the current ATT&CK layer object as a string representation of a dictionary. | + +Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found +[here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/README.md). + +### Example Usage + +```python +example_layer3_dict = { + "name": "example layer", + "version": "3.0", + "domain": "mitre-enterprise" +} + +example_layer4_dict = { + "name": "layer v4.3 example", + "versions" : { + "attack": "8", + "layer" : "4.3", + "navigator": "4.4.4" + }, + "domain": "enterprise-attack" +} + +example_layer_location = "/path/to/layer/file.json" +example_layer_out_location = "/path/to/new/layer/file.json" + +from mitreattack.navlayers.core import Layer + +layer1 = Layer(example_layer3_dict) # Create a new layer and load existing data +layer1.to_file(example_layer_out_location) # Write out the loaded layer to the specified file + +layer2 = Layer() # Create a new layer object +layer2.from_dict(example_layer4_dict) # Load layer data into existing layer object +print(layer2.to_dict()) # Retrieve the loaded layer's data as a dictionary, and print it + +layer3 = Layer() # Create a new layer object +layer3.from_file(example_layer_location) # Load layer data from a file into existing layer object +``` + +### layerops.py + +`Layerops.py` provides the `LayerOps` class, which is a way to combine layer files in an automated way, using user defined lambda functions. +Each LayerOps instance, when created, ingests the provided lambda functions, and stores them for use. +An existing `LayerOps` class can be used to combine layer files according to the initialized lambda using the process method. +The breakdown of this two step process is documented in the table below, while examples of both the list and dictionary modes of operation can be found below. + +#### LayerOps() + +```python +x = LayerOps(score=score, comment=comment, enabled=enabled, colors=colors, metadata=metadata, name=name, desc=desc, default_values=default_values) +``` + +Each of the _inputs_ takes a lambda function that will be used to combine technique object fields matching the parameter. +The one exception to this is _default_values_, which is an optional dictionary argument containing default values +to provide the lambda functions if techniques of the combined layers are missing them. + +##### .process() Method + +```python +x.process(data, default_values=default_values) +``` + +The process method applies the lambda functions stored during initialization to the layer objects in _data_. +_data_ must be either a list or a dictionary of Layer objects, and is expected to match the format of the lambda equations provided during initialization. +`default_values` is an optional dictionary argument that overrides the currently stored default values with new ones for this specific processing operation. + +#### Example Usage + +```python +from mitreattack.navlayers.manipulators.layerops import LayerOps +from mitreattack.navlayers.core.layer import Layer + +demo = Layer() +demo.from_file("C:\Users\attack\Downloads\layer.json") +demo2 = Layer() +demo2.from_file("C:\Users\attack\Downloads\layer2.json") +demo3 = Layer() +demo3.from_file("C:\Users\attack\Downloads\layer3.json") + +# Example 1) Build a LayerOps object that takes a list and averages scores across the layers +lo = LayerOps(score=lambda x: sum(x) / len(x), + name=lambda x: x[1], + desc=lambda x: "This is an list example") # Build LayerOps object +out_layer = lo.process([demo, demo2]) # Trigger processing on a list of demo and demo2 layers +out_layer.to_file("C:\demo_layer1.json") # Save averaged layer to file +out_layer2 = lo.process([demo, demo2, demo3]) # Trigger processing on a list of demo, demo2, demo3 +visual_aid = out_layer2.to_dict() # Retrieve dictionary representation of processed layer + +# Example 2) Build a LayerOps object that takes a dictionary and averages scores across the layers +lo2 = LayerOps(score=lambda x: sum([x[y] for y in x]) / len([x[y] for y in x]), + colors=lambda x: x['b'], + desc=lambda x: "This is a dict example") # Build LayerOps object, with lambda +out_layer3 = lo2.process({'a': demo, 'b': demo2}) # Trigger processing on a dictionary of demo and demo2 +dict_layer = out_layer3.to_dict() # Retrieve dictionary representation of processed layer +print(dict_layer) # Display retrieved dictionary +out_layer4 = lo2.process({'a': demo, 'b': demo2, 'c': demo3})# Trigger processing on a dictionary of demo, demo2, demo3 +out_layer4.to_file("C:\demo_layer4.json") # Save averaged layer to file + +# Example 3) Build a LayerOps object that takes a single element dictionary and inverts the score +lo3 = LayerOps(score=lambda x: 100 - x['a'], + desc= lambda x: "This is a simple example") # Build LayerOps object to invert score (0-100 scale) +out_layer5 = lo3.process({'a': demo}) # Trigger processing on dictionary of demo +print(out_layer5.to_dict()) # Display processed layer in dictionary form +out_layer5.to_file("C:\demo_layer5.json") # Save inverted score layer to file + +# Example 4) Build a LayerOps object that combines the comments from elements in the list, with custom defaults +lo4 = LayerOps(score=lambda x: '; '.join(x), + default_values= { + "comment": "This was an example of new default values" + }, + desc= lambda x: "This is a defaults example") # Build LayerOps object to combine descriptions, defaults +out_layer6 = lo4.process([demo2, demo3]) # Trigger processing on a list of demo2 and demo0 +out_layer6.to_file("C:\demo_layer6.json") # Save combined comment layer to file +``` + +## to_excel.py + +`to_excel.py` provides the `ToExcel` class, which is a way to export an existing layer file as an Excel spreadsheet. +The `ToExcel` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. +Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or retrieving data from an ATT&CK Workbench instance. + +### ToExcel() + +```python +x = ToExcel(domain='enterprise', source='taxii', resource=None) +``` + +The `ToExcel` constructor takes domain, server, and resource arguments during instantiation. +The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. +The source argument tells the matrix generation tool which data source to use when building the matrix. +`taxii` indicates that the tool should utilize the official ATT&CK Taxii Server (`cti-taxii`) when building the matrix, +while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that +it should utilize a remote ATT&CK Workbench instance. +The `resource` argument is only required if the source is set to `local`, in which case it should be a path +to a local stix bundle, or if the source is set to `remote`, in which case it should be the url of a ATT&CK workbench instance. + +### .to_xlsx() Method + +```python +x.to_xlsx(layerInit=layer, filepath="layer.xlsx") +``` + +The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. + +#### Example Usage + +```python +from mitreattack.navlayers import Layer +from mitreattack.navlayers import ToExcel + +lay = Layer() +lay.from_file("path/to/layer/file.json") +# Using taxii server for template +t = ToExcel(domain=lay.layer.domain, source='taxii') +t.to_xlsx(layerInit=lay, filepath="demo.xlsx") +# Using local stix data for template +t2 = ToExcel(domain='mobile', source='local', resource='path/to/local/stix.json') +t2.to_xlsx(layerInit=lay, filepath="demo2.xlsx") +# Using remote ATT&CK Workbench instance for template +workbench_url = 'localhost:3000' +t3 = ToExcel(domain='ics', source='remote', resource=workbench_url) +``` + +## to_svg.py + +`to_svg.py` provides the `ToSvg` class, which is a way to export an existing layer file as an SVG image file. +The `ToSvg` class, like the `ToExcel` class, has an optional parameter for the initialization function, +that tells the exporter what data source to use when building the output matrix. +Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or utilizing a remote ATT&CK Workbench instance. + +### ToSvg() + +```python +x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) +``` + +The `ToSvg` constructor, just like the `ToExcel` constructor, takes domain, server, and resource arguments during instantiation. +The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. +The source argument tells the matrix generation tool which data source to use when building the matrix. +`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, +while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. +The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, +or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +The `config` parameter is an optional `SVGConfig` object that can be used to configure the export as desired. +If not provided, the configuration for the export will be set to default values. + +### SVGConfig() + +```python +y = SVGConfig(width=8.5, height=11, headerHeight=1, unit="in", showSubtechniques="expanded", + font="sans-serif", tableBorderColor="#6B7279", showHeader=True, legendDocked=True, + legendX=0, legendY=0, legendWidth=2, legendHeight=1, showLegend=True, showFilters=True, + showAbout=True, showDomain=True, border=0.104) +``` + +The `SVGConfig` object is used to configure how an SVG export behaves. +The defaults for each of the available values can be found in the declaration above, and a brief explanation for each field is included in the table below. +The config object should be provided to the `ToSvg` object during instantiation, but if values need to be updated on the fly, +the currently loaded configuration can be interacted with at `ToSvg().config`. +The configuration can also be populated from a json file using the `.load_from_file(filename="path/to/file.json")` method, +or stored to one using the `.save_to_file(filename="path/to/file.json)` method. + +| attribute| description | type | default value | +|:-------|:------------|:------------|:------------| +| width | Desired SVG width | number | 8.5 | +| height | Desired SVG height | number | 11 | +| headerHeight | Desired Header Block height | number | 1 | +| unit | SVG measurement units (qualifies width, height, etc.) - "in", "cm", "px", "em", or "pt"| string | "in" | +| showSubtechniques | Display form for subtechniques - "all", "expanded" (decided by layer), or "none" | string | "expanded" | +| font | What font style to use - "serif", "sans-serif", or "monospace" | string | "sans-serif" | +| tableBorderColor | Hex color to use for the technique borders | string | "#6B7279" | +| showHeader | Whether or not to show Header Blocks | bool | True | +| legendDocked | Whether or not the legend should be docked | bool | True | +| legendX | Where to place the legend on the x axis if not docked | number | 0 | +| legendY | Where to place the legend on the y axis if not docked | number | 1 | +| legendWidth | Width of the legend if not docked | number | 2 | +| legendHeight | Height of the legend if not docked | number | 1 | +| showLegend | Whether or not to show the legend | bool | True | +| showFilters | Whether or not to show the Filter Header Block | bool | True | +| showDomain | Whether or not to show the Domain and Version Header Block | bool | True | +| showAbout | Whether or not to show the About Header Block | bool | True | +| border | What default border width to use | number | 0.104 | + +### .to_svg() Method + +```python +x.to_svg(layerInit=layer, filepath="layer.svg") +``` + +The `to_svg` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. + +#### Example Usage + +```python +from mitreattack.navlayers import Layer +from mitreattack.navlayers import ToSvg, SVGConfig + +lay = Layer() +lay.from_file("path/to/layer/file.json") +# Using taxii server for template +t = ToSvg(domain=lay.layer.domain, source='taxii') +t.to_svg(layerInit=lay, filepath="demo.svg") +#Using local stix data for template + +conf = SVGConfig() +conf.load_from_file(filename="path/to/poster/config.json") + +t2 = ToSvg(domain='mobile', source='local', resource='path/to/local/stix.json', config=conf) +t2.to_svg(layerInit=lay, filepath="demo2.svg") + +workbench_url = "localhost:3000" +t3 = ToSvg(domain='enterprise', source='remote', resource=workbench_url, config=conf) +t3.to_svg(layerInit=lay, filepath="demo3.svg") +``` + +## overview_generator.py + +`overview_generator.py` provides the `OverviewLayerGenerator` class, which is designed to allow users to +generate an ATT&CK layer that, on a per technique basis, has a score that corresponds to all instances +of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. + +### OverviewLayerGenerator() + +```python +x = OverviewLayerGenerator(source='taxii', domain='enterprise', resource=None) +``` + +The initialization function for `OverviewLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where +to retrieve data from (taxii server etc.). +The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. +The source argument tells the matrix generation tool which data source to use when building the matrix. +`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, +while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. +The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, +or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +If not provided, the configuration for the generator will be set to default values. + +### .generate_layer() + +```python +x.generate_layer(obj_type=object_type_name) +``` + +The `generate_layer` function generates a layer, customized to the input `object_type_name`. +Valid values include `group`, `mitigation`, `software`, and `datasource`. + +## usage_generator.py + +`usage_ generator.py` provides the `UsageLayerGenerator` class, which is designed to allow users to +generate an ATT&CK layer that scores any relevant techniques that a given input ATT&CK object has. +These objects can be any `group`, `software`, `mitigation`, or `data component`, +and can be referenced by ID or by any alias when provided to the generator. + +### UsageLayerGenerator() + +```python +x = UsageLayerGenerator(source='taxii', domain='enterprise', resource=None) +``` + +The initialization function for `UsageLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where +to retrieve data from (taxii server etc.). +The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. +The source argument tells the matrix generation tool which data source to use when building the matrix. +`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, +while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. +The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, +or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +If not provided, the configuration for the generator will be set to default values. + +### .generate_layer() + +```python +x.generate_layer(match=object_identifier) +``` + +The `generate_layer` function generates a layer, customized to the input `object_identifier`. +Valid values include `ATT&CK ID`, `name`, or any known `alias` for `group`, `mitigation`, `software`, and `data component` objects within the selected ATT&CK data. + +#### Example Usage + +```python +from mitreattack.navlayers import UsageLayerGenerator + +handle = UsageLayerGenerator(source='taxii', domain='enterprise') + +layer1 = handle.generate_layer(match='G0018') +layer2 = handle.generate_layer(match='Adups') +``` + +## sum_generator.py + +`sum_generator.py` provides the `SumLayerGenerator` class, which is designed to allow users to +generate a collection of ATT&CK layers that, on a per technique basis, have a score that corresponds to all instances +of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. +Each one of the generated layers will correspond to a single instance of the specified ATT&CK object type. + +### SumLayerGenerator() + +```python +x = SumLayerGenerator(source='taxii', domain='enterprise', resource=None) +``` + +The initialization function for `SumGeneratorLayer`, like `ToSVG` and `ToExcel`, requires the specification of where +to retrieve data from (taxii server etc.). +The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. +The source argument tells the matrix generation tool which data source to use when building the matrix. +`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, +while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. +The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, +or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +If not provided, the configuration for the generator will be set to default values. + +### .generate_layer() + +```python +x.generate_layer(layers_type=object_type_name) +``` + +The `generate_layer` function generates a collection of layers, each customized to one instance of the input `object_type_name`. +Valid types include `group`, `mitigation`, `software`, and `datasource`. + +## layerExporter_cli.py + +This command line tool allows users to convert a [navigator](https://github.com/mitre-attack/attack-navigator) +layer file to either an svg image or excel file using the functionality provided by the navlayers module. +Details about the SVG configuration json mentioned below can be found in the +[SVGConfig](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md#svgconfig) +entry within the navlayers module documentation. + +```commandline +C:\Users\attack>layerExporter_cli -h +usage: layerExporter_cli [-h] -m {svg,excel} [-s {taxii,local,remote}] + [--resource RESOURCE] -o OUTPUT [OUTPUT ...] + [-l LOAD_SETTINGS] [-d WIDTH HEIGHT] + input [input ...] + +Export an ATT&CK Navigator layer as a svg image or excel file + +positional arguments: + input Path(s) to the file to export + +optional arguments: + -h, --help show this help message and exit + -m {svg,excel}, --mode {svg,excel} + The form to export the layers in + -s {taxii,local,remote}, --source {taxii,local,remote} + What source to utilize when building the matrix + --resource RESOURCE Path to the local resource if --source=local, or url + of an ATT&CK Workbench instance if --source=remote + -o OUTPUT [OUTPUT ...], --output OUTPUT [OUTPUT ...] + Path(s) to the exported svg/xlsx file + -l LOAD_SETTINGS, --load_settings LOAD_SETTINGS + [SVG Only] Path to a SVG configuration json to use + when rendering + -d WIDTH HEIGHT, --size WIDTH HEIGHT + [SVG Only] X and Y size values (in inches) for SVG + export (use -l for other settings) + +C:\Users\attack>layerExporter_cli -m svg -s taxii -l settings/config.json -o output/svg1.json output/svg2.json files/layer1.json files/layer2.json +``` + +## layerGenerator_cli.py + +This command line tool allows users to generate [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) +layer files from either a specific group, software, or mitigation. Alternatively, users can generate a layer file with a +mapping to all associated groups, software, or mitigations across the techniques within ATT&CK. + +```commandline +C:\Users\attack>layerGenerator_cli -h +usage: layerGenerator_cli [-h] + (--overview-type {group,software,mitigation,datasource} | --mapped-to MAPPED_TO | --batch-type {group,software,mitigation,datasource}) + [-o OUTPUT] [--domain {enterprise,mobile,ics}] + [--source {taxii,local,remote}] + [--resource RESOURCE] + +Generate an ATT&CK Navigator layer + +optional arguments: + -h, --help show this help message and exit + --overview-type {group,software,mitigation,datasource} + Output a layer file where the target type is + summarized across the entire dataset. + --mapped-to MAPPED_TO + Output layer file with techniques mapped to the given + group, software, mitigation, or data component. Argument + can be name, associated group/software, or ATT&CK ID. + --batch-type {group,software,mitigation,datasource} + Output a collection of layer files to the specified + folder, each one representing a different instance of + the target type. + -o OUTPUT, --output OUTPUT + Path to the output layer file/directory + --domain {enterprise,mobile,ics} + Which domain to build off of + --source {taxii,local,remote} + What source to utilize when building the layer files + --resource RESOURCE Path to the local resource if --source=local, or url + of an ATT&CK Workbench instance if --source=remote + +C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --mapped-to S0065 --output generated_layer.json +C:\Users\attack>layerGenerator_cli --domain mobile --source taxii --overview-type mitigation --output generated_layer2.json +C:\Users\attack>layerGenerator_cli --domain ics --source taxii --batch-type software +C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --overview-type datasource --output generated_layer3.json +``` \ No newline at end of file From a69b2c046f9edaf977a1dc1ead74d32c3e7543d9 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 13:50:07 -0400 Subject: [PATCH 038/159] Update index.rst --- docs/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/index.rst b/docs/index.rst index 6c375c6f..07592067 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -11,6 +11,7 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: Overview + getting_started installation related_work contributing From 25491378c51f3b6f724e42dc41375fc385f11875 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 14:54:22 -0400 Subject: [PATCH 039/159] adding more --- docs/collections.rst | 106 ++++++++++++++++++++++--------------------- docs/navlayers.rst | 34 +++++++------- 2 files changed, 72 insertions(+), 68 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index 3fd2ec44..8d40efcc 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -1,11 +1,11 @@ -# collections +**collections** This folder contains modules and scripts for working with ATT&CK collections. Collections are sets of ATT&CK STIX objects, grouped for user convienence. For more information about ATT&CK collections, see the corresponding [ATT&CK documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). -## Collections Scripts +**Collections Scripts** | script | description | |:-------|:------------| @@ -13,7 +13,7 @@ For more information about ATT&CK collections, see the corresponding |[collection_to_index](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/collection_to_index.py)| Provides a means by which to convert a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) into a easy-to-share [index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes). More information can be found in the corresponding [section](#collection_to_index.py) below.| |[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| -### index_to_markdown.py +**index_to_markdown.py** `index_to_markdown.py` provides the `IndexToMarkdown` class, which provides a way to transform an existing [collection index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) @@ -22,21 +22,22 @@ The `IndexToMarkdown` class is very simple, and provides a single method, `index which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown. An example of how to use the class, and method, can be found below. -#### Example Usage +**Example Usage** -```python -import json -from mitreattack.collections import IndexToMarkdown +.. code-block:: python + + import json + from mitreattack.collections import IndexToMarkdown -with open('collection_index.json', 'r') as input_file: - with open('collection_index.md', 'w') as output_file: - input_index = json.load(input_file) - generated_md = IndexToMarkdown.index_to_markdown(input_index) # Convert index to markdown - output_file.write(generated_md) -print(generated_md) -``` + with open('collection_index.json', 'r') as input_file: + with open('collection_index.md', 'w') as output_file: + input_index = json.load(input_file) + generated_md = IndexToMarkdown.index_to_markdown(input_index) # Convert index to markdown + output_file.write(generated_md) + print(generated_md) -### collection_to_index.py + +**collection_to_index.py** `collection_to_index.py` provides the `CollectionToIndex` class, which proves a means by which to summarize existing [collections](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) @@ -44,30 +45,31 @@ into a single [collection index](https://github.com/center-for-threat-informed-d The `CollectionToIndex` class contains the `generate_index` function, which when provided with a name, description, root url (pointing to where the raw collections are stored), and a list of either files, folders, or already loaded bundles in the form of dictionaries, will create a summarizing index. -#### Example Usage - -```python -import json -from mitreattack.collections import CollectionToIndex - -output_indexA = CollectionToIndex.generate_index(name='example', description='example index', - root_url='www.example.com', - files=['/path/to/collection1.json', '/path/to/collection2.json'], - folders=None, sets=None) -output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index', - root_url='www.example.com', - files=None, folders=['/path/to/folder/with/collections'], sets=None) -with open('path/to/bundle/bundleC.json', 'r') as f: - data = json.load(f) -output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index', - root_url='www.example.com', - files=None, folders=None, sets=[data]) -print(output_indexA) -print(output_indexB) -print(output_indexC) -``` - -### stix_to_collection.py +**Example Usage** + + .. code-block:: python + + import json + from mitreattack.collections import CollectionToIndex + + output_indexA = CollectionToIndex.generate_index(name='example', description='example index', + root_url='www.example.com', + files=['/path/to/collection1.json', '/path/to/collection2.json'], + folders=None, sets=None) + output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index', + root_url='www.example.com', + files=None, folders=['/path/to/folder/with/collections'], sets=None) + with open('path/to/bundle/bundleC.json', 'r') as f: + data = json.load(f) + output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index', + root_url='www.example.com', + files=None, folders=None, sets=[data]) + print(output_indexA) + print(output_indexB) + print(output_indexC) + + +**stix_to_collection.py** `stix_to_collection.py` provides the `STIXToCollection` class, which proves a means by which to convert existing stix bundles into ones containing a @@ -75,20 +77,20 @@ existing stix bundles into ones containing a The `STIXToCollection` class contains the `stix_to_collection` function, which when provided with a starter bundle, a name, a version, and an optional description, will output a modified bundle that contains a summary collection object. -#### Example Usage +**Example Usage** + +.. code-block:: python -```python -import json -from mitreattack.collections import STIXToCollection + import json + from mitreattack.collections import STIXToCollection -with open('path/to/bundle/bundle2_0.json', 'r') as f: - data = json.load(f) -output_bundleA = STIXToCollection.stix_to_collection(bundle=data, name='collectionA', version='9.1', description="demo bundle (2.0)") + with open('path/to/bundle/bundle2_0.json', 'r') as f: + data = json.load(f) + output_bundleA = STIXToCollection.stix_to_collection(bundle=data, name='collectionA', version='9.1', description="demo bundle (2.0)") -with open('path/to/bundle/bundle2_1.json', 'r') as f: - data = json.load(f) -output_bundleB = STIXToCollection.stix_to_collection(bundle=data, name='collectionB', version='9.0', description="demo bundle (2.1)") + with open('path/to/bundle/bundle2_1.json', 'r') as f: + data = json.load(f) + output_bundleB = STIXToCollection.stix_to_collection(bundle=data, name='collectionB', version='9.0', description="demo bundle (2.1)") -print(output_bundleA) -print(output_bundleB) -``` \ No newline at end of file + print(output_bundleA) + print(output_bundleB) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index ae507a80..3c8440cd 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -386,23 +386,23 @@ If not provided, the configuration for the generator will be set to default valu ### .generate_layer() -```python -x.generate_layer(match=object_identifier) -``` +.. code-block:: python + + x.generate_layer(match=object_identifier) + The `generate_layer` function generates a layer, customized to the input `object_identifier`. Valid values include `ATT&CK ID`, `name`, or any known `alias` for `group`, `mitigation`, `software`, and `data component` objects within the selected ATT&CK data. -#### Example Usage +.. code-block:: python -```python -from mitreattack.navlayers import UsageLayerGenerator + from mitreattack.navlayers import UsageLayerGenerator -handle = UsageLayerGenerator(source='taxii', domain='enterprise') + handle = UsageLayerGenerator(source='taxii', domain='enterprise') + + layer1 = handle.generate_layer(match='G0018') + layer2 = handle.generate_layer(match='Adups') -layer1 = handle.generate_layer(match='G0018') -layer2 = handle.generate_layer(match='Adups') -``` ## sum_generator.py @@ -413,9 +413,10 @@ Each one of the generated layers will correspond to a single instance of the spe ### SumLayerGenerator() -```python -x = SumLayerGenerator(source='taxii', domain='enterprise', resource=None) -``` +.. code-block:: python + + x = SumLayerGenerator(source='taxii', domain='enterprise', resource=None) + The initialization function for `SumGeneratorLayer`, like `ToSVG` and `ToExcel`, requires the specification of where to retrieve data from (taxii server etc.). @@ -429,9 +430,10 @@ If not provided, the configuration for the generator will be set to default valu ### .generate_layer() -```python -x.generate_layer(layers_type=object_type_name) -``` +.. code-block:: python + + x.generate_layer(layers_type=object_type_name) + The `generate_layer` function generates a collection of layers, each customized to one instance of the input `object_type_name`. Valid types include `group`, `mitigation`, `software`, and `datasource`. From 79edf9ec35914f0dca1afc718023afe255d76bf8 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 14:56:37 -0400 Subject: [PATCH 040/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 07592067..80003b58 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -11,7 +11,7 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: Overview - getting_started + docs/getting_started installation related_work contributing From 49e005d04b8c03d5ff4bb3ed1a711198787864f9 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:12:17 -0400 Subject: [PATCH 041/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 80003b58..1c4dcf49 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -11,11 +11,11 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: Overview - docs/getting_started installation related_work contributing notice + getting_started .. toctree:: :maxdepth: 1 From e38b3f28d3807de108fd14dfa1517a6d3b36ea76 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:16:49 -0400 Subject: [PATCH 042/159] Update getting_started.rst --- docs/getting_started.rst | 62 ---------------------------------------- 1 file changed, 62 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index cab010af..7a0430eb 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -1,65 +1,3 @@ **mitreattack-python** -This repository contains a library of Python tools and utilities for working with ATT&CK data. For more information, -see the [full documentation](https://mitreattack-python.readthedocs.io/) on ReadTheDocs. - -**Install** - -To use this package, install the mitreattack-python library with [pip](https://pip.pypa.io/en/stable/): - -```shell -pip install mitreattack-python -``` - -Note: the library requires [python3](https://www.python.org/). - -**MitreAttackData Library** - -The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides -the ability to query the dataset for objects and their related objects. This is the main content of mitreattack-python; -you can read more about other modules in this library under "Additional Modules". - -**Additional Modules** - -More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. - -| module | description | documentation | -|:------------|:------------|:--------------| -| [navlayers](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/navlayers) | A collection of utilities for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md).| -| [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel) | A collection of utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| -| [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| -| [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/diffStix/README.md).| - - -**Related MITRE Work** - -Go to [this link](https://mitreattack-python.readthedocs.io/en/latest/related_work.html) for related MITRE work. - - -**Contributing** - -To contribute to this project, either through a bug report, feature request, or merge request, -please see the [Contributors Guide](https://github.com/mitre-attack/mitreattack-python/blob/master/docs/CONTRIBUTING.md). - -**Notice** - -Copyright 2023 The MITRE Corporation - -Approved for Public Release; Distribution Unlimited. Case Number 19-0486. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - -This project makes use of ATT&CK® - -[ATT&CK Terms of Use](https://attack.mitre.org/resources/terms-of-use/) From 47e07c0967b1bc7faefe151c0d597eccf1582abb Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:19:43 -0400 Subject: [PATCH 043/159] Update and rename getting_started.rst to start.rst --- docs/{getting_started.rst => start.rst} | 1 + 1 file changed, 1 insertion(+) rename docs/{getting_started.rst => start.rst} (86%) diff --git a/docs/getting_started.rst b/docs/start.rst similarity index 86% rename from docs/getting_started.rst rename to docs/start.rst index 7a0430eb..a5aa4d10 100644 --- a/docs/getting_started.rst +++ b/docs/start.rst @@ -1,3 +1,4 @@ **mitreattack-python** +Hi! From 9f241c56e17f12dc2774e374b0f5f2eed2b5aaec Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:19:54 -0400 Subject: [PATCH 044/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 1c4dcf49..64ac438b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -15,7 +15,7 @@ other modules in this library under "Additional Modules". related_work contributing notice - getting_started + start .. toctree:: :maxdepth: 1 From 94ec9c6a2ee8076761e2faef19b6383d074942dc Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:25:50 -0400 Subject: [PATCH 045/159] Update index.rst --- docs/index.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 64ac438b..fe331500 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -15,7 +15,6 @@ other modules in this library under "Additional Modules". related_work contributing notice - start .. toctree:: :maxdepth: 1 @@ -30,7 +29,7 @@ other modules in this library under "Additional Modules". :caption: STIX Overview stix_primer/access-attack - + start .. toctree:: From 389faa31d3cee01203c13c6d1de0bf72c1c8651c Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:29:01 -0400 Subject: [PATCH 046/159] Update index.rst --- docs/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/index.rst b/docs/index.rst index fe331500..be60c344 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -15,6 +15,7 @@ other modules in this library under "Additional Modules". related_work contributing notice + diffstix .. toctree:: :maxdepth: 1 From cdeec0d02aeb92adcf61b889f228f0a9386d9fb5 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:32:42 -0400 Subject: [PATCH 047/159] Rename diffstix.rst to differentstix.rst --- docs/{diffstix.rst => differentstix.rst} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename docs/{diffstix.rst => differentstix.rst} (99%) diff --git a/docs/diffstix.rst b/docs/differentstix.rst similarity index 99% rename from docs/diffstix.rst rename to docs/differentstix.rst index e5324882..80453812 100644 --- a/docs/diffstix.rst +++ b/docs/differentstix.rst @@ -116,4 +116,4 @@ The following table lists the extra fields that can be found in objects in the c | `changelog_detections` | false | object | Three lists for `shared`, `new`, and `dropped` for Detections that are related to a Technique between versions. || `description_change_table` | false | string | HTML rendering of a table that displays the differences between descriptions for an ATT&CK object. | | `detailed_diff` | false | string | A python DeepDiff object that has been JSON serialized which represents STIX changes for an ATT&CK object between releases. | | `previous_version` | false | string | If the object existed in the previous release, then it denotes the version the object was in the previous release. | -| `version_change` | false | string | If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' | \ No newline at end of file +| `version_change` | false | string | If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' | From e3db2e1cf376555e1190498afb27d6d888d59574 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 15:32:56 -0400 Subject: [PATCH 048/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index be60c344..f0f6fa94 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -15,7 +15,7 @@ other modules in this library under "Additional Modules". related_work contributing notice - diffstix + differentstix .. toctree:: :maxdepth: 1 From 40c7c0264b1b702a30c4fa1d5f6091770c67adc9 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 17:30:28 -0400 Subject: [PATCH 049/159] Update access-attack.rst --- docs/stix_primer/access-attack.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index 038eba40..088c9aef 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -1,7 +1,9 @@ Accessing ATT&CK data in python =============== -**Accessing ATT&CK data in python** +**Accessing ATT&CK data in python + +**hello There are several ways to acquire the ATT&CK data in Python. All of them will provide an object implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. From 41fd41171ce03d848da6b3027136db6060a0b98a Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Fri, 7 Jul 2023 17:33:11 -0400 Subject: [PATCH 050/159] Update access-attack.rst --- docs/stix_primer/access-attack.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index 088c9aef..038eba40 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -1,9 +1,7 @@ Accessing ATT&CK data in python =============== -**Accessing ATT&CK data in python - -**hello +**Accessing ATT&CK data in python** There are several ways to acquire the ATT&CK data in Python. All of them will provide an object implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. From 8d93c2fda78c65238e1c425f6b0634f5368da502 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 17:40:10 -0400 Subject: [PATCH 051/159] reformatting new docs --- docs/attacktoexcel.rst | 65 +++--- docs/differentstix.rst | 136 ++++++------ docs/navlayers.rst | 471 ++++++++++++++++++++--------------------- docs/start.rst | 63 +++++- 4 files changed, 397 insertions(+), 338 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 7aa0f722..8f05f95e 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -1,11 +1,11 @@ -# ATT&CK To Excel +**ATT&CK To Excel** This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. -## Usage +**Usage** -### Command Line +**Command Line** Print full usage instructions: @@ -25,19 +25,19 @@ Build a excel files corresponding to a specific domain and version of ATT&CK: python3 attackToExcel -domain mobile-attack -version v5.0 ``` -### Module +**Module** Example execution targeting a specific domain and version: -```python +.. code-block:: python import mitreattack.attackToExcel.attackToExcel as attackToExcel attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") ``` -## Interfaces +**Interfaces** -### attackToExcel +**attackToExcel** attackToExcel provides the means by which to convert/extract the ATT&CK STIX data to Excel spreadsheets. A brief overview of the available methods follows. @@ -49,7 +49,7 @@ overview of the available methods follows. |write_excel| `dataframes`: pandas DataFrame dictionary (generated by build_dataframes)
`domain`: domain of ATT&CK that `dataframes` corresponds to
`version`: optional parameter indicating which version of ATT&CK is in use
`output_dir`: optional parameter specifying output directory| Writes out DataFrame based ATT&CK data to excel files| |export| `domain`: the domain of ATT&CK to download
`version`: optional parameter specifying which version of ATT&CK to download
`output_dir`: optional parameter specifying output directory| Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets | -### stixToDf +**stixToDf** stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for processing. A brief overview of these methods follows. @@ -64,7 +64,7 @@ processing. A brief overview of these methods follows. |relationshipsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX relationships from the provided data and returns corresponding Pandas DataFrames.| |matricesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]`| -## Spreadsheet format +**Spreadsheet format** The Excel representation of the ATT&CK dataset includes both master spreadsheets, containing all object types, and individual spreadsheets for each object type. @@ -76,33 +76,32 @@ A citations sheet can be used to look up the in-text citations which appear in s For domains that include multiple matrices, such as Mobile ATT&CK, each matrix gets its own named sheet. Unlike the STIX dataset, objects that have been revoked or deprecated are not included in the spreadsheets. -## Accessing the Pandas DataFrames +**Accessing the Pandas DataFrames** Internally, attackToExcel stores the parsed STIX data as [Pandas](https://pandas.pydata.org/) DataFrames. These can be retrieved for use in data analysis. Example of accessing [Pandas](https://pandas.pydata.org/) DataFrames: -```python -import mitreattack.attackToExcel.attackToExcel as attackToExcel -import mitreattack.attackToExcel.stixToDf as stixToDf - -# download and parse ATT&CK STIX data -attackdata = attackToExcel.get_stix_data("enterprise-attack") -techniques_data = stixToDf.techniquesToDf(attackdata, "enterprise-attack") - -# show T1102 and sub-techniques of T1102 -techniques_df = techniques_data["techniques"] -print(techniques_df[techniques_df["ID"].str.contains("T1102")]["name"]) -# 512 Web Service -# 38 Web Service: Bidirectional Communication -# 121 Web Service: Dead Drop Resolver -# 323 Web Service: One-Way Communication -# Name: name, dtype: object - -# show citation data for LOLBAS Wmic reference -citations_df = techniques_data["citations"] -print(citations_df[citations_df["reference"].str.contains("LOLBAS Wmic")]) -# reference citation url -# 1010 LOLBAS Wmic LOLBAS. (n.d.). Wmic.exe. Retrieved July 31, 2... https://lolbas-project.github.io/lolbas/Binari... -``` \ No newline at end of file +.. code-block:: python + import mitreattack.attackToExcel.attackToExcel as attackToExcel + import mitreattack.attackToExcel.stixToDf as stixToDf + + # download and parse ATT&CK STIX data + attackdata = attackToExcel.get_stix_data("enterprise-attack") + techniques_data = stixToDf.techniquesToDf(attackdata, "enterprise-attack") + + # show T1102 and sub-techniques of T1102 + techniques_df = techniques_data["techniques"] + print(techniques_df[techniques_df["ID"].str.contains("T1102")]["name"]) + # 512 Web Service + # 38 Web Service: Bidirectional Communication + # 121 Web Service: Dead Drop Resolver + # 323 Web Service: One-Way Communication + # Name: name, dtype: object + + # show citation data for LOLBAS Wmic reference + citations_df = techniques_data["citations"] + print(citations_df[citations_df["reference"].str.contains("LOLBAS Wmic")]) + # reference citation url + # 1010 LOLBAS Wmic LOLBAS. (n.d.). Wmic.exe. Retrieved July 31, 2... https://lolbas-project.github.io/lolbas/Binari... diff --git a/docs/differentstix.rst b/docs/differentstix.rst index 80453812..62ebe7d9 100644 --- a/docs/differentstix.rst +++ b/docs/differentstix.rst @@ -1,91 +1,91 @@ -# Diff Stix +**Diff Stix** This folder contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. -## Usage +**Usage** -### Command Line +**Command Line** Print full usage instructions: -```shell +.. code:: bash # You must run `pip install mitreattack-python` in order to access the diff_stix command -diff_stix --help -usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] - [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] - -Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. - -options: - -h, --help show this help message and exit - --old OLD Directory to load old STIX data from. - --new NEW Directory to load new STIX data from. - --domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...] - Which domains to report on. Choices (and defaults) are enterprise-attack, mobile-attack, ics-attack - --markdown-file MARKDOWN_FILE - Create a markdown file reporting changes. - --html-file HTML_FILE - Create HTML page from markdown content. - --html-file-detailed HTML_FILE_DETAILED - Create an HTML file reporting detailed changes. - --json-file JSON_FILE - Create a JSON file reporting changes. - --layers [LAYERS ...] - Create layer files showing changes in each domain expected order of filenames is 'enterprise', 'mobile', 'ics', 'pre attack'. If values are unspecified, defaults to output/January_2023_Updates_Enterprise.json, - output/January_2023_Updates_Mobile.json, output/January_2023_Updates_ICS.json, output/January_2023_Updates_Pre.json - --site_prefix SITE_PREFIX - Prefix links in markdown output, e.g. [prefix]/techniques/T1484 - --unchanged Show objects without changes in the markdown output - --use-mitre-cti Use content from the MITRE CTI repo for the -old data - --show-key Add a key explaining the change types to the markdown - --contributors Show new contributors between releases - --no-contributors Do not show new contributors between releases - -v, --verbose Print status messages -``` + diff_stix --help + usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] + [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] + + Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. + + options: + -h, --help show this help message and exit + --old OLD Directory to load old STIX data from. + --new NEW Directory to load new STIX data from. + --domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...] + Which domains to report on. Choices (and defaults) are enterprise-attack, mobile-attack, ics-attack + --markdown-file MARKDOWN_FILE + Create a markdown file reporting changes. + --html-file HTML_FILE + Create HTML page from markdown content. + --html-file-detailed HTML_FILE_DETAILED + Create an HTML file reporting detailed changes. + --json-file JSON_FILE + Create a JSON file reporting changes. + --layers [LAYERS ...] + Create layer files showing changes in each domain expected order of filenames is 'enterprise', 'mobile', 'ics', 'pre attack'. If values are unspecified, defaults to output/January_2023_Updates_Enterprise.json, + output/January_2023_Updates_Mobile.json, output/January_2023_Updates_ICS.json, output/January_2023_Updates_Pre.json + --site_prefix SITE_PREFIX + Prefix links in markdown output, e.g. [prefix]/techniques/T1484 + --unchanged Show objects without changes in the markdown output + --use-mitre-cti Use content from the MITRE CTI repo for the -old data + --show-key Add a key explaining the change types to the markdown + --contributors Show new contributors between releases + --no-contributors Do not show new contributors between releases + -v, --verbose Print status messages + Example execution: -```shell +.. code:: bash diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ -``` -## Changelog JSON format + +**Changelog JSON format** The changelog helper script has the option to output a JSON file with detailed differences between ATT&CK releases. This is the overall structure you can expect to find in the file. A brief explanation of key pieces can be found below. -```JSON -{ - "enterprise-attack": { - "techniques": { - "additions": [], - "major_version_changes": [], - "minor_version_changes": [], - "other_version_changes": [], - "patches": [], - "revocations": [], - "deprecations": [], - "deletions": [], +.. code-block:: json + { + "enterprise-attack": { + "techniques": { + "additions": [], + "major_version_changes": [], + "minor_version_changes": [], + "other_version_changes": [], + "patches": [], + "revocations": [], + "deprecations": [], + "deletions": [], + }, + "software": {}, + "groups": {}, + "campaigns": {}, + "mitigations": {}, + "datasources": {}, + "datacomponents": {} }, - "software": {}, - "groups": {}, - "campaigns": {}, - "mitigations": {}, - "datasources": {}, - "datacomponents": {} - }, - "mobile-attack": {}, - "ics-attack": {}, - "new-contributors": [ - "Contributor A", - "Contributor B", - "Contributor C" - ] -} -``` + "mobile-attack": {}, + "ics-attack": {}, + "new-contributors": [ + "Contributor A", + "Contributor B", + "Contributor C" + ] + } + * The top-level objects include information about specific domains as well as `new-contributors`, which are only found in the newer ATT&CK release. * For domain changes, they are broken down by object type, e.g. `techniques` or `mitigations`. @@ -102,7 +102,7 @@ A brief explanation of key pieces can be found below. | `deprecations` | array[object] | ATT&CK objects which are deprecated and no longer in use, and not replaced. | | `deletions` | array[object] | ATT&CK objects which are no longer found in the STIX data. This should almost never happen. | -### Changed Objects +**Changed Objects The bulk of the changelog file consists of lists of JSON objects. If you are familiar with reading the STIX format, they may look famliar, yet a little "off". diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 3c8440cd..01fc4e5a 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -1,4 +1,4 @@ -# navlayers +**navlayers** This folder contains modules and scripts for working with ATT&CK Navigator layers. ATT&CK Navigator Layers are a set of annotations overlaid on top of the ATT&CK Matrix. @@ -10,7 +10,7 @@ All scripts adhere to the MITRE ATT&CK Navigator Layer file format, but will accept legacy [version 3.0](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv3.md) and version 4.X layers, upgrading them to version 4.3. -## Core Modules +**Core Modules** | script | description | |:-------|:------------| @@ -23,20 +23,20 @@ and version 4.X layers, upgrading them to version 4.3. | [technique](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/technique.py) | Implements a basic [technique object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#technique-object-properties). | | [versions](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/versions.py) | Implements a basic [versions object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#versions-object-properties).| -### Manipulator Scripts +**Manipulator Scripts | script | description | |:-------|:------------| | [layerops](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/manipulators/layerops.py) | Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. | -### Exporter Scripts +**Exporter Scripts | script | description | |:-------|:------------| | [to_excel](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_excel.py) | Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. | | [to_svg](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_svg.py) | Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export.| -### Generator Scripts +**Generator Scripts | script | description | |:-------|:------------| @@ -44,7 +44,7 @@ and version 4.X layers, upgrading them to version 4.3. | [usage_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/usage_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | | [sum_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/sum_generator.py)| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | -### Utility Modules +**Utility Modules | script | description | |:-------|:------------| @@ -53,14 +53,14 @@ and version 4.X layers, upgrading them to version 4.3. | [svg_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_templates.py) | Provides a means by which to convert a layer file into a marked up svg file. | | [svg_objects](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_objects.py) | Provides raw templates and supporting functionality for generating svg objects. | -### Command Line Tools +**Command Line Tools | script | description | |:-------|:------------| | [layerExporter_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerExporter_cli.py) | A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. | | [layerGenerator_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerGenerator_cli.py) | A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. | -## Layer +**Layer The `Layer` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. It is the primary interface through which other Layer-related classes defined in the core module should be used. @@ -79,128 +79,129 @@ The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/README.md). -### Example Usage +**Example Usage -```python -example_layer3_dict = { - "name": "example layer", - "version": "3.0", - "domain": "mitre-enterprise" -} +.. code-block:: python + example_layer3_dict = { + "name": "example layer", + "version": "3.0", + "domain": "mitre-enterprise" + } -example_layer4_dict = { - "name": "layer v4.3 example", - "versions" : { - "attack": "8", - "layer" : "4.3", - "navigator": "4.4.4" - }, - "domain": "enterprise-attack" -} + example_layer4_dict = { + "name": "layer v4.3 example", + "versions" : { + "attack": "8", + "layer" : "4.3", + "navigator": "4.4.4" + }, + "domain": "enterprise-attack" + } -example_layer_location = "/path/to/layer/file.json" -example_layer_out_location = "/path/to/new/layer/file.json" + example_layer_location = "/path/to/layer/file.json" + example_layer_out_location = "/path/to/new/layer/file.json" -from mitreattack.navlayers.core import Layer + from mitreattack.navlayers.core import Layer -layer1 = Layer(example_layer3_dict) # Create a new layer and load existing data -layer1.to_file(example_layer_out_location) # Write out the loaded layer to the specified file + layer1 = Layer(example_layer3_dict) # Create a new layer and load existing data + layer1.to_file(example_layer_out_location) # Write out the loaded layer to the specified file -layer2 = Layer() # Create a new layer object -layer2.from_dict(example_layer4_dict) # Load layer data into existing layer object -print(layer2.to_dict()) # Retrieve the loaded layer's data as a dictionary, and print it + layer2 = Layer() # Create a new layer object + layer2.from_dict(example_layer4_dict) # Load layer data into existing layer object + print(layer2.to_dict()) # Retrieve the loaded layer's data as a dictionary, and print it + + layer3 = Layer() # Create a new layer object + layer3.from_file(example_layer_location) # Load layer data from a file into existing layer object -layer3 = Layer() # Create a new layer object -layer3.from_file(example_layer_location) # Load layer data from a file into existing layer object -``` -### layerops.py +**layerops.py `Layerops.py` provides the `LayerOps` class, which is a way to combine layer files in an automated way, using user defined lambda functions. Each LayerOps instance, when created, ingests the provided lambda functions, and stores them for use. An existing `LayerOps` class can be used to combine layer files according to the initialized lambda using the process method. The breakdown of this two step process is documented in the table below, while examples of both the list and dictionary modes of operation can be found below. -#### LayerOps() +**# LayerOps() + +.. code-block:: python + + x = LayerOps(score=score, comment=comment, enabled=enabled, colors=colors, metadata=metadata, name=name, desc=desc, default_values=default_values) -```python -x = LayerOps(score=score, comment=comment, enabled=enabled, colors=colors, metadata=metadata, name=name, desc=desc, default_values=default_values) -``` Each of the _inputs_ takes a lambda function that will be used to combine technique object fields matching the parameter. The one exception to this is _default_values_, which is an optional dictionary argument containing default values to provide the lambda functions if techniques of the combined layers are missing them. -##### .process() Method +****.process() Method + +.. code-block:: python + x.process(data, default_values=default_values) -```python -x.process(data, default_values=default_values) -``` The process method applies the lambda functions stored during initialization to the layer objects in _data_. _data_ must be either a list or a dictionary of Layer objects, and is expected to match the format of the lambda equations provided during initialization. `default_values` is an optional dictionary argument that overrides the currently stored default values with new ones for this specific processing operation. -#### Example Usage - -```python -from mitreattack.navlayers.manipulators.layerops import LayerOps -from mitreattack.navlayers.core.layer import Layer - -demo = Layer() -demo.from_file("C:\Users\attack\Downloads\layer.json") -demo2 = Layer() -demo2.from_file("C:\Users\attack\Downloads\layer2.json") -demo3 = Layer() -demo3.from_file("C:\Users\attack\Downloads\layer3.json") - -# Example 1) Build a LayerOps object that takes a list and averages scores across the layers -lo = LayerOps(score=lambda x: sum(x) / len(x), - name=lambda x: x[1], - desc=lambda x: "This is an list example") # Build LayerOps object -out_layer = lo.process([demo, demo2]) # Trigger processing on a list of demo and demo2 layers -out_layer.to_file("C:\demo_layer1.json") # Save averaged layer to file -out_layer2 = lo.process([demo, demo2, demo3]) # Trigger processing on a list of demo, demo2, demo3 -visual_aid = out_layer2.to_dict() # Retrieve dictionary representation of processed layer - -# Example 2) Build a LayerOps object that takes a dictionary and averages scores across the layers -lo2 = LayerOps(score=lambda x: sum([x[y] for y in x]) / len([x[y] for y in x]), - colors=lambda x: x['b'], - desc=lambda x: "This is a dict example") # Build LayerOps object, with lambda -out_layer3 = lo2.process({'a': demo, 'b': demo2}) # Trigger processing on a dictionary of demo and demo2 -dict_layer = out_layer3.to_dict() # Retrieve dictionary representation of processed layer -print(dict_layer) # Display retrieved dictionary -out_layer4 = lo2.process({'a': demo, 'b': demo2, 'c': demo3})# Trigger processing on a dictionary of demo, demo2, demo3 -out_layer4.to_file("C:\demo_layer4.json") # Save averaged layer to file - -# Example 3) Build a LayerOps object that takes a single element dictionary and inverts the score -lo3 = LayerOps(score=lambda x: 100 - x['a'], - desc= lambda x: "This is a simple example") # Build LayerOps object to invert score (0-100 scale) -out_layer5 = lo3.process({'a': demo}) # Trigger processing on dictionary of demo -print(out_layer5.to_dict()) # Display processed layer in dictionary form -out_layer5.to_file("C:\demo_layer5.json") # Save inverted score layer to file - -# Example 4) Build a LayerOps object that combines the comments from elements in the list, with custom defaults -lo4 = LayerOps(score=lambda x: '; '.join(x), - default_values= { - "comment": "This was an example of new default values" - }, - desc= lambda x: "This is a defaults example") # Build LayerOps object to combine descriptions, defaults -out_layer6 = lo4.process([demo2, demo3]) # Trigger processing on a list of demo2 and demo0 -out_layer6.to_file("C:\demo_layer6.json") # Save combined comment layer to file -``` +**# Example Usage -## to_excel.py +.. code-block:: python + from mitreattack.navlayers.manipulators.layerops import LayerOps + from mitreattack.navlayers.core.layer import Layer + + demo = Layer() + demo.from_file("C:\Users\attack\Downloads\layer.json") + demo2 = Layer() + demo2.from_file("C:\Users\attack\Downloads\layer2.json") + demo3 = Layer() + demo3.from_file("C:\Users\attack\Downloads\layer3.json") + + # Example 1) Build a LayerOps object that takes a list and averages scores across the layers + lo = LayerOps(score=lambda x: sum(x) / len(x), + name=lambda x: x[1], + desc=lambda x: "This is an list example") # Build LayerOps object + out_layer = lo.process([demo, demo2]) # Trigger processing on a list of demo and demo2 layers + out_layer.to_file("C:\demo_layer1.json") # Save averaged layer to file + out_layer2 = lo.process([demo, demo2, demo3]) # Trigger processing on a list of demo, demo2, demo3 + visual_aid = out_layer2.to_dict() # Retrieve dictionary representation of processed layer + + # Example 2) Build a LayerOps object that takes a dictionary and averages scores across the layers + lo2 = LayerOps(score=lambda x: sum([x[y] for y in x]) / len([x[y] for y in x]), + colors=lambda x: x['b'], + desc=lambda x: "This is a dict example") # Build LayerOps object, with lambda + out_layer3 = lo2.process({'a': demo, 'b': demo2}) # Trigger processing on a dictionary of demo and demo2 + dict_layer = out_layer3.to_dict() # Retrieve dictionary representation of processed layer + print(dict_layer) # Display retrieved dictionary + out_layer4 = lo2.process({'a': demo, 'b': demo2, 'c': demo3})# Trigger processing on a dictionary of demo, demo2, demo3 + out_layer4.to_file("C:\demo_layer4.json") # Save averaged layer to file + + # Example 3) Build a LayerOps object that takes a single element dictionary and inverts the score + lo3 = LayerOps(score=lambda x: 100 - x['a'], + desc= lambda x: "This is a simple example") # Build LayerOps object to invert score (0-100 scale) + out_layer5 = lo3.process({'a': demo}) # Trigger processing on dictionary of demo + print(out_layer5.to_dict()) # Display processed layer in dictionary form + out_layer5.to_file("C:\demo_layer5.json") # Save inverted score layer to file + + # Example 4) Build a LayerOps object that combines the comments from elements in the list, with custom defaults + lo4 = LayerOps(score=lambda x: '; '.join(x), + default_values= { + "comment": "This was an example of new default values" + }, + desc= lambda x: "This is a defaults example") # Build LayerOps object to combine descriptions, defaults + out_layer6 = lo4.process([demo2, demo3]) # Trigger processing on a list of demo2 and demo0 + out_layer6.to_file("C:\demo_layer6.json") # Save combined comment layer to file + + +**to_excel.py `to_excel.py` provides the `ToExcel` class, which is a way to export an existing layer file as an Excel spreadsheet. The `ToExcel` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or retrieving data from an ATT&CK Workbench instance. -### ToExcel() +**ToExcel() + +.. code-block:: python + x = ToExcel(domain='enterprise', source='taxii', resource=None) -```python -x = ToExcel(domain='enterprise', source='taxii', resource=None) -``` The `ToExcel` constructor takes domain, server, and resource arguments during instantiation. The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. @@ -211,45 +212,45 @@ it should utilize a remote ATT&CK Workbench instance. The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, or if the source is set to `remote`, in which case it should be the url of a ATT&CK workbench instance. -### .to_xlsx() Method +**.to_xlsx() Method -```python +.. code-block:: python x.to_xlsx(layerInit=layer, filepath="layer.xlsx") ``` The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. -#### Example Usage +**# Example Usage -```python +.. code-block:: python from mitreattack.navlayers import Layer from mitreattack.navlayers import ToExcel -lay = Layer() -lay.from_file("path/to/layer/file.json") -# Using taxii server for template -t = ToExcel(domain=lay.layer.domain, source='taxii') -t.to_xlsx(layerInit=lay, filepath="demo.xlsx") -# Using local stix data for template -t2 = ToExcel(domain='mobile', source='local', resource='path/to/local/stix.json') -t2.to_xlsx(layerInit=lay, filepath="demo2.xlsx") -# Using remote ATT&CK Workbench instance for template -workbench_url = 'localhost:3000' -t3 = ToExcel(domain='ics', source='remote', resource=workbench_url) -``` + lay = Layer() + lay.from_file("path/to/layer/file.json") + # Using taxii server for template + t = ToExcel(domain=lay.layer.domain, source='taxii') + t.to_xlsx(layerInit=lay, filepath="demo.xlsx") + # Using local stix data for template + t2 = ToExcel(domain='mobile', source='local', resource='path/to/local/stix.json') + t2.to_xlsx(layerInit=lay, filepath="demo2.xlsx") + # Using remote ATT&CK Workbench instance for template + workbench_url = 'localhost:3000' + t3 = ToExcel(domain='ics', source='remote', resource=workbench_url) + -## to_svg.py +**to_svg.py `to_svg.py` provides the `ToSvg` class, which is a way to export an existing layer file as an SVG image file. The `ToSvg` class, like the `ToExcel` class, has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or utilizing a remote ATT&CK Workbench instance. -### ToSvg() +**ToSvg() + +.. code-block:: python + x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) -```python -x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) -``` The `ToSvg` constructor, just like the `ToExcel` constructor, takes domain, server, and resource arguments during instantiation. The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. @@ -261,14 +262,14 @@ or if the source is set to `remote`, in which case it should be the url of an AT The `config` parameter is an optional `SVGConfig` object that can be used to configure the export as desired. If not provided, the configuration for the export will be set to default values. -### SVGConfig() +**SVGConfig() + +.. code-block:: python + y = SVGConfig(width=8.5, height=11, headerHeight=1, unit="in", showSubtechniques="expanded", + font="sans-serif", tableBorderColor="#6B7279", showHeader=True, legendDocked=True, + legendX=0, legendY=0, legendWidth=2, legendHeight=1, showLegend=True, showFilters=True, + showAbout=True, showDomain=True, border=0.104) -```python -y = SVGConfig(width=8.5, height=11, headerHeight=1, unit="in", showSubtechniques="expanded", - font="sans-serif", tableBorderColor="#6B7279", showHeader=True, legendDocked=True, - legendX=0, legendY=0, legendWidth=2, legendHeight=1, showLegend=True, showFilters=True, - showAbout=True, showDomain=True, border=0.104) -``` The `SVGConfig` object is used to configure how an SVG export behaves. The defaults for each of the available values can be found in the declaration above, and a brief explanation for each field is included in the table below. @@ -298,49 +299,48 @@ or stored to one using the `.save_to_file(filename="path/to/file.json)` method. | showAbout | Whether or not to show the About Header Block | bool | True | | border | What default border width to use | number | 0.104 | -### .to_svg() Method +**.to_svg() Method -```python -x.to_svg(layerInit=layer, filepath="layer.svg") -``` +.. code-block:: python + x.to_svg(layerInit=layer, filepath="layer.svg") The `to_svg` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. -#### Example Usage +**# Example Usage -```python -from mitreattack.navlayers import Layer -from mitreattack.navlayers import ToSvg, SVGConfig +.. code-block:: python + from mitreattack.navlayers import Layer + from mitreattack.navlayers import ToSvg, SVGConfig -lay = Layer() -lay.from_file("path/to/layer/file.json") -# Using taxii server for template -t = ToSvg(domain=lay.layer.domain, source='taxii') -t.to_svg(layerInit=lay, filepath="demo.svg") -#Using local stix data for template + lay = Layer() + lay.from_file("path/to/layer/file.json") + # Using taxii server for template + t = ToSvg(domain=lay.layer.domain, source='taxii') + t.to_svg(layerInit=lay, filepath="demo.svg") + #Using local stix data for template -conf = SVGConfig() -conf.load_from_file(filename="path/to/poster/config.json") + conf = SVGConfig() + conf.load_from_file(filename="path/to/poster/config.json") -t2 = ToSvg(domain='mobile', source='local', resource='path/to/local/stix.json', config=conf) -t2.to_svg(layerInit=lay, filepath="demo2.svg") + t2 = ToSvg(domain='mobile', source='local', resource='path/to/local/stix.json', config=conf) + t2.to_svg(layerInit=lay, filepath="demo2.svg") + + workbench_url = "localhost:3000" + t3 = ToSvg(domain='enterprise', source='remote', resource=workbench_url, config=conf) + t3.to_svg(layerInit=lay, filepath="demo3.svg") -workbench_url = "localhost:3000" -t3 = ToSvg(domain='enterprise', source='remote', resource=workbench_url, config=conf) -t3.to_svg(layerInit=lay, filepath="demo3.svg") -``` -## overview_generator.py +**overview_generator.py** `overview_generator.py` provides the `OverviewLayerGenerator` class, which is designed to allow users to generate an ATT&CK layer that, on a per technique basis, has a score that corresponds to all instances of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. -### OverviewLayerGenerator() +**OverviewLayerGenerator()** + +.. code-block:: python + x = OverviewLayerGenerator(source='taxii', domain='enterprise', resource=None) -```python -x = OverviewLayerGenerator(source='taxii', domain='enterprise', resource=None) -``` The initialization function for `OverviewLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where to retrieve data from (taxii server etc.). @@ -352,27 +352,27 @@ The `resource` argument is only required if the source is set to `local`, in whi or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. If not provided, the configuration for the generator will be set to default values. -### .generate_layer() +**.generate_layer()** + +.. code-block:: python + x.generate_layer(obj_type=object_type_name) -```python -x.generate_layer(obj_type=object_type_name) -``` The `generate_layer` function generates a layer, customized to the input `object_type_name`. Valid values include `group`, `mitigation`, `software`, and `datasource`. -## usage_generator.py +**usage_generator.py** `usage_ generator.py` provides the `UsageLayerGenerator` class, which is designed to allow users to generate an ATT&CK layer that scores any relevant techniques that a given input ATT&CK object has. These objects can be any `group`, `software`, `mitigation`, or `data component`, and can be referenced by ID or by any alias when provided to the generator. -### UsageLayerGenerator() +**UsageLayerGenerator()** + +.. code-block:: python + x = UsageLayerGenerator(source='taxii', domain='enterprise', resource=None) -```python -x = UsageLayerGenerator(source='taxii', domain='enterprise', resource=None) -``` The initialization function for `UsageLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where to retrieve data from (taxii server etc.). @@ -384,7 +384,7 @@ The `resource` argument is only required if the source is set to `local`, in whi or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. If not provided, the configuration for the generator will be set to default values. -### .generate_layer() +**.generate_layer()** .. code-block:: python @@ -404,14 +404,14 @@ Valid values include `ATT&CK ID`, `name`, or any known `alias` for `group`, `mit layer2 = handle.generate_layer(match='Adups') -## sum_generator.py +**sum_generator.py** `sum_generator.py` provides the `SumLayerGenerator` class, which is designed to allow users to generate a collection of ATT&CK layers that, on a per technique basis, have a score that corresponds to all instances of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. Each one of the generated layers will correspond to a single instance of the specified ATT&CK object type. -### SumLayerGenerator() +**SumLayerGenerator()** .. code-block:: python @@ -428,7 +428,7 @@ The `resource` argument is only required if the source is set to `local`, in whi or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. If not provided, the configuration for the generator will be set to default values. -### .generate_layer() +**.generate_layer()** .. code-block:: python @@ -438,7 +438,7 @@ If not provided, the configuration for the generator will be set to default valu The `generate_layer` function generates a collection of layers, each customized to one instance of the input `object_type_name`. Valid types include `group`, `mitigation`, `software`, and `datasource`. -## layerExporter_cli.py +**layerExporter_cli.py** This command line tool allows users to convert a [navigator](https://github.com/mitre-attack/attack-navigator) layer file to either an svg image or excel file using the functionality provided by the navlayers module. @@ -446,78 +446,77 @@ Details about the SVG configuration json mentioned below can be found in the [SVGConfig](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md#svgconfig) entry within the navlayers module documentation. -```commandline -C:\Users\attack>layerExporter_cli -h -usage: layerExporter_cli [-h] -m {svg,excel} [-s {taxii,local,remote}] - [--resource RESOURCE] -o OUTPUT [OUTPUT ...] - [-l LOAD_SETTINGS] [-d WIDTH HEIGHT] - input [input ...] - -Export an ATT&CK Navigator layer as a svg image or excel file - -positional arguments: - input Path(s) to the file to export - -optional arguments: - -h, --help show this help message and exit - -m {svg,excel}, --mode {svg,excel} - The form to export the layers in - -s {taxii,local,remote}, --source {taxii,local,remote} - What source to utilize when building the matrix - --resource RESOURCE Path to the local resource if --source=local, or url - of an ATT&CK Workbench instance if --source=remote - -o OUTPUT [OUTPUT ...], --output OUTPUT [OUTPUT ...] - Path(s) to the exported svg/xlsx file - -l LOAD_SETTINGS, --load_settings LOAD_SETTINGS - [SVG Only] Path to a SVG configuration json to use - when rendering - -d WIDTH HEIGHT, --size WIDTH HEIGHT - [SVG Only] X and Y size values (in inches) for SVG - export (use -l for other settings) - -C:\Users\attack>layerExporter_cli -m svg -s taxii -l settings/config.json -o output/svg1.json output/svg2.json files/layer1.json files/layer2.json -``` - -## layerGenerator_cli.py +.. code:: bash + C:\Users\attack>layerExporter_cli -h + usage: layerExporter_cli [-h] -m {svg,excel} [-s {taxii,local,remote}] + [--resource RESOURCE] -o OUTPUT [OUTPUT ...] + [-l LOAD_SETTINGS] [-d WIDTH HEIGHT] + input [input ...] + + Export an ATT&CK Navigator layer as a svg image or excel file + + positional arguments: + input Path(s) to the file to export + + optional arguments: + -h, --help show this help message and exit + -m {svg,excel}, --mode {svg,excel} + The form to export the layers in + -s {taxii,local,remote}, --source {taxii,local,remote} + What source to utilize when building the matrix + --resource RESOURCE Path to the local resource if --source=local, or url + of an ATT&CK Workbench instance if --source=remote + -o OUTPUT [OUTPUT ...], --output OUTPUT [OUTPUT ...] + Path(s) to the exported svg/xlsx file + -l LOAD_SETTINGS, --load_settings LOAD_SETTINGS + [SVG Only] Path to a SVG configuration json to use + when rendering + -d WIDTH HEIGHT, --size WIDTH HEIGHT + [SVG Only] X and Y size values (in inches) for SVG + export (use -l for other settings) + + C:\Users\attack>layerExporter_cli -m svg -s taxii -l settings/config.json -o output/svg1.json output/svg2.json files/layer1.json files/layer2.json + + +**layerGenerator_cli.py** This command line tool allows users to generate [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layer files from either a specific group, software, or mitigation. Alternatively, users can generate a layer file with a mapping to all associated groups, software, or mitigations across the techniques within ATT&CK. -```commandline -C:\Users\attack>layerGenerator_cli -h -usage: layerGenerator_cli [-h] - (--overview-type {group,software,mitigation,datasource} | --mapped-to MAPPED_TO | --batch-type {group,software,mitigation,datasource}) - [-o OUTPUT] [--domain {enterprise,mobile,ics}] - [--source {taxii,local,remote}] - [--resource RESOURCE] - -Generate an ATT&CK Navigator layer - -optional arguments: - -h, --help show this help message and exit - --overview-type {group,software,mitigation,datasource} - Output a layer file where the target type is - summarized across the entire dataset. - --mapped-to MAPPED_TO - Output layer file with techniques mapped to the given - group, software, mitigation, or data component. Argument - can be name, associated group/software, or ATT&CK ID. - --batch-type {group,software,mitigation,datasource} - Output a collection of layer files to the specified - folder, each one representing a different instance of - the target type. - -o OUTPUT, --output OUTPUT - Path to the output layer file/directory - --domain {enterprise,mobile,ics} - Which domain to build off of - --source {taxii,local,remote} - What source to utilize when building the layer files - --resource RESOURCE Path to the local resource if --source=local, or url - of an ATT&CK Workbench instance if --source=remote - -C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --mapped-to S0065 --output generated_layer.json -C:\Users\attack>layerGenerator_cli --domain mobile --source taxii --overview-type mitigation --output generated_layer2.json -C:\Users\attack>layerGenerator_cli --domain ics --source taxii --batch-type software -C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --overview-type datasource --output generated_layer3.json -``` \ No newline at end of file +.. code:: bash + C:\Users\attack>layerGenerator_cli -h + usage: layerGenerator_cli [-h] + (--overview-type {group,software,mitigation,datasource} | --mapped-to MAPPED_TO | --batch-type {group,software,mitigation,datasource}) + [-o OUTPUT] [--domain {enterprise,mobile,ics}] + [--source {taxii,local,remote}] + [--resource RESOURCE] + + Generate an ATT&CK Navigator layer + + optional arguments: + -h, --help show this help message and exit + --overview-type {group,software,mitigation,datasource} + Output a layer file where the target type is + summarized across the entire dataset. + --mapped-to MAPPED_TO + Output layer file with techniques mapped to the given + group, software, mitigation, or data component. Argument + can be name, associated group/software, or ATT&CK ID. + --batch-type {group,software,mitigation,datasource} + Output a collection of layer files to the specified + folder, each one representing a different instance of + the target type. + -o OUTPUT, --output OUTPUT + Path to the output layer file/directory + --domain {enterprise,mobile,ics} + Which domain to build off of + --source {taxii,local,remote} + What source to utilize when building the layer files + --resource RESOURCE Path to the local resource if --source=local, or url + of an ATT&CK Workbench instance if --source=remote + + C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --mapped-to S0065 --output generated_layer.json + C:\Users\attack>layerGenerator_cli --domain mobile --source taxii --overview-type mitigation --output generated_layer2.json + C:\Users\attack>layerGenerator_cli --domain ics --source taxii --batch-type software + C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --overview-type datasource --output generated_layer3.json diff --git a/docs/start.rst b/docs/start.rst index a5aa4d10..cab010af 100644 --- a/docs/start.rst +++ b/docs/start.rst @@ -1,4 +1,65 @@ **mitreattack-python** -Hi! +This repository contains a library of Python tools and utilities for working with ATT&CK data. For more information, +see the [full documentation](https://mitreattack-python.readthedocs.io/) on ReadTheDocs. + +**Install** + +To use this package, install the mitreattack-python library with [pip](https://pip.pypa.io/en/stable/): + +```shell +pip install mitreattack-python +``` + +Note: the library requires [python3](https://www.python.org/). + +**MitreAttackData Library** + +The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides +the ability to query the dataset for objects and their related objects. This is the main content of mitreattack-python; +you can read more about other modules in this library under "Additional Modules". + +**Additional Modules** + +More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. + +| module | description | documentation | +|:------------|:------------|:--------------| +| [navlayers](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/navlayers) | A collection of utilities for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md).| +| [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel) | A collection of utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| +| [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| +| [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/diffStix/README.md).| + + +**Related MITRE Work** + +Go to [this link](https://mitreattack-python.readthedocs.io/en/latest/related_work.html) for related MITRE work. + + +**Contributing** + +To contribute to this project, either through a bug report, feature request, or merge request, +please see the [Contributors Guide](https://github.com/mitre-attack/mitreattack-python/blob/master/docs/CONTRIBUTING.md). + +**Notice** + +Copyright 2023 The MITRE Corporation + +Approved for Public Release; Distribution Unlimited. Case Number 19-0486. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +This project makes use of ATT&CK® + +[ATT&CK Terms of Use](https://attack.mitre.org/resources/terms-of-use/) From dad6a84e676488bd72e1d2cb963615a28241c540 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 18:14:05 -0400 Subject: [PATCH 052/159] reformatting attacktoexcel --- docs/attacktoexcel.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 8f05f95e..92618f62 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -9,31 +9,31 @@ It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata Print full usage instructions: -```shell -python3 attackToExcel.py -h -``` +.. code:: bash + python3 attackToExcel.py -h + Example execution: -```shell -python3 attackToExcel.py -``` +.. code:: bash + python3 attackToExcel.py + Build a excel files corresponding to a specific domain and version of ATT&CK: -```shell -python3 attackToExcel -domain mobile-attack -version v5.0 -``` +.. code:: bash + python3 attackToExcel -domain mobile-attack -version v5.0 + **Module** Example execution targeting a specific domain and version: .. code-block:: python -import mitreattack.attackToExcel.attackToExcel as attackToExcel + import mitreattack.attackToExcel.attackToExcel as attackToExcel + + attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") -attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") -``` **Interfaces** From 46a4f9544a64bbdb4a68a3593630e1b721f6e20c Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 7 Jul 2023 18:17:27 -0400 Subject: [PATCH 053/159] reformatting start.rst --- docs/start.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/start.rst b/docs/start.rst index cab010af..2efcead7 100644 --- a/docs/start.rst +++ b/docs/start.rst @@ -8,9 +8,9 @@ see the [full documentation](https://mitreattack-python.readthedocs.io/) on Read To use this package, install the mitreattack-python library with [pip](https://pip.pypa.io/en/stable/): -```shell -pip install mitreattack-python -``` +.. code:: bash + pip install mitreattack-python + Note: the library requires [python3](https://www.python.org/). From d0613a446fd1c60b53df6dbafb84393f49080d31 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 14:39:58 -0400 Subject: [PATCH 054/159] Update index.rst --- docs/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/index.rst b/docs/index.rst index f0f6fa94..1ba5654e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -17,6 +17,7 @@ other modules in this library under "Additional Modules". notice differentstix + .. toctree:: :maxdepth: 1 :caption: MITRE ATT&CK Data Library From 68b5f6ff0bbf164ed5a3d58dd00d136854314974 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 14:43:40 -0400 Subject: [PATCH 055/159] Rename differentstix.rst to differenceinstix.rst --- docs/{differentstix.rst => differenceinstix.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{differentstix.rst => differenceinstix.rst} (100%) diff --git a/docs/differentstix.rst b/docs/differenceinstix.rst similarity index 100% rename from docs/differentstix.rst rename to docs/differenceinstix.rst From b52e1ff1e4a01ac626e9d8869d5de120065a67c8 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 14:43:51 -0400 Subject: [PATCH 056/159] Update index.rst --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 1ba5654e..766515d2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -15,7 +15,7 @@ other modules in this library under "Additional Modules". related_work contributing notice - differentstix + differenceinstix .. toctree:: From 6bded72b0c04663e13391e78ade12b5b6d5a0878 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 14:44:23 -0400 Subject: [PATCH 057/159] Update index.rst --- docs/index.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/index.rst b/docs/index.rst index 766515d2..fe8c7312 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,6 +16,9 @@ other modules in this library under "Additional Modules". contributing notice differenceinstix + attacktoexcel + collections + navlayers .. toctree:: From 4bfec530489ed7b0173e1b4feb4fd458d2bf38b2 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 15:03:45 -0400 Subject: [PATCH 058/159] removed index.rst to test out --- docs/index.rst | 47 ----------------------------------------------- 1 file changed, 47 deletions(-) delete mode 100644 docs/index.rst diff --git a/docs/index.rst b/docs/index.rst deleted file mode 100644 index fe8c7312..00000000 --- a/docs/index.rst +++ /dev/null @@ -1,47 +0,0 @@ -mitreattack-python library -============================================== - -**mitreattack-python** is a library of Python tools and utilities for working with ATT&CK -content. - -The main content of this library is in :ref:`MitreAttackData ref`; you can read more about -other modules in this library under "Additional Modules". - -.. toctree:: - :maxdepth: 1 - :caption: Overview - - installation - related_work - contributing - notice - differenceinstix - attacktoexcel - collections - navlayers - - -.. toctree:: - :maxdepth: 1 - :caption: MITRE ATT&CK Data Library - - mitre_attack_data/mitre_attack_data - mitre_attack_data/examples - mitre_attack_data/custom_objects - -.. toctree:: - :maxdepth: 1 - :caption: STIX Overview - - stix_primer/access-attack - start - - -.. toctree:: - :maxdepth: 1 - :caption: Additional Modules - - additional_modules/navlayers - additional_modules/attackToExcel - additional_modules/collections - additional_modules/diffStix From c8625e610cda70de597bb020d6019fdee3b2ce63 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 15:29:34 -0400 Subject: [PATCH 059/159] swapping additional modules section --- docs/index.rst | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 docs/index.rst diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 00000000..dec534b7 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,46 @@ +mitreattack-python library +============================================== + +**mitreattack-python** is a library of Python tools and utilities for working with ATT&CK +content. + +The main content of this library is in :ref:`MitreAttackData ref`; you can read more about +other modules in this library under "Additional Modules". + +.. toctree:: + :maxdepth: 1 + :caption: Overview + + installation + related_work + contributing + notice + + +.. toctree:: + :maxdepth: 1 + :caption: MITRE ATT&CK Data Library + + mitre_attack_data/mitre_attack_data + mitre_attack_data/examples + mitre_attack_data/custom_objects + +.. toctree:: + :maxdepth: 1 + :caption: STIX Overview + + stix_primer/access-attack + start + + +.. toctree:: + :maxdepth: 1 + :caption: Additional Modules + + differenceinstix + attacktoexcel + collections + navlayers + + + From 9a92c059ab11fd5527eead0013a519547de46613 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 15:36:21 -0400 Subject: [PATCH 060/159] changing stix section title --- docs/index.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index dec534b7..37013c9f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,7 +27,7 @@ other modules in this library under "Additional Modules". .. toctree:: :maxdepth: 1 - :caption: STIX Overview + :caption: STIX Primer stix_primer/access-attack start @@ -36,11 +36,12 @@ other modules in this library under "Additional Modules". .. toctree:: :maxdepth: 1 :caption: Additional Modules - + differenceinstix attacktoexcel collections navlayers + start From 5b9b83c81d85e861554b00fef6c8b5f436cc1e7a Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 15:50:34 -0400 Subject: [PATCH 061/159] changing addl modules section location --- docs/index.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 37013c9f..01ebd7cb 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,6 +16,15 @@ other modules in this library under "Additional Modules". contributing notice +.. toctree:: + :maxdepth: 1 + :caption: Additional Modules + + differenceinstix + attacktoexcel + collections + navlayers + start .. toctree:: :maxdepth: 1 @@ -33,15 +42,7 @@ other modules in this library under "Additional Modules". start -.. toctree:: - :maxdepth: 1 - :caption: Additional Modules - differenceinstix - attacktoexcel - collections - navlayers - start From 1a4af824a341454914a385c59744ed0903bb8761 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 16:02:39 -0400 Subject: [PATCH 062/159] undoing changes --- docs/index.rst | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 01ebd7cb..df858b11 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,15 +16,6 @@ other modules in this library under "Additional Modules". contributing notice -.. toctree:: - :maxdepth: 1 - :caption: Additional Modules - - differenceinstix - attacktoexcel - collections - navlayers - start .. toctree:: :maxdepth: 1 @@ -39,10 +30,13 @@ other modules in this library under "Additional Modules". :caption: STIX Primer stix_primer/access-attack - start - - - +.. toctree:: + :maxdepth: 1 + :caption: Additional Modules + additional_modules/navlayers + additional_modules/attackToExcel + additional_modules/collections + additional_modules/diffStix From b63275f1e22c7e6a98b3af335901c847d06ddaea Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 16:06:09 -0400 Subject: [PATCH 063/159] added titles to new docs --- docs/attacktoexcel.rst | 3 ++- docs/collections.rst | 4 +++- docs/{differenceinstix.rst => diffinstix.rst} | 3 ++- docs/{start.rst => getting_started.rst} | 3 ++- docs/index.rst | 10 +++++----- docs/navlayers.rst | 3 ++- 6 files changed, 16 insertions(+), 10 deletions(-) rename docs/{differenceinstix.rst => diffinstix.rst} (99%) rename docs/{start.rst => getting_started.rst} (98%) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 92618f62..62e3b802 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -1,4 +1,5 @@ -**ATT&CK To Excel** +ATT&CK to Excel +============================================== This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. diff --git a/docs/collections.rst b/docs/collections.rst index 8d40efcc..372a6364 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -1,4 +1,6 @@ -**collections** + +Collections +============================================== This folder contains modules and scripts for working with ATT&CK collections. Collections are sets of ATT&CK STIX objects, grouped for user convienence. diff --git a/docs/differenceinstix.rst b/docs/diffinstix.rst similarity index 99% rename from docs/differenceinstix.rst rename to docs/diffinstix.rst index 62ebe7d9..ecc896e5 100644 --- a/docs/differenceinstix.rst +++ b/docs/diffinstix.rst @@ -1,4 +1,5 @@ -**Diff Stix** +Diff Stix +============================================== This folder contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. diff --git a/docs/start.rst b/docs/getting_started.rst similarity index 98% rename from docs/start.rst rename to docs/getting_started.rst index 2efcead7..64540e4a 100644 --- a/docs/start.rst +++ b/docs/getting_started.rst @@ -1,5 +1,6 @@ -**mitreattack-python** +Getting started +============================================== This repository contains a library of Python tools and utilities for working with ATT&CK data. For more information, see the [full documentation](https://mitreattack-python.readthedocs.io/) on ReadTheDocs. diff --git a/docs/index.rst b/docs/index.rst index df858b11..d02a6075 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -15,7 +15,7 @@ other modules in this library under "Additional Modules". related_work contributing notice - + getting_started .. toctree:: :maxdepth: 1 @@ -36,7 +36,7 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: Additional Modules - additional_modules/navlayers - additional_modules/attackToExcel - additional_modules/collections - additional_modules/diffStix + diffinstix + collections + navlayers + attacktoexcel diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 01fc4e5a..3993b601 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -1,4 +1,5 @@ -**navlayers** +NavLayers +============================================== This folder contains modules and scripts for working with ATT&CK Navigator layers. ATT&CK Navigator Layers are a set of annotations overlaid on top of the ATT&CK Matrix. From e60cbf81d0cc4b660f34325af680132cd6c5a436 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 10 Jul 2023 16:10:18 -0400 Subject: [PATCH 064/159] adding back original addl modules for comparison --- docs/index.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index d02a6075..b543de1d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -17,6 +17,7 @@ other modules in this library under "Additional Modules". notice getting_started + .. toctree:: :maxdepth: 1 :caption: MITRE ATT&CK Data Library @@ -36,7 +37,16 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: Additional Modules + additional_modules/navlayers + additional_modules/attackToExcel + additional_modules/collections + additional_modules/diffStix + +.. toctree:: + :maxdepth: 1 + :caption: Additional Modules part 2 + diffinstix collections navlayers - attacktoexcel + attacktoexcel \ No newline at end of file From 7b03e152d721727e39a0fb9bb1347dad79b10e74 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:19:54 -0400 Subject: [PATCH 065/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index 372a6364..289e08c3 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -11,7 +11,7 @@ For more information about ATT&CK collections, see the corresponding | script | description | |:-------|:------------| -|[index_to_markdown](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/index_to_markdown.py)| Provides a means by which to convert a [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| +`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| |[collection_to_index](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/collection_to_index.py)| Provides a means by which to convert a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) into a easy-to-share [index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes). More information can be found in the corresponding [section](#collection_to_index.py) below.| |[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| From 0ba585be1136fc476c10de9cceaa7e0d044e8715 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:21:16 -0400 Subject: [PATCH 066/159] Update collections.rst --- docs/collections.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index 289e08c3..acf747a7 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,10 +9,8 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -| script | description | -|:-------|:------------| `index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| -|[collection_to_index](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/collection_to_index.py)| Provides a means by which to convert a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) into a easy-to-share [index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes). More information can be found in the corresponding [section](#collection_to_index.py) below.| +'collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| |[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| **index_to_markdown.py** From 49f5adf54649ca4920ebc4184a8b5b5e44f361cf Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:21:38 -0400 Subject: [PATCH 067/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index acf747a7..e02f161a 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -10,7 +10,7 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** `index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| -'collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| +`collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| |[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| **index_to_markdown.py** From 2a6981e6b5763d7acd3443fa6bf10cde04de20fa Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:22:08 -0400 Subject: [PATCH 068/159] Update collections.rst --- docs/collections.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index e02f161a..a0124767 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -24,17 +24,17 @@ An example of how to use the class, and method, can be found below. **Example Usage** -.. code-block:: python + .. code-block:: python + + import json + from mitreattack.collections import IndexToMarkdown - import json - from mitreattack.collections import IndexToMarkdown - - with open('collection_index.json', 'r') as input_file: - with open('collection_index.md', 'w') as output_file: - input_index = json.load(input_file) - generated_md = IndexToMarkdown.index_to_markdown(input_index) # Convert index to markdown - output_file.write(generated_md) - print(generated_md) + with open('collection_index.json', 'r') as input_file: + with open('collection_index.md', 'w') as output_file: + input_index = json.load(input_file) + generated_md = IndexToMarkdown.index_to_markdown(input_index) # Convert index to markdown + output_file.write(generated_md) + print(generated_md) **collection_to_index.py** From e89d0937086309eaa75dc0e2e4824191ef3334fd Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:22:32 -0400 Subject: [PATCH 069/159] Update collections.rst --- docs/collections.rst | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index a0124767..f5694dc0 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -49,24 +49,24 @@ and a list of either files, folders, or already loaded bundles in the form of di .. code-block:: python - import json - from mitreattack.collections import CollectionToIndex - - output_indexA = CollectionToIndex.generate_index(name='example', description='example index', - root_url='www.example.com', - files=['/path/to/collection1.json', '/path/to/collection2.json'], - folders=None, sets=None) - output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index', - root_url='www.example.com', - files=None, folders=['/path/to/folder/with/collections'], sets=None) - with open('path/to/bundle/bundleC.json', 'r') as f: - data = json.load(f) - output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index', - root_url='www.example.com', - files=None, folders=None, sets=[data]) - print(output_indexA) - print(output_indexB) - print(output_indexC) + import json + from mitreattack.collections import CollectionToIndex + + output_indexA = CollectionToIndex.generate_index(name='example', description='example index', + root_url='www.example.com', + files=['/path/to/collection1.json', '/path/to/collection2.json'], + folders=None, sets=None) + output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index', + root_url='www.example.com', + files=None, folders=['/path/to/folder/with/collections'], sets=None) + with open('path/to/bundle/bundleC.json', 'r') as f: + data = json.load(f) + output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index', + root_url='www.example.com', + files=None, folders=None, sets=[data]) + print(output_indexA) + print(output_indexB) + print(output_indexC) **stix_to_collection.py** From c092ea0a72154e4a59fec3bfe352e8b1d71e785a Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:26:27 -0400 Subject: [PATCH 070/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index f5694dc0..051eda12 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -5,7 +5,7 @@ Collections This folder contains modules and scripts for working with ATT&CK collections. Collections are sets of ATT&CK STIX objects, grouped for user convienence. For more information about ATT&CK collections, see the corresponding -[ATT&CK documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). +`ATT&CK documentation `_. **Collections Scripts** From bdb45fa4b3f6ad6b69b6825ed5a68a5e68f78fe7 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:27:49 -0400 Subject: [PATCH 071/159] Update collections.rst --- docs/collections.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index 051eda12..3e2abb7b 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -11,12 +11,12 @@ For more information about ATT&CK collections, see the corresponding `index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| `collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| -|[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| +`stix_to_collection `_ | Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below.| **index_to_markdown.py** `index_to_markdown.py` provides the `IndexToMarkdown` class, which provides a way to transform an existing -[collection index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) +`collection index file `_ into a markdown file for easy of use and reference. The `IndexToMarkdown` class is very simple, and provides a single method, `index_to_markdown`, which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown. From 722468397ae0a982ae31af9904fefcb144085ef7 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:29:10 -0400 Subject: [PATCH 072/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index 3e2abb7b..d485c7f0 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,7 +9,7 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| +`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below.| `collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| `stix_to_collection `_ | Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below.| From b8c423584675005284d24c442fa4c538d3364c5c Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:32:44 -0400 Subject: [PATCH 073/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index d485c7f0..76167811 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,7 +9,7 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below.| +`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py ref` below.| `collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| `stix_to_collection `_ | Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below.| From 96fdba32b4fccaf04d4a4f2e21eb78c18524725a Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:37:00 -0400 Subject: [PATCH 074/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index 76167811..9ca8e76a 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,7 +9,7 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py ref` below.| +`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py ref` below.... `collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| `stix_to_collection `_ | Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below.| From ff5e39560934fbeeb8c1fbe016a221e95c364f7d Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:42:14 -0400 Subject: [PATCH 075/159] Update collections.rst --- docs/collections.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/collections.rst b/docs/collections.rst index 9ca8e76a..49d5ea0b 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,7 +9,7 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py ref` below.... +`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below.... `collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| `stix_to_collection `_ | Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below.| From be65763048f3b1e8b4b76ccf3b14036b417be607 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:50:21 -0400 Subject: [PATCH 076/159] Update collections.rst --- docs/collections.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index 49d5ea0b..88af02d2 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -40,8 +40,8 @@ An example of how to use the class, and method, can be found below. **collection_to_index.py** `collection_to_index.py` provides the `CollectionToIndex` class, which proves a means by which to summarize existing -[collections](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) -into a single [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) file. +`collections `_ +into a single `collection index `_ file. The `CollectionToIndex` class contains the `generate_index` function, which when provided with a name, description, root url (pointing to where the raw collections are stored), and a list of either files, folders, or already loaded bundles in the form of dictionaries, will create a summarizing index. @@ -73,7 +73,7 @@ and a list of either files, folders, or already loaded bundles in the form of di `stix_to_collection.py` provides the `STIXToCollection` class, which proves a means by which to convert existing stix bundles into ones containing a -[collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) object. +`collection `_ object. The `STIXToCollection` class contains the `stix_to_collection` function, which when provided with a starter bundle, a name, a version, and an optional description, will output a modified bundle that contains a summary collection object. From c1219cdbf0f049066032926cb459e36581a8124f Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:50:56 -0400 Subject: [PATCH 077/159] Update diffinstix.rst --- docs/diffinstix.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index ecc896e5..c4118821 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -14,7 +14,7 @@ Print full usage instructions: .. code:: bash # You must run `pip install mitreattack-python` in order to access the diff_stix command diff_stix --help - usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] + usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. From ea9bc074252f48dcc7611c9a36997cb954c70330 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:52:20 -0400 Subject: [PATCH 078/159] Update getting_started.rst --- docs/getting_started.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 64540e4a..d0f49e35 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -25,9 +25,7 @@ you can read more about other modules in this library under "Additional Modules" More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. -| module | description | documentation | -|:------------|:------------|:--------------| -| [navlayers](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/navlayers) | A collection of utilities for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md).| +`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ .| | [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel) | A collection of utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| | [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| | [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/diffStix/README.md).| From d2f84df27be9b9f85382c4234eb24455c1f1924d Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 16:52:58 -0400 Subject: [PATCH 079/159] Update getting_started.rst --- docs/getting_started.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index d0f49e35..1cd569a2 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -25,8 +25,7 @@ you can read more about other modules in this library under "Additional Modules" More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. -`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ .| -| [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel) | A collection of utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| +`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| | [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| | [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/diffStix/README.md).| From 8402953e2a67ce61c5dc40b4cae828b7695f15da Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:04:44 -0400 Subject: [PATCH 080/159] Update getting_started.rst --- docs/getting_started.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 1cd569a2..4db9b324 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -25,20 +25,20 @@ you can read more about other modules in this library under "Additional Modules" More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. -`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).| -| [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) | A set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md). Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| -| [diffStix](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/diffStix) | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/diffStix/README.md).| +`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found `here ` +`collections `_ A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found `here `_| +| `diffStix `_ | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found `here `_.| **Related MITRE Work** -Go to [this link](https://mitreattack-python.readthedocs.io/en/latest/related_work.html) for related MITRE work. +Go to `this link `_ for related MITRE work. **Contributing** To contribute to this project, either through a bug report, feature request, or merge request, -please see the [Contributors Guide](https://github.com/mitre-attack/mitreattack-python/blob/master/docs/CONTRIBUTING.md). +please see the `Contributors Guide `_. **Notice** From 5c4913c2abee0474144549692acd65954316cb73 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:05:55 -0400 Subject: [PATCH 081/159] Update getting_started.rst --- docs/getting_started.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 4db9b324..3b54ecfa 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -25,8 +25,8 @@ you can read more about other modules in this library under "Additional Modules" More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. -`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis. | Further documentation can be found `here ` -`collections `_ A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found `here `_| +`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to `Pandas ` DataFrames representing the dataset for use in data analysis. | Further documentation can be found `here `_. +`collections `_. A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found `here `_. | `diffStix `_ | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found `here `_.| From cbd232f9e4a436a20e8167564bec4fd1818e4ea4 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:06:13 -0400 Subject: [PATCH 082/159] Update getting_started.rst --- docs/getting_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 3b54ecfa..7ca57005 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -25,7 +25,7 @@ you can read more about other modules in this library under "Additional Modules" More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. -`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to `Pandas ` DataFrames representing the dataset for use in data analysis. | Further documentation can be found `here `_. +`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to `Pandas `_ DataFrames representing the dataset for use in data analysis. | Further documentation can be found `here `_. `collections `_. A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found `here `_. | `diffStix `_ | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found `here `_.| From c0b35f4857caf65f0ad894dfceb81ec116962403 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:08:00 -0400 Subject: [PATCH 083/159] Update navlayers.rst --- docs/navlayers.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 3993b601..29564576 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -13,12 +13,10 @@ and version 4.X layers, upgrading them to version 4.3. **Core Modules** -| script | description | -|:-------|:------------| -| [filter](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/filter.py) | Implements a basic [filter object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#filter-object-properties). | -| [gradient](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/gradient.py) | Implements a basic [gradient object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#gradient-object-properties). | -| [layer](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/layer.py) | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | -| [layout](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/layout.py) | Implements a basic [layout object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#layout-object-properties). | +`filter `_ | Implements a basic `filter object `_. | +| `gradient `_ | Implements a basic `gradient object `_. | +| `layer `_ | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | +| `layout `_ | Implements a basic `layout object `_. | | [legenditem](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/legenditem.py) | Implements a basic [legenditem object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#legenditem-object-properties). | | [metadata](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/metadata.py) | Implements a basic [metadata object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#metadata-object-properties). | | [technique](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/technique.py) | Implements a basic [technique object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#technique-object-properties). | From 9e47f189042b6b6269b4de621b9287b43917de2d Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:10:51 -0400 Subject: [PATCH 084/159] Update navlayers.rst --- docs/navlayers.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 29564576..89bfac18 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -7,8 +7,8 @@ For more about ATT&CK Navigator layers, visit the ATT&CK Navigator repository. The core module allows users to load, validate, manipulate, and save ATT&CK layers. A brief overview of the components can be found below. All scripts adhere to the MITRE ATT&CK Navigator Layer file format, -[version 4.3](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_3.md), -but will accept legacy [version 3.0](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv3.md) +`version 4.3 `_, +but will accept legacy `version 3.0 `_ and version 4.X layers, upgrading them to version 4.3. **Core Modules** @@ -17,10 +17,10 @@ and version 4.X layers, upgrading them to version 4.3. | `gradient `_ | Implements a basic `gradient object `_. | | `layer `_ | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | | `layout `_ | Implements a basic `layout object `_. | -| [legenditem](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/legenditem.py) | Implements a basic [legenditem object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#legenditem-object-properties). | -| [metadata](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/metadata.py) | Implements a basic [metadata object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#metadata-object-properties). | -| [technique](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/technique.py) | Implements a basic [technique object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#technique-object-properties). | -| [versions](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/versions.py) | Implements a basic [versions object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#versions-object-properties).| +| `legenditem `_ | Implements a basic `legenditem object `_. | +| `metadata `_ | Implements a basic `metadata object `_. | +| `technique `_ | Implements a basic `technique object `_. | +| `versions `_ | Implements a basic `versions object `_.| **Manipulator Scripts From abf685296c1d5d1dc8173ed02df183b63b9be460 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:13:02 -0400 Subject: [PATCH 085/159] Update navlayers.rst --- docs/navlayers.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 89bfac18..df7744ff 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -26,31 +26,31 @@ and version 4.X layers, upgrading them to version 4.3. | script | description | |:-------|:------------| -| [layerops](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/manipulators/layerops.py) | Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. | +| `layerops `_ | Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. | **Exporter Scripts | script | description | |:-------|:------------| -| [to_excel](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_excel.py) | Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. | -| [to_svg](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_svg.py) | Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export.| +| `to_excel `_ | Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. | +| `to_svg `_ | Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export.| **Generator Scripts | script | description | |:-------|:------------| -| [overview_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/overview_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | -| [usage_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/usage_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | -| [sum_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/sum_generator.py)| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | +| `overview_generator `_| Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | +| `usage_generator `_ | Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | +| `sum_generator `_| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | **Utility Modules | script | description | |:-------|:------------| -| [excel_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/excel_templates.py) | Provides a means by which to convert a matrix into a clean excel matrix template. | -| [matrix_gen](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/matrix_gen.py) | Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). | -| [svg_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_templates.py) | Provides a means by which to convert a layer file into a marked up svg file. | -| [svg_objects](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_objects.py) | Provides raw templates and supporting functionality for generating svg objects. | +| `excel_templates `_ | Provides a means by which to convert a matrix into a clean excel matrix template. | +| `matrix_gen `_ | Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). | +| `svg_templates `_ | Provides a means by which to convert a layer file into a marked up svg file. | +| `svg_objects `_ | Provides raw templates and supporting functionality for generating svg objects. | **Command Line Tools From 19c1c7758d72cbe56faa3f811694918c7672ceaf Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:13:33 -0400 Subject: [PATCH 086/159] Update navlayers.rst --- docs/navlayers.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index df7744ff..d43acf74 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -37,9 +37,7 @@ and version 4.X layers, upgrading them to version 4.3. **Generator Scripts -| script | description | -|:-------|:------------| -| `overview_generator `_| Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | +`overview_generator `_. Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | | `usage_generator `_ | Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | | `sum_generator `_| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | From 2db1289ace134789f39d0690b474eeb75feded22 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Mon, 10 Jul 2023 17:15:47 -0400 Subject: [PATCH 087/159] Update navlayers.rst --- docs/navlayers.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index d43acf74..031e74bf 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -39,7 +39,7 @@ and version 4.X layers, upgrading them to version 4.3. `overview_generator `_. Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | | `usage_generator `_ | Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | -| `sum_generator `_| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | +| `sum_generator `_ . Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | **Utility Modules @@ -54,8 +54,8 @@ and version 4.X layers, upgrading them to version 4.3. | script | description | |:-------|:------------| -| [layerExporter_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerExporter_cli.py) | A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. | -| [layerGenerator_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerGenerator_cli.py) | A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. | +| `layerExporter_cli.py `_ | A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. | +| `layerGenerator_cli.py `_ | A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. | **Layer @@ -74,7 +74,7 @@ The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will | `x.to_str()` | Returns a representation of the current ATT&CK layer object as a string representation of a dictionary. | Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found -[here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/README.md). +`here `_. **Example Usage @@ -437,10 +437,10 @@ Valid types include `group`, `mitigation`, `software`, and `datasource`. **layerExporter_cli.py** -This command line tool allows users to convert a [navigator](https://github.com/mitre-attack/attack-navigator) +This command line tool allows users to convert a `navigator `_ layer file to either an svg image or excel file using the functionality provided by the navlayers module. Details about the SVG configuration json mentioned below can be found in the -[SVGConfig](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md#svgconfig) +`SVGConfig `_ entry within the navlayers module documentation. .. code:: bash @@ -477,7 +477,7 @@ entry within the navlayers module documentation. **layerGenerator_cli.py** -This command line tool allows users to generate [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) +This command line tool allows users to generate `ATT&CK Navigator `_ layer files from either a specific group, software, or mitigation. Alternatively, users can generate a layer file with a mapping to all associated groups, software, or mitigations across the techniques within ATT&CK. From 21992eb231dec1fedfa1a67203ebb0e4dd58872b Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Tue, 11 Jul 2023 14:57:29 -0400 Subject: [PATCH 088/159] Update README.md --- mitreattack/navlayers/README.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mitreattack/navlayers/README.md b/mitreattack/navlayers/README.md index ae507a80..b783fa09 100644 --- a/mitreattack/navlayers/README.md +++ b/mitreattack/navlayers/README.md @@ -12,6 +12,20 @@ and version 4.X layers, upgrading them to version 4.3. ## Core Modules +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + + * - Heading row 1, column 1 + - Heading row 1, column 2 + - Heading row 1, column 3 + * - Row 1, column 1 + - + - Row 1, column 3 + * - Row 2, column 1 + - Row 2, column 2 + - Row 2, column 3 + | script | description | |:-------|:------------| | [filter](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/filter.py) | Implements a basic [filter object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#filter-object-properties). | @@ -518,4 +532,4 @@ C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --mapped-t C:\Users\attack>layerGenerator_cli --domain mobile --source taxii --overview-type mitigation --output generated_layer2.json C:\Users\attack>layerGenerator_cli --domain ics --source taxii --batch-type software C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --overview-type datasource --output generated_layer3.json -``` \ No newline at end of file +``` From 63b929cced3ab5bf8ce2772a95e95a4ec47e20f0 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Tue, 11 Jul 2023 15:01:14 -0400 Subject: [PATCH 089/159] Update README.md --- mitreattack/navlayers/README.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/mitreattack/navlayers/README.md b/mitreattack/navlayers/README.md index b783fa09..31bb93d0 100644 --- a/mitreattack/navlayers/README.md +++ b/mitreattack/navlayers/README.md @@ -10,22 +10,6 @@ All scripts adhere to the MITRE ATT&CK Navigator Layer file format, but will accept legacy [version 3.0](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv3.md) and version 4.X layers, upgrading them to version 4.3. -## Core Modules - -.. list-table:: Title - :widths: 25 25 50 - :header-rows: 1 - - * - Heading row 1, column 1 - - Heading row 1, column 2 - - Heading row 1, column 3 - * - Row 1, column 1 - - - - Row 1, column 3 - * - Row 2, column 1 - - Row 2, column 2 - - Row 2, column 3 - | script | description | |:-------|:------------| | [filter](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/filter.py) | Implements a basic [filter object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#filter-object-properties). | From 86ee4f5331f2202c3abe24a2bfbbd070a8bab87f Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Tue, 11 Jul 2023 15:01:58 -0400 Subject: [PATCH 090/159] Update navlayers.rst --- docs/navlayers.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 031e74bf..d1047ceb 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -13,6 +13,23 @@ and version 4.X layers, upgrading them to version 4.3. **Core Modules** + +## Core Modules + +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + + * - Heading row 1, column 1 + - Heading row 1, column 2 + - Heading row 1, column 3 + * - Row 1, column 1 + - + - Row 1, column 3 + * - Row 2, column 1 + - Row 2, column 2 + - Row 2, column 3 + `filter `_ | Implements a basic `filter object `_. | | `gradient `_ | Implements a basic `gradient object `_. | | `layer `_ | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | From a7ca6175334dec699b43c1dca2111ac666f77033 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Tue, 11 Jul 2023 15:17:38 -0400 Subject: [PATCH 091/159] Update navlayers.rst --- docs/navlayers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index d1047ceb..fd46841b 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -20,9 +20,9 @@ and version 4.X layers, upgrading them to version 4.3. :widths: 25 25 50 :header-rows: 1 - * - Heading row 1, column 1 + * - Heading row 1, column 11 - Heading row 1, column 2 - - Heading row 1, column 3 + - Heading row 1, column 32 * - Row 1, column 1 - - Row 1, column 3 From d729446e6a8a61d0f41992cc32972fb0605c1066 Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Tue, 11 Jul 2023 15:18:22 -0400 Subject: [PATCH 092/159] Update navlayers.rst --- docs/navlayers.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index fd46841b..2a06e982 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -29,6 +29,9 @@ and version 4.X layers, upgrading them to version 4.3. * - Row 2, column 1 - Row 2, column 2 - Row 2, column 3 + * - row 1 + - row 2 + - row 3 `filter `_ | Implements a basic `filter object `_. | | `gradient `_ | Implements a basic `gradient object `_. | From 310175a5dca156d8792eacdd371ff8489ff5776f Mon Sep 17 00:00:00 2001 From: vsun757 <52419532+vsun757@users.noreply.github.com> Date: Tue, 11 Jul 2023 15:18:45 -0400 Subject: [PATCH 093/159] Update navlayers.rst --- docs/navlayers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 2a06e982..798c47f6 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -24,7 +24,7 @@ and version 4.X layers, upgrading them to version 4.3. - Heading row 1, column 2 - Heading row 1, column 32 * - Row 1, column 1 - - + - hi - Row 1, column 3 * - Row 2, column 1 - Row 2, column 2 From 741b52b9c6562828fc47b6822dcf038831b7df5f Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 15:35:44 -0400 Subject: [PATCH 094/159] adding tables to navlayers --- docs/navlayers.rst | 49 +++++++++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 798c47f6..1120ee15 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -20,33 +20,34 @@ and version 4.X layers, upgrading them to version 4.3. :widths: 25 25 50 :header-rows: 1 - * - Heading row 1, column 11 - - Heading row 1, column 2 - - Heading row 1, column 32 - * - Row 1, column 1 - - hi - - Row 1, column 3 - * - Row 2, column 1 - - Row 2, column 2 - - Row 2, column 3 - * - row 1 - - row 2 - - row 3 - -`filter `_ | Implements a basic `filter object `_. | -| `gradient `_ | Implements a basic `gradient object `_. | -| `layer `_ | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | -| `layout `_ | Implements a basic `layout object `_. | -| `legenditem `_ | Implements a basic `legenditem object `_. | -| `metadata `_ | Implements a basic `metadata object `_. | -| `technique `_ | Implements a basic `technique object `_. | -| `versions `_ | Implements a basic `versions object `_.| + * - + - + + * - script + - description + * - `filter `_ + - Implements a basic `filter object `_ + * - `gradient `_ + - Implements a basic `gradient object `_ + * - `layer `_ + - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. + * - `layout `_ + - Implements a basic `layout object `_ + * - `legenditem `_ + - Implements a basic `legenditem object `_ + * - `metadata `_ + - Implements a basic `metadata object `_ + * - `technique `_ + - Implements a basic `technique object `_ + * - `versions `_ + - Implements a basic `versions object `_ **Manipulator Scripts -| script | description | -|:-------|:------------| -| `layerops `_ | Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. | + * - script + - description + * - `layerops `_ + - Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. **Exporter Scripts From 24758ce1ac06809933c45cff3c89785519c9e6b2 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 15:43:46 -0400 Subject: [PATCH 095/159] adding more tables to navlayers --- docs/navlayers.rst | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 1120ee15..1c50ba75 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -20,9 +20,6 @@ and version 4.X layers, upgrading them to version 4.3. :widths: 25 25 50 :header-rows: 1 - * - - - - * - script - description * - `filter `_ @@ -51,16 +48,25 @@ and version 4.X layers, upgrading them to version 4.3. **Exporter Scripts -| script | description | -|:-------|:------------| -| `to_excel `_ | Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. | -| `to_svg `_ | Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export.| + * - script + - description + * - `to_excel `_ + - Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. + * - `to_svg `_ + - Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export. + + **Generator Scripts -`overview_generator `_. Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | -| `usage_generator `_ | Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | -| `sum_generator `_ . Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | + * - script + - description + * - `overview_generator `_ + - Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. + * - `usage_generator `_ + - Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. + * - `sum_generator `_ + - Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. **Utility Modules From 0ae8abbef757ad5f5c1e3f78d931738380b3231f Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 16:06:43 -0400 Subject: [PATCH 096/159] adding more tables part 2 --- docs/navlayers.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 1c50ba75..262588e4 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -70,12 +70,16 @@ and version 4.X layers, upgrading them to version 4.3. **Utility Modules -| script | description | -|:-------|:------------| -| `excel_templates `_ | Provides a means by which to convert a matrix into a clean excel matrix template. | -| `matrix_gen `_ | Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). | -| `svg_templates `_ | Provides a means by which to convert a layer file into a marked up svg file. | -| `svg_objects `_ | Provides raw templates and supporting functionality for generating svg objects. | + * - script + - description + * - `excel_templates `_ + - Provides a means by which to convert a matrix into a clean excel matrix template. + * - `matrix_gen `_ + - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). + * - `svg_templates `_ + - Provides a means by which to convert a layer file into a marked up svg file. + * - `svg_objects `_ + - Provides raw templates and supporting functionality for generating svg objects. **Command Line Tools From b42811fc891eb02ed4f5c844cb563fdc88d02300 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 17:03:24 -0400 Subject: [PATCH 097/159] adding more tables part 3 --- docs/navlayers.rst | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 262588e4..503dcb5c 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -83,10 +83,12 @@ and version 4.X layers, upgrading them to version 4.3. **Command Line Tools -| script | description | -|:-------|:------------| -| `layerExporter_cli.py `_ | A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. | -| `layerGenerator_cli.py `_ | A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. | + * - script + - description + * - `layerExporter_cli.py `_ + - A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. + * - `layerGenerator_cli.py `_ + - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. **Layer @@ -95,14 +97,20 @@ It is the primary interface through which other Layer-related classes defined in The Layer class API and a usage example are below. The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. -| method [x = Layer()]| description | -|:-------|:------------| -| `x.from_str(_input_)` | Loads an ATT&CK layer from a string representation of a json layer. | -| `x.from_dict(_input_)` | Loads an ATT&CK layer from a dictionary. | -| `x.from_file(_filepath_)` | Loads an ATT&CK layer from a file location specified by the _filepath_. | -| `x.to_file(_filepath_)` | Saves the current state of the loaded ATT&CK layer to a json file denoted by the _filepath_. | -| `x.to_dict()` | Returns a representation of the current ATT&CK layer object as a dictionary. | -| `x.to_str()` | Returns a representation of the current ATT&CK layer object as a string representation of a dictionary. | + * - method [x = Layer()] + - description + * - `x.from_str(_input_)` + - Loads an ATT&CK layer from a string representation of a json layer. + * - `x.from_dict(_input_)` + - Loads an ATT&CK layer from a dictionary. + * - `x.from_file(_filepath_)` + - Loads an ATT&CK layer from a file location specified by the _filepath_. + * - `x.to_file(_filepath_)` + - Saves the current state of the loaded ATT&CK layer to a json file denoted by the _filepath_. + * - `x.to_dict()` + - Returns a representation of the current ATT&CK layer object as a dictionary. + * - `x.to_str()` + - Returns a representation of the current ATT&CK layer object as a string representation of a dictionary. Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found `here `_. From 0fbc19f78dd66a91002a3c6edb79f4c0c7ad5cc9 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 17:39:08 -0400 Subject: [PATCH 098/159] adding more tables part 4 --- docs/navlayers.rst | 96 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 87 insertions(+), 9 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 503dcb5c..c3e3208b 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -314,15 +314,56 @@ the currently loaded configuration can be interacted with at `ToSvg().config`. The configuration can also be populated from a json file using the `.load_from_file(filename="path/to/file.json")` method, or stored to one using the `.save_to_file(filename="path/to/file.json)` method. -| attribute| description | type | default value | -|:-------|:------------|:------------|:------------| -| width | Desired SVG width | number | 8.5 | -| height | Desired SVG height | number | 11 | -| headerHeight | Desired Header Block height | number | 1 | -| unit | SVG measurement units (qualifies width, height, etc.) - "in", "cm", "px", "em", or "pt"| string | "in" | -| showSubtechniques | Display form for subtechniques - "all", "expanded" (decided by layer), or "none" | string | "expanded" | -| font | What font style to use - "serif", "sans-serif", or "monospace" | string | "sans-serif" | -| tableBorderColor | Hex color to use for the technique borders | string | "#6B7279" | + * - attribute + - description + - type + - default value + * - width + - Desired SVG width + - number + - 8.5 + * - height + - Desired SVG height + - number + - 11 + * - headerHeight + - Desired Header Block height + - number + - 1 + * - unit + - SVG measurement units (qualifies width, height, etc.) - "in", "cm", "px", "em", or "pt" + - string + - "in" + * - showSubtechniques + - Display form for subtechniques - "all", "expanded" (decided by layer), or "none" + - string + - "expanded" + * - font + - What font style to use - "serif", "sans-serif", or "monospace" + - string + - "sans-serif" + * - tableBorderColor + - Hex color to use for the technique borders + - string + - "#6B7279" + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + + | showHeader | Whether or not to show Header Blocks | bool | True | | legendDocked | Whether or not the legend should be docked | bool | True | | legendX | Where to place the legend on the x axis if not docked | number | 0 | @@ -335,6 +376,43 @@ or stored to one using the `.save_to_file(filename="path/to/file.json)` method. | showAbout | Whether or not to show the About Header Block | bool | True | | border | What default border width to use | number | 0.104 | + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + * - + - + - + - + **.to_svg() Method .. code-block:: python From f014de55aa715cf40da8169665e569194f63ae8f Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 17:52:52 -0400 Subject: [PATCH 099/159] adding more tables final to navlayers --- docs/navlayers.rst | 110 ++++++++++++++++++--------------------------- 1 file changed, 44 insertions(+), 66 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index c3e3208b..4a36938a 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -346,72 +346,50 @@ or stored to one using the `.save_to_file(filename="path/to/file.json)` method. - Hex color to use for the technique borders - string - "#6B7279" - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - - - -| showHeader | Whether or not to show Header Blocks | bool | True | -| legendDocked | Whether or not the legend should be docked | bool | True | -| legendX | Where to place the legend on the x axis if not docked | number | 0 | -| legendY | Where to place the legend on the y axis if not docked | number | 1 | -| legendWidth | Width of the legend if not docked | number | 2 | -| legendHeight | Height of the legend if not docked | number | 1 | -| showLegend | Whether or not to show the legend | bool | True | -| showFilters | Whether or not to show the Filter Header Block | bool | True | -| showDomain | Whether or not to show the Domain and Version Header Block | bool | True | -| showAbout | Whether or not to show the About Header Block | bool | True | -| border | What default border width to use | number | 0.104 | - - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - - * - - - - - - - + * - showHeader + - Whether or not to show Header Blocks + - bool + - True + * - legendDocked + - Whether or not the legend should be docked + - bool + - True + * - legendX + - Where to place the legend on the x axis if not docked + - number + - 0 + * - legendY + - Where to place the legend on the y axis if not docked + - number + - 1 + * - legendWidth + - Width of the legend if not docked + - number + - 2 + * - legendHeight + - Height of the legend if not docked + - number + - 1 + * - showLegend + - Whether or not to show the legend + - bool + - True + * - showFilters + - Whether or not to show the Filter Header Block + - bool + - True + * - showDomain + - Whether or not to show the Domain and Version Header Block + - bool + - True + * - showAbout + - Whether or not to show the About Header Block + - bool + - True + * - border + - What default border width to use + - number + - 0.104 **.to_svg() Method From 64b543dcbd6d96427a41096862554462220ff901 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 11 Jul 2023 17:55:15 -0400 Subject: [PATCH 100/159] adding more tables final to collections --- docs/collections.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index 88af02d2..7b04de38 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,9 +9,14 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -`index_to_markdown `_ | Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below.... -`collection_to_index `_ Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below.| -`stix_to_collection `_ | Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below.| + * - script + - description + * - `index_to_markdown `_ + - Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below. + * - `collection_to_index `_ + - Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below. + * - `stix_to_collection `_ + - Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below. **index_to_markdown.py** From 588e86b0c0f8387c18d522a8506c286d5702b6a5 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 12 Jul 2023 17:21:57 -0400 Subject: [PATCH 101/159] added another setof tables to diffstix --- docs/diffinstix.rst | 71 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 54 insertions(+), 17 deletions(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index c4118821..35b6c128 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -92,16 +92,34 @@ A brief explanation of key pieces can be found below. * For domain changes, they are broken down by object type, e.g. `techniques` or `mitigations`. * The following table helps break down the change types that are currently tracked. -| Field | Type | Description | -|-------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| `additions` | array[object] | ATT&CK objects which are only present in the new STIX data. | -| `major_version_changes` | array[object] | ATT&CK objects that have a major version change. (e.g. 1.0 → 2.0). | -| `minor_version_changes` | array[object] | ATT&CK objects that have a minor version change. (e.g. 1.0 → 1.1). | -| `other_version_changes` | array[object] | ATT&CK objects that have a version change of any other kind. (e.g. 1.0 → 1.3). These are unintended, but can be found in previous releases. | -| `patches` | array[object] | ATT&CK objects that have been patched while keeping the version the same. | -| `revocations` | array[object] | ATT&CK objects which are revoked by a different object. | -| `deprecations` | array[object] | ATT&CK objects which are deprecated and no longer in use, and not replaced. | -| `deletions` | array[object] | ATT&CK objects which are no longer found in the STIX data. This should almost never happen. | + * - field + - type + - description + * - `additions` + -array[object] + - ATT&CK objects which are only present in the new STIX data. + * - `major_version_changes`` + - array[object] + - ATT&CK objects that have a major version change. (e.g. 1.0 → 2.0). + * - `minor_version_changes` + - array[object] + - ATT&CK objects that have a minor version change. (e.g. 1.0 → 1.1). + * - `other_version_changes` + - array[object] + - array[object] | ATT&CK objects that have a version change of any other kind. (e.g. 1.0 → 1.3). These are unintended, but can be found in previous releases. + * - `patches` + - array[object] + - ATT&CK objects that have been patched while keeping the version the same. + * - `revocations` + - array[object] + - ATT&CK objects which are revoked by a different object. + * - `deprecations` + - array[object] + - ATT&CK objects which are deprecated and no longer in use, and not replaced. + * - `deletions` + - array[object + - ATT&CK objects which are no longer found in the STIX data. This should almost never happen. + **Changed Objects @@ -111,10 +129,29 @@ That is because there are a few fields that have been added in some cases depend For example, objects that are brand new do not have `previous_version` available to them. The following table lists the extra fields that can be found in objects in the changelog. -| Field | Required | Type | Description | -|----------------------------|----------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `changelog_mitigations` | false | object | Three lists for `shared`, `new`, and `dropped` for Mitigations that are related to a Technique between versions. | -| `changelog_detections` | false | object | Three lists for `shared`, `new`, and `dropped` for Detections that are related to a Technique between versions. || `description_change_table` | false | string | HTML rendering of a table that displays the differences between descriptions for an ATT&CK object. | -| `detailed_diff` | false | string | A python DeepDiff object that has been JSON serialized which represents STIX changes for an ATT&CK object between releases. | -| `previous_version` | false | string | If the object existed in the previous release, then it denotes the version the object was in the previous release. | -| `version_change` | false | string | If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' | + * - Field + - Required + - Type + - Description + * - `changelog_mitigations` + - false + - object + - Three lists for `shared`, `new`, and `dropped` for Mitigations that are related to a Technique between versions. + * - `changelog_detections` + - false + - object + - HTML rendering of a table that displays the differences between descriptions for an ATT&CK object. + * - `detailed_diff` + - false + - string + - A python DeepDiff object that has been JSON serialized which represents STIX changes for an ATT&CK object between releases. + * - `previous_version` + - false + - string + - If the object existed in the previous release, then it denotes the version the object was in the previous release. + * - `version_change` + - false + - string + - If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' + + From 9a1bbde443bc50cf3ee74e2e2ef1fe53dac3e7e0 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 12 Jul 2023 17:57:03 -0400 Subject: [PATCH 102/159] added another setof tables to attacktoexcel --- docs/attacktoexcel.rst | 57 +++++++++++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 62e3b802..f50d72e3 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -43,27 +43,54 @@ Example execution targeting a specific domain and version: attackToExcel provides the means by which to convert/extract the ATT&CK STIX data to Excel spreadsheets. A brief overview of the available methods follows. -| method name | arguments | usage | -|:------------|:----------|:------| -|get_stix_data|`domain`: the domain of ATT&CK to fetch data from
`version`: optional parameter indicating which version to fetch data from (such as "v8.1"). If omitted retrieves the most recent version of ATT&CK.
`remote`: optional parameter that provides a URL of a remote ATT&CK Workbench instance to grab data from.| Retrieves the ATT&CK STIX data for the specified version and returns it as a MemoryStore object| -|build_dataframes| `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to| Builds a Pandas DataFrame collection as a dictionary, with keys for each type, based on the ATT&CK data provided| -|write_excel| `dataframes`: pandas DataFrame dictionary (generated by build_dataframes)
`domain`: domain of ATT&CK that `dataframes` corresponds to
`version`: optional parameter indicating which version of ATT&CK is in use
`output_dir`: optional parameter specifying output directory| Writes out DataFrame based ATT&CK data to excel files| -|export| `domain`: the domain of ATT&CK to download
`version`: optional parameter specifying which version of ATT&CK to download
`output_dir`: optional parameter specifying output directory| Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets | + + * - method name + - arguments + - usage + * - get_stix_data + - `domain`: the domain of ATT&CK to fetch data from
`version`: optional parameter indicating which version to fetch data from (such as "v8.1"). If omitted retrieves the most recent version of ATT&CK.
`remote`: optional parameter that provides a URL of a remote ATT&CK Workbench instance to grab data from. + - Retrieves the ATT&CK STIX data for the specified version and returns it as a MemoryStore object + * - build_dataframes + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Builds a Pandas DataFrame collection as a dictionary, with keys for each type, based on the ATT&CK data provided + * - write_excel + - `dataframes`: pandas DataFrame dictionary (generated by build_dataframes)
`domain`: domain of ATT&CK that `dataframes` corresponds to
`version`: optional parameter indicating which version of ATT&CK is in use
`output_dir`: optional parameter specifying output directory + - Writes out DataFrame based ATT&CK data to excel files + * - export + - `domain`: the domain of ATT&CK to download
`version`: optional parameter specifying which version of ATT&CK to download
`output_dir`: optional parameter specifying output directory + - Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets **stixToDf** stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for processing. A brief overview of these methods follows. -| method name | arguments | usage | -|:------------|:----------|:------| -|techniquesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX techniques from the provided data and returns corresponding Pandas DataFrames.| -|tacticsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX tactics from the provided data and returns corresponding Pandas DataFrames.| -|softwareToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX software from the provided data and returns corresponding Pandas DataFrames.| -|groupsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX groups from the provided data and returns corresponding Pandas DataFrames.| -|mitigationsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX mitigations from the provided data and returns corresponding Pandas DataFrames.| -|relationshipsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX relationships from the provided data and returns corresponding Pandas DataFrames.| -|matricesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]`| + + * - method name + - arguments + - usage + * - techniquesToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX techniques from the provided data and returns corresponding Pandas DataFrames. + * - tacticsToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX tactics from the provided data and returns corresponding Pandas DataFrames. + * - softwareToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX software from the provided data and returns corresponding Pandas DataFrames. + * - groupsToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX groups from the provided data and returns corresponding Pandas DataFrames. + * - mitigationsToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX mitigations from the provided data and returns corresponding Pandas DataFrames. + * - relationshipsToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX relationships from the provided data and returns corresponding Pandas DataFrames. + * - matricesToDf + - `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to + - Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]` + **Spreadsheet format** From 4c758b330afba8b80467b026b604a7d8295c86c5 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 13 Jul 2023 08:41:32 -0400 Subject: [PATCH 103/159] added table syntax --- docs/attacktoexcel.rst | 7 +++++++ docs/collections.rst | 5 +++++ docs/diffinstix.rst | 8 ++++++++ docs/navlayers.rst | 28 ++++++++++++++++++++++++++++ 4 files changed, 48 insertions(+) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index f50d72e3..e9766577 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -44,6 +44,10 @@ attackToExcel provides the means by which to convert/extract the ATT&CK STIX dat overview of the available methods follows. +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - method name - arguments - usage @@ -65,6 +69,9 @@ overview of the available methods follows. stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for processing. A brief overview of these methods follows. +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 * - method name - arguments diff --git a/docs/collections.rst b/docs/collections.rst index 7b04de38..8825853d 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -9,6 +9,11 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** + +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - script - description * - `index_to_markdown `_ diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 35b6c128..88352a48 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -92,6 +92,10 @@ A brief explanation of key pieces can be found below. * For domain changes, they are broken down by object type, e.g. `techniques` or `mitigations`. * The following table helps break down the change types that are currently tracked. +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - field - type - description @@ -129,6 +133,10 @@ That is because there are a few fields that have been added in some cases depend For example, objects that are brand new do not have `previous_version` available to them. The following table lists the extra fields that can be found in objects in the changelog. +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - Field - Required - Type diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 4a36938a..1203d167 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -41,6 +41,10 @@ and version 4.X layers, upgrading them to version 4.3. **Manipulator Scripts +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - script - description * - `layerops `_ @@ -48,6 +52,10 @@ and version 4.X layers, upgrading them to version 4.3. **Exporter Scripts +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - script - description * - `to_excel `_ @@ -59,6 +67,10 @@ and version 4.X layers, upgrading them to version 4.3. **Generator Scripts +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - script - description * - `overview_generator `_ @@ -70,6 +82,10 @@ and version 4.X layers, upgrading them to version 4.3. **Utility Modules +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - script - description * - `excel_templates `_ @@ -83,6 +99,10 @@ and version 4.X layers, upgrading them to version 4.3. **Command Line Tools +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - script - description * - `layerExporter_cli.py `_ @@ -97,6 +117,10 @@ It is the primary interface through which other Layer-related classes defined in The Layer class API and a usage example are below. The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - method [x = Layer()] - description * - `x.from_str(_input_)` @@ -314,6 +338,10 @@ the currently loaded configuration can be interacted with at `ToSvg().config`. The configuration can also be populated from a json file using the `.load_from_file(filename="path/to/file.json")` method, or stored to one using the `.save_to_file(filename="path/to/file.json)` method. +.. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + * - attribute - description - type From 4f9fb8eecab2057326dd60fcca2f7f8c54241d76 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 13 Jul 2023 08:44:39 -0400 Subject: [PATCH 104/159] added table syntax part 2 --- docs/navlayers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 1203d167..14d8a776 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -100,7 +100,7 @@ and version 4.X layers, upgrading them to version 4.3. **Command Line Tools .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 100 :header-rows: 1 * - script From 34de82a33232496948a9dd1bda479375dc4d9bbc Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 13 Jul 2023 11:44:19 -0400 Subject: [PATCH 105/159] added table syntax to nayvlayers --- docs/navlayers.rst | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 14d8a776..9e9a163c 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -102,13 +102,12 @@ and version 4.X layers, upgrading them to version 4.3. .. list-table:: Title :widths: 50 50 100 :header-rows: 1 - - * - script - - description - * - `layerExporter_cli.py `_ - - A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. - * - `layerGenerator_cli.py `_ - - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. + * - script + - description + * - `layerExporter_cli.py `_ + - A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. + * - `layerGenerator_cli.py `_ + - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. **Layer From 23ee86532e7d7e195f54ba91a83658a5e4c3565e Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 13 Jul 2023 11:51:09 -0400 Subject: [PATCH 106/159] added table syntax to navlayers --- docs/navlayers.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 9e9a163c..f0bfec81 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -86,15 +86,15 @@ and version 4.X layers, upgrading them to version 4.3. :widths: 25 25 50 :header-rows: 1 - * - script - - description - * - `excel_templates `_ - - Provides a means by which to convert a matrix into a clean excel matrix template. - * - `matrix_gen `_ - - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). - * - `svg_templates `_ - - Provides a means by which to convert a layer file into a marked up svg file. - * - `svg_objects `_ + * - script + - description + * - `excel_templates `_ + - Provides a means by which to convert a matrix into a clean excel matrix template. + * - `matrix_gen `_ + - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). + * - `svg_templates `_ + - Provides a means by which to convert a layer file into a marked up svg file. + * - `svg_objects `_ - Provides raw templates and supporting functionality for generating svg objects. **Command Line Tools From cd76f3d4417be07a74c0cb5081430f57d9a78f34 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 13 Jul 2023 13:51:26 -0400 Subject: [PATCH 107/159] added table syntax to navlayers part 2 --- docs/navlayers.rst | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index f0bfec81..2450a507 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -82,20 +82,20 @@ and version 4.X layers, upgrading them to version 4.3. **Utility Modules -.. list-table:: Title - :widths: 25 25 50 - :header-rows: 1 - - * - script - - description - * - `excel_templates `_ - - Provides a means by which to convert a matrix into a clean excel matrix template. - * - `matrix_gen `_ - - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). - * - `svg_templates `_ - - Provides a means by which to convert a layer file into a marked up svg file. - * - `svg_objects `_ - - Provides raw templates and supporting functionality for generating svg objects. + .. list-table:: Title + :widths: 25 25 50 + :header-rows: 1 + + * - script + - description + * - `excel_templates `_ + - Provides a means by which to convert a matrix into a clean excel matrix template. + * - `matrix_gen `_ + - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). + * - `svg_templates `_ + - Provides a means by which to convert a layer file into a marked up svg file. + * - `svg_objects `_ + - Provides raw templates and supporting functionality for generating svg objects. **Command Line Tools From 7a49d1d2227e83f7d64610cdb9e3c193de03642d Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 13:45:42 -0400 Subject: [PATCH 108/159] fixed column spacing --- docs/attacktoexcel.rst | 4 ++-- docs/collections.rst | 2 +- docs/diffinstix.rst | 4 ++-- docs/navlayers.rst | 16 ++++++++-------- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index e9766577..50d82ddb 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -45,7 +45,7 @@ overview of the available methods follows. .. list-table:: Title - :widths: 25 25 50 + :widths: 33 33 34 :header-rows: 1 * - method name @@ -70,7 +70,7 @@ stixToDf provides various methods to process and manipulate the STIX data in ord processing. A brief overview of these methods follows. .. list-table:: Title - :widths: 25 25 50 + :widths: 33 33 34 :header-rows: 1 * - method name diff --git a/docs/collections.rst b/docs/collections.rst index 8825853d..f6503d59 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -11,7 +11,7 @@ For more information about ATT&CK collections, see the corresponding .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - script diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 88352a48..e1b33c05 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -93,7 +93,7 @@ A brief explanation of key pieces can be found below. * The following table helps break down the change types that are currently tracked. .. list-table:: Title - :widths: 25 25 50 + :widths: 33 33 34 :header-rows: 1 * - field @@ -134,7 +134,7 @@ For example, objects that are brand new do not have `previous_version` available The following table lists the extra fields that can be found in objects in the changelog. .. list-table:: Title - :widths: 25 25 50 + :widths: 25 25 25 25 :header-rows: 1 * - Field diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 2450a507..976d5b97 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -17,7 +17,7 @@ and version 4.X layers, upgrading them to version 4.3. ## Core Modules .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - script @@ -42,7 +42,7 @@ and version 4.X layers, upgrading them to version 4.3. **Manipulator Scripts .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - script @@ -53,7 +53,7 @@ and version 4.X layers, upgrading them to version 4.3. **Exporter Scripts .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - script @@ -68,7 +68,7 @@ and version 4.X layers, upgrading them to version 4.3. **Generator Scripts .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - script @@ -83,7 +83,7 @@ and version 4.X layers, upgrading them to version 4.3. **Utility Modules .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - script @@ -100,7 +100,7 @@ and version 4.X layers, upgrading them to version 4.3. **Command Line Tools .. list-table:: Title - :widths: 50 50 100 + :widths: 50 50 :header-rows: 1 * - script - description @@ -117,7 +117,7 @@ The Layer class API and a usage example are below. The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. .. list-table:: Title - :widths: 25 25 50 + :widths: 50 50 :header-rows: 1 * - method [x = Layer()] @@ -338,7 +338,7 @@ The configuration can also be populated from a json file using the `.load_from_f or stored to one using the `.save_to_file(filename="path/to/file.json)` method. .. list-table:: Title - :widths: 25 25 50 + :widths: 25 25 25 25 :header-rows: 1 * - attribute From 633d3563b538b4967a41a57b46546aa5e26267b2 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 14:14:32 -0400 Subject: [PATCH 109/159] fixed column spacing part 2 --- docs/attacktoexcel.rst | 2 +- docs/collections.rst | 2 +- docs/diffinstix.rst | 2 +- docs/navlayers.rst | 16 ++++++++-------- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 50d82ddb..cfa6d16e 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -69,7 +69,7 @@ overview of the available methods follows. stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for processing. A brief overview of these methods follows. -.. list-table:: Title +.. list-table:: :widths: 33 33 34 :header-rows: 1 diff --git a/docs/collections.rst b/docs/collections.rst index f6503d59..04bc730a 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -10,7 +10,7 @@ For more information about ATT&CK collections, see the corresponding **Collections Scripts** -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index e1b33c05..2fe021b6 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -133,7 +133,7 @@ That is because there are a few fields that have been added in some cases depend For example, objects that are brand new do not have `previous_version` available to them. The following table lists the extra fields that can be found in objects in the changelog. -.. list-table:: Title +.. list-table:: :widths: 25 25 25 25 :header-rows: 1 diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 976d5b97..71a5b419 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -16,7 +16,7 @@ and version 4.X layers, upgrading them to version 4.3. ## Core Modules -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 @@ -41,7 +41,7 @@ and version 4.X layers, upgrading them to version 4.3. **Manipulator Scripts -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 @@ -52,7 +52,7 @@ and version 4.X layers, upgrading them to version 4.3. **Exporter Scripts -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 @@ -67,7 +67,7 @@ and version 4.X layers, upgrading them to version 4.3. **Generator Scripts -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 @@ -82,7 +82,7 @@ and version 4.X layers, upgrading them to version 4.3. **Utility Modules - .. list-table:: Title + .. list-table:: :widths: 50 50 :header-rows: 1 @@ -99,7 +99,7 @@ and version 4.X layers, upgrading them to version 4.3. **Command Line Tools -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 * - script @@ -116,7 +116,7 @@ It is the primary interface through which other Layer-related classes defined in The Layer class API and a usage example are below. The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. -.. list-table:: Title +.. list-table:: :widths: 50 50 :header-rows: 1 @@ -337,7 +337,7 @@ the currently loaded configuration can be interacted with at `ToSvg().config`. The configuration can also be populated from a json file using the `.load_from_file(filename="path/to/file.json")` method, or stored to one using the `.save_to_file(filename="path/to/file.json)` method. -.. list-table:: Title +.. list-table:: :widths: 25 25 25 25 :header-rows: 1 From 8b918053e2414db296a4dd4b4d488e99ad3e97d5 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 14:24:29 -0400 Subject: [PATCH 110/159] fixed table syntax --- docs/navlayers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 71a5b419..e7e212e9 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -95,7 +95,7 @@ and version 4.X layers, upgrading them to version 4.3. * - `svg_templates `_ - Provides a means by which to convert a layer file into a marked up svg file. * - `svg_objects `_ - - Provides raw templates and supporting functionality for generating svg objects. + - Provides raw templates and supporting functionality for generating svg objects. **Command Line Tools From 9e087a1bcc86e69fae01d3dd6c302a7fa42b5acd Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 14:28:09 -0400 Subject: [PATCH 111/159] fixed table syntax part 2 --- docs/navlayers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index e7e212e9..f9bb04c5 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -82,7 +82,7 @@ and version 4.X layers, upgrading them to version 4.3. **Utility Modules - .. list-table:: +.. list-table:: :widths: 50 50 :header-rows: 1 From 37018abe4bc2ff57d320998b64c151e9583ff958 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 14:51:09 -0400 Subject: [PATCH 112/159] fixed table syntax part 3 --- docs/navlayers.rst | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index f9bb04c5..c60cd00e 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -86,28 +86,28 @@ and version 4.X layers, upgrading them to version 4.3. :widths: 50 50 :header-rows: 1 - * - script - - description - * - `excel_templates `_ - - Provides a means by which to convert a matrix into a clean excel matrix template. - * - `matrix_gen `_ - - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). - * - `svg_templates `_ - - Provides a means by which to convert a layer file into a marked up svg file. - * - `svg_objects `_ - - Provides raw templates and supporting functionality for generating svg objects. + * - script + - description + * - `excel_templates `_ + - Provides a means by which to convert a matrix into a clean excel matrix template. + * - `matrix_gen `_ + - Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). + * - `svg_templates `_ + - Provides a means by which to convert a layer file into a marked up svg file. + * - `svg_objects `_ + - Provides raw templates and supporting functionality for generating svg objects. **Command Line Tools .. list-table:: :widths: 50 50 :header-rows: 1 - * - script - - description - * - `layerExporter_cli.py `_ - - A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. - * - `layerGenerator_cli.py `_ - - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. + * - script + - description + * - `layerExporter_cli.py `_ + - A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. + * - `layerGenerator_cli.py `_ + - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. **Layer From 3ed10513bf08efd54f2fd6d5bb47b9af8262d32b Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 15:14:29 -0400 Subject: [PATCH 113/159] fixed navlayer syntax part 3 --- docs/navlayers.rst | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index c60cd00e..0f5c9c6d 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -39,7 +39,7 @@ and version 4.X layers, upgrading them to version 4.3. * - `versions `_ - Implements a basic `versions object `_ -**Manipulator Scripts +**Manipulator Scripts** .. list-table:: :widths: 50 50 @@ -50,7 +50,7 @@ and version 4.X layers, upgrading them to version 4.3. * - `layerops `_ - Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. -**Exporter Scripts +**Exporter Scripts** .. list-table:: :widths: 50 50 @@ -65,7 +65,7 @@ and version 4.X layers, upgrading them to version 4.3. -**Generator Scripts +**Generator Scripts** .. list-table:: :widths: 50 50 @@ -80,7 +80,7 @@ and version 4.X layers, upgrading them to version 4.3. * - `sum_generator `_ - Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. -**Utility Modules +**Utility Modules** .. list-table:: :widths: 50 50 @@ -97,7 +97,7 @@ and version 4.X layers, upgrading them to version 4.3. * - `svg_objects `_ - Provides raw templates and supporting functionality for generating svg objects. -**Command Line Tools +**Command Line Tools** .. list-table:: :widths: 50 50 @@ -109,7 +109,7 @@ and version 4.X layers, upgrading them to version 4.3. * - `layerGenerator_cli.py `_ - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. -**Layer +**Layer** The `Layer` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. It is the primary interface through which other Layer-related classes defined in the core module should be used. @@ -173,14 +173,14 @@ Examples on how to create a layer programmatically, as opposed to loading it fro layer3.from_file(example_layer_location) # Load layer data from a file into existing layer object -**layerops.py +**layerops.py** `Layerops.py` provides the `LayerOps` class, which is a way to combine layer files in an automated way, using user defined lambda functions. Each LayerOps instance, when created, ingests the provided lambda functions, and stores them for use. An existing `LayerOps` class can be used to combine layer files according to the initialized lambda using the process method. The breakdown of this two step process is documented in the table below, while examples of both the list and dictionary modes of operation can be found below. -**# LayerOps() +**# LayerOps()** .. code-block:: python @@ -201,7 +201,7 @@ The process method applies the lambda functions stored during initialization to _data_ must be either a list or a dictionary of Layer objects, and is expected to match the format of the lambda equations provided during initialization. `default_values` is an optional dictionary argument that overrides the currently stored default values with new ones for this specific processing operation. -**# Example Usage +**Example Usage** .. code-block:: python from mitreattack.navlayers.manipulators.layerops import LayerOps @@ -250,13 +250,13 @@ _data_ must be either a list or a dictionary of Layer objects, and is expected t out_layer6.to_file("C:\demo_layer6.json") # Save combined comment layer to file -**to_excel.py +**to_excel.py** `to_excel.py` provides the `ToExcel` class, which is a way to export an existing layer file as an Excel spreadsheet. The `ToExcel` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or retrieving data from an ATT&CK Workbench instance. -**ToExcel() +**ToExcel()** .. code-block:: python x = ToExcel(domain='enterprise', source='taxii', resource=None) @@ -271,7 +271,7 @@ it should utilize a remote ATT&CK Workbench instance. The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, or if the source is set to `remote`, in which case it should be the url of a ATT&CK workbench instance. -**.to_xlsx() Method +**.to_xlsx() Method** .. code-block:: python x.to_xlsx(layerInit=layer, filepath="layer.xlsx") @@ -279,7 +279,7 @@ x.to_xlsx(layerInit=layer, filepath="layer.xlsx") The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. -**# Example Usage +**Example Usage** .. code-block:: python from mitreattack.navlayers import Layer @@ -298,14 +298,14 @@ from mitreattack.navlayers import ToExcel t3 = ToExcel(domain='ics', source='remote', resource=workbench_url) -**to_svg.py +**to_svg.py** `to_svg.py` provides the `ToSvg` class, which is a way to export an existing layer file as an SVG image file. The `ToSvg` class, like the `ToExcel` class, has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or utilizing a remote ATT&CK Workbench instance. -**ToSvg() +**ToSvg()** .. code-block:: python x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) @@ -321,7 +321,7 @@ or if the source is set to `remote`, in which case it should be the url of an AT The `config` parameter is an optional `SVGConfig` object that can be used to configure the export as desired. If not provided, the configuration for the export will be set to default values. -**SVGConfig() +**SVGConfig()** .. code-block:: python y = SVGConfig(width=8.5, height=11, headerHeight=1, unit="in", showSubtechniques="expanded", @@ -418,14 +418,14 @@ or stored to one using the `.save_to_file(filename="path/to/file.json)` method. - number - 0.104 -**.to_svg() Method +**.to_svg() Method** .. code-block:: python x.to_svg(layerInit=layer, filepath="layer.svg") The `to_svg` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. -**# Example Usage +**# Example Usage** .. code-block:: python from mitreattack.navlayers import Layer From 53b6180669915aa7f81250b93f7419e19a0b567e Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 15:14:57 -0400 Subject: [PATCH 114/159] fixed navlayer syntax part 4 --- docs/diffinstix.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 2fe021b6..6ba0c45b 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -125,7 +125,7 @@ A brief explanation of key pieces can be found below. - ATT&CK objects which are no longer found in the STIX data. This should almost never happen. -**Changed Objects +**Changed Objects** The bulk of the changelog file consists of lists of JSON objects. If you are familiar with reading the STIX format, they may look famliar, yet a little "off". From 5028544103792b86848970a206cc29ab91274404 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 14 Jul 2023 15:25:53 -0400 Subject: [PATCH 115/159] fixed navlayer commandline table syntax --- docs/navlayers.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 0f5c9c6d..bb2d4474 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -13,9 +13,6 @@ and version 4.X layers, upgrading them to version 4.3. **Core Modules** - -## Core Modules - .. list-table:: :widths: 50 50 :header-rows: 1 @@ -100,8 +97,9 @@ and version 4.X layers, upgrading them to version 4.3. **Command Line Tools** .. list-table:: - :widths: 50 50 + :widths: 50 50 :header-rows: 1 + * - script - description * - `layerExporter_cli.py `_ From e82b1cab96281b5880e79ad675bccca46c473480 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 13:40:54 -0400 Subject: [PATCH 116/159] visual improvements --- docs/index.rst | 22 ++++++---------------- docs/navlayers.rst | 6 +++--- 2 files changed, 9 insertions(+), 19 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index b543de1d..5de682ba 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -26,27 +26,17 @@ other modules in this library under "Additional Modules". mitre_attack_data/examples mitre_attack_data/custom_objects -.. toctree:: - :maxdepth: 1 - :caption: STIX Primer - - stix_primer/access-attack - - .. toctree:: :maxdepth: 1 :caption: Additional Modules - additional_modules/navlayers - additional_modules/attackToExcel - additional_modules/collections - additional_modules/diffStix + diffinstix + collections + navlayers + attacktoexcel .. toctree:: :maxdepth: 1 - :caption: Additional Modules part 2 + :caption: STIX Primer - diffinstix - collections - navlayers - attacktoexcel \ No newline at end of file + stix_primer/access-attack \ No newline at end of file diff --git a/docs/navlayers.rst b/docs/navlayers.rst index bb2d4474..8cf49c74 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -136,7 +136,7 @@ The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found `here `_. -**Example Usage +**Example Usage** .. code-block:: python example_layer3_dict = { @@ -178,7 +178,7 @@ Each LayerOps instance, when created, ingests the provided lambda functions, and An existing `LayerOps` class can be used to combine layer files according to the initialized lambda using the process method. The breakdown of this two step process is documented in the table below, while examples of both the list and dictionary modes of operation can be found below. -**# LayerOps()** +**LayerOps()** .. code-block:: python @@ -189,7 +189,7 @@ Each of the _inputs_ takes a lambda function that will be used to combine techni The one exception to this is _default_values_, which is an optional dictionary argument containing default values to provide the lambda functions if techniques of the combined layers are missing them. -****.process() Method +**.process() Method** .. code-block:: python x.process(data, default_values=default_values) From 9ea41a6c6073137639826c87361e8c15e076811b Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 13:47:22 -0400 Subject: [PATCH 117/159] fixing syntax --- docs/diffinstix.rst | 118 ++++++++++++++--------------- docs/stix_primer/access-attack.rst | 19 +++-- 2 files changed, 72 insertions(+), 65 deletions(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 6ba0c45b..9ac6c8d8 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -12,44 +12,44 @@ Run `diff_stix -h` for full usage instructions. Print full usage instructions: .. code:: bash -# You must run `pip install mitreattack-python` in order to access the diff_stix command - diff_stix --help - usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] - [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] - - Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. - - options: - -h, --help show this help message and exit - --old OLD Directory to load old STIX data from. - --new NEW Directory to load new STIX data from. - --domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...] - Which domains to report on. Choices (and defaults) are enterprise-attack, mobile-attack, ics-attack - --markdown-file MARKDOWN_FILE - Create a markdown file reporting changes. - --html-file HTML_FILE - Create HTML page from markdown content. - --html-file-detailed HTML_FILE_DETAILED - Create an HTML file reporting detailed changes. - --json-file JSON_FILE - Create a JSON file reporting changes. - --layers [LAYERS ...] - Create layer files showing changes in each domain expected order of filenames is 'enterprise', 'mobile', 'ics', 'pre attack'. If values are unspecified, defaults to output/January_2023_Updates_Enterprise.json, - output/January_2023_Updates_Mobile.json, output/January_2023_Updates_ICS.json, output/January_2023_Updates_Pre.json - --site_prefix SITE_PREFIX - Prefix links in markdown output, e.g. [prefix]/techniques/T1484 - --unchanged Show objects without changes in the markdown output - --use-mitre-cti Use content from the MITRE CTI repo for the -old data - --show-key Add a key explaining the change types to the markdown - --contributors Show new contributors between releases - --no-contributors Do not show new contributors between releases - -v, --verbose Print status messages + # You must run `pip install mitreattack-python` in order to access the diff_stix command + diff_stix --help + usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] + [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] + + Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. + + options: + -h, --help show this help message and exit + --old OLD Directory to load old STIX data from. + --new NEW Directory to load new STIX data from. + --domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...] + Which domains to report on. Choices (and defaults) are enterprise-attack, mobile-attack, ics-attack + --markdown-file MARKDOWN_FILE + Create a markdown file reporting changes. + --html-file HTML_FILE + Create HTML page from markdown content. + --html-file-detailed HTML_FILE_DETAILED + Create an HTML file reporting detailed changes. + --json-file JSON_FILE + Create a JSON file reporting changes. + --layers [LAYERS ...] + Create layer files showing changes in each domain expected order of filenames is 'enterprise', 'mobile', 'ics', 'pre attack'. If values are unspecified, defaults to output/January_2023_Updates_Enterprise.json, + output/January_2023_Updates_Mobile.json, output/January_2023_Updates_ICS.json, output/January_2023_Updates_Pre.json + --site_prefix SITE_PREFIX + Prefix links in markdown output, e.g. [prefix]/techniques/T1484 + --unchanged Show objects without changes in the markdown output + --use-mitre-cti Use content from the MITRE CTI repo for the -old data + --show-key Add a key explaining the change types to the markdown + --contributors Show new contributors between releases + --no-contributors Do not show new contributors between releases + -v, --verbose Print status messages Example execution: .. code:: bash -diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ + diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ **Changelog JSON format** @@ -59,33 +59,33 @@ This is the overall structure you can expect to find in the file. A brief explanation of key pieces can be found below. .. code-block:: json - { - "enterprise-attack": { - "techniques": { - "additions": [], - "major_version_changes": [], - "minor_version_changes": [], - "other_version_changes": [], - "patches": [], - "revocations": [], - "deprecations": [], - "deletions": [], + { + "enterprise-attack": { + "techniques": { + "additions": [], + "major_version_changes": [], + "minor_version_changes": [], + "other_version_changes": [], + "patches": [], + "revocations": [], + "deprecations": [], + "deletions": [], + }, + "software": {}, + "groups": {}, + "campaigns": {}, + "mitigations": {}, + "datasources": {}, + "datacomponents": {} }, - "software": {}, - "groups": {}, - "campaigns": {}, - "mitigations": {}, - "datasources": {}, - "datacomponents": {} - }, - "mobile-attack": {}, - "ics-attack": {}, - "new-contributors": [ - "Contributor A", - "Contributor B", - "Contributor C" - ] - } + "mobile-attack": {}, + "ics-attack": {}, + "new-contributors": [ + "Contributor A", + "Contributor B", + "Contributor C" + ] + } * The top-level objects include information about specific domains as well as `new-contributors`, which are only found in the newer ATT&CK release. diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index 038eba40..6cc5c436 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -56,7 +56,7 @@ Many users may opt to access the ATT&CK content via a local copy of the STIX dat - User can modify the ATT&CK content if desired - Downloaded copy is static, so updates to the ATT&CK catalog won't cause bugs in automated workflows. User can still manually update by cloning a fresh version of the data -#### Access via FileSystemSource +**Access via FileSystemSource** Each domain in this repo is formatted according to the [STIX2 FileSystem spec](https://stix2.readthedocs.io/en/latest/guide/filesystem.html). Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: @@ -91,11 +91,18 @@ Some users may instead prefer to access "live" ATT&CK content over the internet. Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII, the ATT&CK domains are represented as collections with static IDs: -| domain | collection ID | -|:-------|:--------------| -| `enterprise-attack` | `95ecc380-afe9-11e4-9b6c-751b66dd541e` | -| `mobile-attack` | `2f669986-b40b-4423-b720-4396ca6a462b` | -| `ics-attack` | `02c3ef24-9cd4-48f3-a99f-b74ce24f1d34` | +.. list-table:: + :widths: 50 50 + :header-rows: 1 + + * - domain + - collection ID + * - `enterprise-attack` + - `95ecc380-afe9-11e4-9b6c-751b66dd541e` + * - `mobile-attack` + - `2f669986-b40b-4423-b720-4396ca6a462b` + * - `ics-attack` + - `02c3ef24-9cd4-48f3-a99f-b74ce24f1d34` You can also get a list of available collection from the server directly: From 219d4910a3ae41df1b345510fc6125ee86e13629 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 13:50:17 -0400 Subject: [PATCH 118/159] fixing syntax part 2 --- docs/diffinstix.rst | 52 ++++++++++++++++++++++----------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 9ac6c8d8..ee4225cb 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -59,33 +59,33 @@ This is the overall structure you can expect to find in the file. A brief explanation of key pieces can be found below. .. code-block:: json - { - "enterprise-attack": { - "techniques": { - "additions": [], - "major_version_changes": [], - "minor_version_changes": [], - "other_version_changes": [], - "patches": [], - "revocations": [], - "deprecations": [], - "deletions": [], - }, - "software": {}, - "groups": {}, - "campaigns": {}, - "mitigations": {}, - "datasources": {}, - "datacomponents": {} + { + "enterprise-attack": { + "techniques": { + "additions": [], + "major_version_changes": [], + "minor_version_changes": [], + "other_version_changes": [], + "patches": [], + "revocations": [], + "deprecations": [], + "deletions": [], }, - "mobile-attack": {}, - "ics-attack": {}, - "new-contributors": [ - "Contributor A", - "Contributor B", - "Contributor C" - ] - } + "software": {}, + "groups": {}, + "campaigns": {}, + "mitigations": {}, + "datasources": {}, + "datacomponents": {} + }, + "mobile-attack": {}, + "ics-attack": {}, + "new-contributors": [ + "Contributor A", + "Contributor B", + "Contributor C" + ] + } * The top-level objects include information about specific domains as well as `new-contributors`, which are only found in the newer ATT&CK release. From d827727d139ee9263b88ba84093733f7df9027f9 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 14:05:45 -0400 Subject: [PATCH 119/159] fixing navlayer code syntax --- docs/diffinstix.rst | 2 ++ docs/navlayers.rst | 21 +++++++++++++++++---- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index ee4225cb..7e4acc00 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -12,6 +12,7 @@ Run `diff_stix -h` for full usage instructions. Print full usage instructions: .. code:: bash + # You must run `pip install mitreattack-python` in order to access the diff_stix command diff_stix --help usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] @@ -49,6 +50,7 @@ Print full usage instructions: Example execution: .. code:: bash + diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 8cf49c74..5c9cd024 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -139,6 +139,7 @@ Examples on how to create a layer programmatically, as opposed to loading it fro **Example Usage** .. code-block:: python + example_layer3_dict = { "name": "example layer", "version": "3.0", @@ -192,6 +193,7 @@ to provide the lambda functions if techniques of the combined layers are missing **.process() Method** .. code-block:: python + x.process(data, default_values=default_values) @@ -202,6 +204,7 @@ _data_ must be either a list or a dictionary of Layer objects, and is expected t **Example Usage** .. code-block:: python + from mitreattack.navlayers.manipulators.layerops import LayerOps from mitreattack.navlayers.core.layer import Layer @@ -257,6 +260,7 @@ Valid options include using live data from cti-taxii.mitre.org, using a local ST **ToExcel()** .. code-block:: python + x = ToExcel(domain='enterprise', source='taxii', resource=None) @@ -272,16 +276,18 @@ to a local stix bundle, or if the source is set to `remote`, in which case it sh **.to_xlsx() Method** .. code-block:: python + x.to_xlsx(layerInit=layer, filepath="layer.xlsx") -``` + The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. **Example Usage** .. code-block:: python -from mitreattack.navlayers import Layer -from mitreattack.navlayers import ToExcel + + from mitreattack.navlayers import Layer + from mitreattack.navlayers import ToExcel lay = Layer() lay.from_file("path/to/layer/file.json") @@ -306,6 +312,7 @@ Valid options include using live data from cti-taxii.mitre.org, using a local ST **ToSvg()** .. code-block:: python + x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) @@ -322,6 +329,7 @@ If not provided, the configuration for the export will be set to default values. **SVGConfig()** .. code-block:: python + y = SVGConfig(width=8.5, height=11, headerHeight=1, unit="in", showSubtechniques="expanded", font="sans-serif", tableBorderColor="#6B7279", showHeader=True, legendDocked=True, legendX=0, legendY=0, legendWidth=2, legendHeight=1, showLegend=True, showFilters=True, @@ -419,13 +427,15 @@ or stored to one using the `.save_to_file(filename="path/to/file.json)` method. **.to_svg() Method** .. code-block:: python + x.to_svg(layerInit=layer, filepath="layer.svg") The `to_svg` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. -**# Example Usage** +**Example Usage** .. code-block:: python + from mitreattack.navlayers import Layer from mitreattack.navlayers import ToSvg, SVGConfig @@ -456,6 +466,7 @@ of the specified ATT&CK object type (group, mitigation, etc.), and a comment tha **OverviewLayerGenerator()** .. code-block:: python + x = OverviewLayerGenerator(source='taxii', domain='enterprise', resource=None) @@ -564,6 +575,7 @@ Details about the SVG configuration json mentioned below can be found in the entry within the navlayers module documentation. .. code:: bash + C:\Users\attack>layerExporter_cli -h usage: layerExporter_cli [-h] -m {svg,excel} [-s {taxii,local,remote}] [--resource RESOURCE] -o OUTPUT [OUTPUT ...] @@ -602,6 +614,7 @@ layer files from either a specific group, software, or mitigation. Alternatively mapping to all associated groups, software, or mitigations across the techniques within ATT&CK. .. code:: bash + C:\Users\attack>layerGenerator_cli -h usage: layerGenerator_cli [-h] (--overview-type {group,software,mitigation,datasource} | --mapped-to MAPPED_TO | --batch-type {group,software,mitigation,datasource}) From c5124181726e6362b9173d83d1fbb84da35e5e78 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 14:12:01 -0400 Subject: [PATCH 120/159] fixing navlayer code syntax part 2 --- docs/navlayers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 5c9cd024..d4bd5d0c 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -139,7 +139,7 @@ Examples on how to create a layer programmatically, as opposed to loading it fro **Example Usage** .. code-block:: python - + example_layer3_dict = { "name": "example layer", "version": "3.0", @@ -277,7 +277,7 @@ to a local stix bundle, or if the source is set to `remote`, in which case it sh .. code-block:: python -x.to_xlsx(layerInit=layer, filepath="layer.xlsx") + x.to_xlsx(layerInit=layer, filepath="layer.xlsx") The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. From 6daf7aea157191c93ff8583239bedd91a316b2c6 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 14:15:07 -0400 Subject: [PATCH 121/159] added table of contents for stix primer --- docs/index.rst | 2 +- docs/stix_primer/access-attack.rst | 26 +----------------------- docs/stix_primer/overview.rst | 32 ++++++++++++++++++++++++++++++ 3 files changed, 34 insertions(+), 26 deletions(-) create mode 100644 docs/stix_primer/overview.rst diff --git a/docs/index.rst b/docs/index.rst index 5de682ba..c5e2a510 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -39,4 +39,4 @@ other modules in this library under "Additional Modules". :maxdepth: 1 :caption: STIX Primer - stix_primer/access-attack \ No newline at end of file + stix_primer/overview \ No newline at end of file diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index 6cc5c436..dc561e58 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -205,29 +205,5 @@ a single CompositeDataSource: You can then use this CompositeDataSource just as you would the DataSource for an individual domain. -.. toctree:: - :maxdepth: 1 - :caption: Table of Contents - - stix-recipes - attack_id - by_alias - by_name - deprecated_revoked - getting_related_objects - getting_software - multiple_objects - objects_by_content - objects_by_type - objects_since_date - stix_id - tactics_by_matrix - techniques_by_platform - techniques_by_platform - techniques_by_tactic - techniques_subtechniques - relationships_microlibrary - techniques_by_group_sw - remove_revoked_deprecated - getting_revoked_objects + diff --git a/docs/stix_primer/overview.rst b/docs/stix_primer/overview.rst new file mode 100644 index 00000000..a9424b17 --- /dev/null +++ b/docs/stix_primer/overview.rst @@ -0,0 +1,32 @@ +Overview +=============== + + +This is a short primer on how to manipulate STIX data from ATT&CK. While not comprehensive, it will provide you with a quick starting guide. + +.. toctree:: + :maxdepth: 1 + :caption: Table of Contents + + access-attack + stix-recipes + attack_id + by_alias + by_name + deprecated_revoked + getting_related_objects + getting_software + multiple_objects + objects_by_content + objects_by_type + objects_since_date + stix_id + tactics_by_matrix + techniques_by_platform + techniques_by_platform + techniques_by_tactic + techniques_subtechniques + relationships_microlibrary + techniques_by_group_sw + remove_revoked_deprecated + getting_revoked_objects \ No newline at end of file From 2e5a3039453928d54bfddd964c7fb465500ba57d Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 14:23:16 -0400 Subject: [PATCH 122/159] added nav layer core page --- docs/index.rst | 1 + docs/navlayercore.rst | 261 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+) create mode 100644 docs/navlayercore.rst diff --git a/docs/index.rst b/docs/index.rst index c5e2a510..6d7acb45 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -33,6 +33,7 @@ other modules in this library under "Additional Modules". diffinstix collections navlayers + navlayercore attacktoexcel .. toctree:: diff --git a/docs/navlayercore.rst b/docs/navlayercore.rst new file mode 100644 index 00000000..31ecb580 --- /dev/null +++ b/docs/navlayercore.rst @@ -0,0 +1,261 @@ +Layers Core +=============== + +This subcomponent, as part of the larger navlayers module, is responsible for Layer objects. Please note, this +documentation assumes familiarity with the [ATT&CK Navigator layer format](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md). +The main handle for this implementation is the Layer, which stores an individual instance of a LayerObj object, +which further references the various sub-objects that make up a complete Layer. A visual representation of this +object breakdown can be seen here (please note there are other fields, these are just the objects): + +.. code-block:: + + demo (Layer instance) <------------------------------------------------> The container for a layer object + |---> demo.layer (_LayerObj instance)--------------------------------> The raw layer object itself + |---> demo.layer.version (Versions instance)-----------------> A versions object + |---> demo.layer.filters (Filter instance)-------------------> A filter object + |---> demo.layer.layout (Layout instance)--------------------> A layout object + |---> demo.layer.techniques (List of Technique instances)----> A collection of technique objects + |---> demo.layer.gradient (Gradient instance)----------------> A gradient object + |---> demo.layer.legendItems (List of LegendItem instances)--> A collection of legend item objects + |---> demo.layer.metadata (List of Metadata instances)-------> A collection of metadata objects + + +**Creating Layers Programmatically** +With knowledge of the objects involved, as well as the additional fields (which have a 1:1 mapping with the +default ATT&CK Navigator spec), it is possible to programmatically generate a layer. Below is an example of +how this might be accomplished, piece by piece. + +.. code-block:: python + + import mitreattack.navlayers as navlayers + + layer_example = navlayers.Layer() + layer_example.from_dict(dict(name="example", domain="enterprise-attack")) # arguments required for every layer + + # configure the versions object + layer_example.layer.versions = dict(layer="4.2", attack="9.1", navigator="4.2") + + # set a description + layer_example.layer.description = "This is a demonstration of how to set up a layer piece by piece" + + # configure the "filters" object + layer_example.layer.filters = dict(platforms=['macOS']) # platforms can be provided during initialization + layer_example.layer.filters.platforms = ['Windows'] # or separately + + # configure the 'sorting' setting + layer_example.layer.sorting = 3 # 0: sort ascending alphabetically by technique name + # 1: sort descending alphabetically by technique name + # 2: sort ascending by technique score + # 3: sort descending by technique score + + # configure the layout object + layer_example.layer.layout = dict(layout="side", + showID=True, + showName=True, + showAggregateScores=True, + countUnscored=True, + aggregateFunction="sum") # average, sum, max, min + + # configure whether or not to hide disabled techniques + layer_example.layer.hideDisabled = True + # configure the gradient object + layer_example.layer.gradient = dict(minValue=-100, maxValue=100, + colors=["#DAF7A6", "#FFC300", "#FF5733", "#C70039", "#900C3F", "#581845"]) + # configure collection of legend items + layer_example.layer.legendItems = [dict(label='A', color='#DAF7A6'), dict(label='B', color='#581845')] + # configure collection of metatdata values + layer_example.layer.metadata = [dict(name='example metadata', value='This is an example')] + # create listing of techniques in this layer + layer_example.layer.techniques = [dict(techniqueID='T1000', tactic='privilege-escalation', score=15, color='#AABBCC'), + dict(techniqueID='T1000.1', tactic='privilege-escalation', score=1, comment='Demo')] + +This first example utilizes the native dictionary form for initializing the layer. This approach is similar to the +method used by the automated import process, but may not be the most intuitive for users. An alternative method, +displayed below, is to create and modify instances of the core objects in the library. Please note, these two examples +produce equivalent internal layers once completed. + +.. code-block:: python + import mitreattack.navlayers as navlayers + + layer_example = navlayers.Layer(name="example", domain="enterprise-attack") # arguments required for every layer + layer_build = layer_example.layer # short handle to make the rest of this example easier to read + + # configure the versions object + versions_obj = navlayers.Versions() + versions_obj.layer = "4.2" + versions_obj.attack = "9.1" + versions_obj.navigator = "4.2" + layer_build.versions = versions_obj + + # set a description + layer_build.description = "This is a demonstration of how to set up a layer piece by piece" + + # configure the "filters" object + filter_obj = navlayers.core.Filter(domain="enterprise-attack") + filter_obj.platforms = ['Windows'] + layer_build.filters = filter_obj + + # configure the 'sorting' setting + layer_build.sorting = 3 # 0: sort ascending alphabetically by technique name + # 1: sort descending alphabetically by technique name + # 2: sort ascending by technique score + # 3: sort descending by technique score + + # configure the layout object + layout_obj = navlayers.core.Layout() + layout_obj.layout = "side" + layout_obj.showID = True + layout_obj.showName = True + layout_obj.showAggregateScores = True + layout_obj.countUnscored = True + layout_obj.aggregateFunction = "sum" # average, sum, max, min + layer_build.layout = layout_obj + + # configure whether or not to hide disabled techniques + layer_build.hideDisabled = True + + # configure the gradient object + gradient_obj = navlayers.core.Gradient(colors=["#DAF7A6", "#FFC300", "#FF5733", "#C70039", "#900C3F", "#581845"], + minValue=-100, maxValue=100) + layer_build.gradient = gradient_obj + + # configure collection of legend items + legend_item_obj_a = navlayers.core.LegendItem(label='A', color='#DAF7A6') + legend_item_obj_b = navlayers.core.LegendItem(label='B', color='#581845') + list_of_legend_items = [legend_item_obj_a, legend_item_obj_b] + layer_build.legendItems = list_of_legend_items + + # configure collection of metatdata values + metadata_object = navlayers.core.Metadata(name='example metadata', value='This is an example') + layer_build.metadata = [metadata_object] + + # create listing of techniques in this layer + technique_obj_a = navlayers.core.Technique(tID='T1000') + technique_obj_a.tactic = 'privilege-escalation' + technique_obj_a.score = 15 + technique_obj_a.color = '#AABBCC' + technique_obj_b = navlayers.core.Technique(tID='T1000.1') + technique_obj_b.tactic = 'privilege-escalation' + technique_obj_b.score = 1 + technique_obj_b.comment = "Demo" + layer_build.techniques = [technique_obj_a, technique_obj_b] + + + +**Object Documentation** +Should it be helpful, the following section provides a breakdown of the available fields and methods for +each of the objects in the Core. This only includes 'public' methods and fields; there may be others used +for processing and other functionality that are not documented here, though documentation does exist for these +in the source code for them. + +**Layer Object** +.. code-block:: python + + Layer().layer # Stores the raw LayerObj file + Layer().strict # Determines whether or not to be strict about loading files + Layer().from_str() # Initializes data from a string + Layer().from_dict() # Initializes data from a dictionary + Layer().from_file() # Initializes data from a file + Layer().to_file() # Exports the layer data to a file + Layer().to_dict() # Exports the layer data to a dictionary + Layer().to_str() # Exports the layer data to a string + +**LayerObj Object** + +.. code-block:: python + + _LayerObj().versions # Link to a Versions object instance + _LayerObj().name # The Name for the Layer + _LayerObj().description # A description string for the Layer + _LayerObj().domain # The domain for the Layer + _LayerObj().filters # Link to a Filter object instance + _LayerObj().sorting # An integer denoting which sorting form to use + _LayerObj().layout # Link to a Layout object instance + _LayerObj().hideDisabled # Bool determining whether or not to show disabled techniques + _LayerObj().techniques # List of links to Technique objects + _LayerObj().gradient # Link to Gradient object + _LayerObj().legendItems # List of links to LegendItems objects + _LayerObj().showTacticRowBackground # Bool determining whether or not to show a background for tactics + _LayerObj().tacticRowBackground # Color code for tactic background + _LayerObj().selectTechniquesAcrossTactics # Bool determining whether or not to select cross-tactic + _LayerObj().selectSubtechniquesWithParent # Bool determining whether or not to select subtechniques + _LayerObj().metadata # List of links to Metadata items + _LayerObj().get_dict() # Export Layer as a dictionary object + +**Versions Object** +.. code-block:: python + + Versions().layer # String denoting Layer format version + Versions().__attack # String denoting ATT&CK version + Versions().navigator # String denoting Navigator version + Versions().get_dict() # Export Version data as a dictionary object + +**Filter Object** +.. code-block:: python + + Filter().domain # String denoting the domain for the Filter + Filter().platforms # String denoting platforms within this filter + Filter().get_dict() # Export Filter data as a dictionary object + +Please note that although not documented here, there is another Filter object variant, Filterv3, which exists +for backwards compatibility reasons. +**Layout Object** +.. code-block:: python + + Layout().layout # String denoting which layout form to use + Layout().showID # Bool denoting whether or not to show technique IDs + Layout().showName # Bool denoting whether or not to show technique names + Layout().showAggregateScores # Bool denoting whether or not to utilize Aggregate scores + Layout().countUnscored # Bool denoting whether ot not to count unscored techniques as 0s for Aggregates + Layout().aggregateFunction # A enum integer denoting which aggregate function to utilize + # 1 - Average, 2 - min, 3 - max, 4 - sum + Layout().get_dict() # Export Layout data as a dictionary object + Layout().compute_aggregate() # Compute the aggregate score for a technique and it's subtechniques + +**Technique Object** + +.. code-block:: python + + Technique().techniqueID # String denoting the technique's ID + Technique().tactic # String denoting the technique's tactic + Technique().comment # String denoting any comments + Technique().enabled # Bool denoting if the technique is enabled + Technique().score # Integer denoting technique score + Technique().aggregateScore # Integer denoting pre-configured aggregate score + Technique().color # String denoting manually configured color code + Technique().metadata # List of links to metadata objects + Technique().showSubtechniques # Bool denoting whether or not to show subtechniques + Technique().get_dict() # Export Technique data as a dictionary object + +**Gradient Object** + +.. code-block:: python + Gradient().colors # Array of colors (string codes) over which the gradient is to be calculated + Gradient().minValue # Integer denoting minimum viable value on the gradient + Gradient().maxValue # Integer denoting maximum viable value on the gradient + Gradient().compute_color() # Calculate the appropriate color for a given score on the gradient + Gradient().get_dict() # Export Gradient data as a dictionary object + +**LegendItem Object** + +.. code-block:: python + + LegendItem().label # String denoting the label for this Legend Item' item + LegendItem().color # String denoting the color code for the Legend Item + LegendItem().get_dict() # Export Legend Item data as a dictionary object + +**Metadata/Metadiv Object** + +.. code-block:: python + + Metadata().name # String denoting metadata keypair name + Metadata().value # String denoting metadata keypair value + Metadata().get_dict() # Export metadata data as a dictionary object + +.. code-block:: python + + Metadiv().name # Always set to "DIVIDER" + Metadiv().value # Bool denoting active or not + Metadiv().get_dict() # Export metadiv as a dictionary object + +A `Metadiv` object is simply a modified version of a `Metadata` object used as a visual divider. \ No newline at end of file From ba7746b096ce41572f259763c7f27f4db90a0765 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 16:23:24 -0400 Subject: [PATCH 123/159] adding in code for words --- docs/navlayercore.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayercore.rst b/docs/navlayercore.rst index 31ecb580..13cd8ddd 100644 --- a/docs/navlayercore.rst +++ b/docs/navlayercore.rst @@ -258,4 +258,4 @@ for backwards compatibility reasons. Metadiv().value # Bool denoting active or not Metadiv().get_dict() # Export metadiv as a dictionary object -A `Metadiv` object is simply a modified version of a `Metadata` object used as a visual divider. \ No newline at end of file +A ``Metadiv`` object is simply a modified version of a ``Metadata`` object used as a visual divider. \ No newline at end of file From e7592bba500406286bbad8eca926c6e5e9da1242 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 16:47:13 -0400 Subject: [PATCH 124/159] adding in code for words part 2 --- docs/navlayers.rst | 102 ++++++++++++++++++++++----------------------- 1 file changed, 51 insertions(+), 51 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index d4bd5d0c..8a3db488 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -109,7 +109,7 @@ and version 4.X layers, upgrading them to version 4.3. **Layer** -The `Layer` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. +The ``Layer`` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. It is the primary interface through which other Layer-related classes defined in the core module should be used. The Layer class API and a usage example are below. The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. @@ -120,17 +120,17 @@ The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will * - method [x = Layer()] - description - * - `x.from_str(_input_)` + * - ``x.from_str(_input_)`` - Loads an ATT&CK layer from a string representation of a json layer. - * - `x.from_dict(_input_)` + * - ``x.from_dict(_input_)`` - Loads an ATT&CK layer from a dictionary. - * - `x.from_file(_filepath_)` + * - ``x.from_file(_filepath_)`` - Loads an ATT&CK layer from a file location specified by the _filepath_. - * - `x.to_file(_filepath_)` + * - ``x.to_file(_filepath_)`` - Saves the current state of the loaded ATT&CK layer to a json file denoted by the _filepath_. - * - `x.to_dict()` + * - ``x.to_dict()`` - Returns a representation of the current ATT&CK layer object as a dictionary. - * - `x.to_str()` + * - ``x.to_str()`` - Returns a representation of the current ATT&CK layer object as a string representation of a dictionary. Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found @@ -174,9 +174,9 @@ Examples on how to create a layer programmatically, as opposed to loading it fro **layerops.py** -`Layerops.py` provides the `LayerOps` class, which is a way to combine layer files in an automated way, using user defined lambda functions. +``Layerops.py`` provides the ``LayerOps`` class, which is a way to combine layer files in an automated way, using user defined lambda functions. Each LayerOps instance, when created, ingests the provided lambda functions, and stores them for use. -An existing `LayerOps` class can be used to combine layer files according to the initialized lambda using the process method. +An existing ``LayerOps`` class can be used to combine layer files according to the initialized lambda using the process method. The breakdown of this two step process is documented in the table below, while examples of both the list and dictionary modes of operation can be found below. **LayerOps()** @@ -199,7 +199,7 @@ to provide the lambda functions if techniques of the combined layers are missing The process method applies the lambda functions stored during initialization to the layer objects in _data_. _data_ must be either a list or a dictionary of Layer objects, and is expected to match the format of the lambda equations provided during initialization. -`default_values` is an optional dictionary argument that overrides the currently stored default values with new ones for this specific processing operation. +``default_values`` is an optional dictionary argument that overrides the currently stored default values with new ones for this specific processing operation. **Example Usage** @@ -253,8 +253,8 @@ _data_ must be either a list or a dictionary of Layer objects, and is expected t **to_excel.py** -`to_excel.py` provides the `ToExcel` class, which is a way to export an existing layer file as an Excel spreadsheet. -The `ToExcel` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. +``to_excel.py`` provides the ``ToExcel`` class, which is a way to export an existing layer file as an Excel spreadsheet. +The ``ToExcel`` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or retrieving data from an ATT&CK Workbench instance. **ToExcel()** @@ -264,14 +264,14 @@ Valid options include using live data from cti-taxii.mitre.org, using a local ST x = ToExcel(domain='enterprise', source='taxii', resource=None) -The `ToExcel` constructor takes domain, server, and resource arguments during instantiation. -The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. +The ``ToExcel`` constructor takes domain, server, and resource arguments during instantiation. +The domain can be either ``enterprise`` or ``mobile``, and can be pulled directly from a layer file as ``layer.domain``. The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the official ATT&CK Taxii Server (`cti-taxii`) when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that +``taxii`` indicates that the tool should utilize the official ATT&CK Taxii Server (``cti-taxii``) when building the matrix, +while the ``local`` option indicates that it should use a local bundle, and the ``remote`` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path -to a local stix bundle, or if the source is set to `remote`, in which case it should be the url of a ATT&CK workbench instance. +The ``resource`` argument is only required if the source is set to ``local``, in which case it should be a path +to a local stix bundle, or if the source is set to ``remote``, in which case it should be the url of a ATT&CK workbench instance. **.to_xlsx() Method** @@ -280,7 +280,7 @@ to a local stix bundle, or if the source is set to `remote`, in which case it sh x.to_xlsx(layerInit=layer, filepath="layer.xlsx") -The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. +The ``to_xlsx`` method exports the layer file referenced as ``layer``, as an excel file to the ``filepath`` specified. **Example Usage** @@ -304,8 +304,8 @@ The `to_xlsx` method exports the layer file referenced as `layer`, as an excel f **to_svg.py** -`to_svg.py` provides the `ToSvg` class, which is a way to export an existing layer file as an SVG image file. -The `ToSvg` class, like the `ToExcel` class, has an optional parameter for the initialization function, +``to_svg.py`` provides the ``ToSvg`` class, which is a way to export an existing layer file as an SVG image file. +The ``ToSvg`` class, like the ``ToExcel`` class, has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or utilizing a remote ATT&CK Workbench instance. @@ -316,14 +316,14 @@ Valid options include using live data from cti-taxii.mitre.org, using a local ST x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) -The `ToSvg` constructor, just like the `ToExcel` constructor, takes domain, server, and resource arguments during instantiation. -The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. +The ``ToSvg`` constructor, just like the ``ToExcel`` constructor, takes domain, server, and resource arguments during instantiation. +The domain can be either ``enterprise`` or ``mobile``, and can be pulled directly from a layer file as ``layer.domain``. The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. -The `config` parameter is an optional `SVGConfig` object that can be used to configure the export as desired. +``taxii`` indicates that the tool should utilize the ``cti-taxii`` server when building the matrix, +while the ``local`` option indicates that it should use a local bundle, and the ``remote`` option indicates that it should utilize a remote ATT&CK Workbench instance. +The ``resource`` argument is only required if the source is set to ``local``, in which case it should be a path to a local stix bundle, +or if the source is set to ``remote``, in which case it should be the url of an ATT&CK Workbench instance. +The ``config`` parameter is an optional ``SVGConfig`` object that can be used to configure the export as desired. If not provided, the configuration for the export will be set to default values. **SVGConfig()** @@ -336,12 +336,12 @@ If not provided, the configuration for the export will be set to default values. showAbout=True, showDomain=True, border=0.104) -The `SVGConfig` object is used to configure how an SVG export behaves. +The ``SVGConfig`` object is used to configure how an SVG export behaves. The defaults for each of the available values can be found in the declaration above, and a brief explanation for each field is included in the table below. -The config object should be provided to the `ToSvg` object during instantiation, but if values need to be updated on the fly, -the currently loaded configuration can be interacted with at `ToSvg().config`. -The configuration can also be populated from a json file using the `.load_from_file(filename="path/to/file.json")` method, -or stored to one using the `.save_to_file(filename="path/to/file.json)` method. +The config object should be provided to the ``ToSvg`` object during instantiation, but if values need to be updated on the fly, +the currently loaded configuration can be interacted with at ``ToSvg().config``. +The configuration can also be populated from a json file using the ``.load_from_file(filename="path/to/file.json")`` method, +or stored to one using the ``.save_to_file(filename="path/to/file.json)`` method. .. list-table:: :widths: 25 25 25 25 @@ -430,7 +430,7 @@ or stored to one using the `.save_to_file(filename="path/to/file.json)` method. x.to_svg(layerInit=layer, filepath="layer.svg") -The `to_svg` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. +The ``to_svg`` method exports the layer file referenced as ``layer``, as an excel file to the ``filepath`` specified. **Example Usage** @@ -459,7 +459,7 @@ The `to_svg` method exports the layer file referenced as `layer`, as an excel fi **overview_generator.py** -`overview_generator.py` provides the `OverviewLayerGenerator` class, which is designed to allow users to +``overview_generator.py`` provides the ``OverviewLayerGenerator`` class, which is designed to allow users to generate an ATT&CK layer that, on a per technique basis, has a score that corresponds to all instances of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. @@ -504,12 +504,12 @@ and can be referenced by ID or by any alias when provided to the generator. The initialization function for `UsageLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where to retrieve data from (taxii server etc.). -The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. +The domain can be either ``enterprise``, ``mobile``, or ``ics``, and can be pulled directly from a layer file as ``layer.domain``. The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +``taxii`` indicates that the tool should utilize the ``cti-taxii`` server when building the matrix, +while the ``local`` option indicates that it should use a local bundle, and the ``remote`` option indicates that it should utilize a remote ATT&CK Workbench instance. +The ``resource`` argument is only required if the source is set to ``local``, in which case it should be a path to a local stix bundle, +or if the source is set to ``remote``, in which case it should be the url of an ATT&CK Workbench instance. If not provided, the configuration for the generator will be set to default values. **.generate_layer()** @@ -519,8 +519,8 @@ If not provided, the configuration for the generator will be set to default valu x.generate_layer(match=object_identifier) -The `generate_layer` function generates a layer, customized to the input `object_identifier`. -Valid values include `ATT&CK ID`, `name`, or any known `alias` for `group`, `mitigation`, `software`, and `data component` objects within the selected ATT&CK data. +The ``generate_layer`` function generates a layer, customized to the input ``object_identifier``. +Valid values include ``ATT&CK ID``, ``name``, or any known ``alias`` for ``group``, ``mitigation``, ``software``, and ``data component`` objects within the selected ATT&CK data. .. code-block:: python @@ -534,7 +534,7 @@ Valid values include `ATT&CK ID`, `name`, or any known `alias` for `group`, `mit **sum_generator.py** -`sum_generator.py` provides the `SumLayerGenerator` class, which is designed to allow users to +``sum_generator.py`` provides the ``SumLayerGenerator`` class, which is designed to allow users to generate a collection of ATT&CK layers that, on a per technique basis, have a score that corresponds to all instances of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. Each one of the generated layers will correspond to a single instance of the specified ATT&CK object type. @@ -546,14 +546,14 @@ Each one of the generated layers will correspond to a single instance of the spe x = SumLayerGenerator(source='taxii', domain='enterprise', resource=None) -The initialization function for `SumGeneratorLayer`, like `ToSVG` and `ToExcel`, requires the specification of where +The initialization function for ``SumGeneratorLayer``, like ``ToSVG`` and ``ToExcel``, requires the specification of where to retrieve data from (taxii server etc.). -The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. +The domain can be either ``enterprise``, ``mobile``, or ``ics``, and can be pulled directly from a layer file as ``layer.domain``. The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +``taxii`` indicates that the tool should utilize the ``cti-taxii`` server when building the matrix, +while the ``local`` option indicates that it should use a local bundle, and the ``remote`` option indicates that it should utilize a remote ATT&CK Workbench instance. +The ``resource`` argument is only required if the source is set to ``local``, in which case it should be a path to a local stix bundle, +or if the source is set to ``remote``, in which case it should be the url of an ATT&CK Workbench instance. If not provided, the configuration for the generator will be set to default values. **.generate_layer()** @@ -563,8 +563,8 @@ If not provided, the configuration for the generator will be set to default valu x.generate_layer(layers_type=object_type_name) -The `generate_layer` function generates a collection of layers, each customized to one instance of the input `object_type_name`. -Valid types include `group`, `mitigation`, `software`, and `datasource`. +The ``generate_layer`` function generates a collection of layers, each customized to one instance of the input ``object_type_name``. +Valid types include ``group``, ``mitigation``, ``software``, and ``datasource``. **layerExporter_cli.py** From 4ea4b890fe06fd56a34c91f71d49d7ff7f26f303 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 18 Jul 2023 17:23:56 -0400 Subject: [PATCH 125/159] finished word formatting for navlayers --- docs/navlayers.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 8a3db488..1bbd27a6 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -470,14 +470,14 @@ of the specified ATT&CK object type (group, mitigation, etc.), and a comment tha x = OverviewLayerGenerator(source='taxii', domain='enterprise', resource=None) -The initialization function for `OverviewLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where +The initialization function for ``OverviewLayerGenerator``, like ``ToSVG`` and ``ToExcel``, requires the specification of where to retrieve data from (taxii server etc.). -The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. +The domain can be either ``enterprise``, ``mobile``, or ``ics``, and can be pulled directly from a layer file as ``layer.domain``. The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. +``taxii`` indicates that the tool should utilize the ``cti-taxii`` server when building the matrix, +while the ``local`` option indicates that it should use a local bundle, and the ``remote`` option indicates that it should utilize a remote ATT&CK Workbench instance. +The ``resource`` argument is only required if the source is set to ``local``, in which case it should be a path to a local stix bundle, +or if the source is set to ``remote``, in which case it should be the url of an ATT&CK Workbench instance. If not provided, the configuration for the generator will be set to default values. **.generate_layer()** @@ -486,14 +486,14 @@ If not provided, the configuration for the generator will be set to default valu x.generate_layer(obj_type=object_type_name) -The `generate_layer` function generates a layer, customized to the input `object_type_name`. -Valid values include `group`, `mitigation`, `software`, and `datasource`. +The ``generate_layer`` function generates a layer, customized to the input ``object_type_name``. +Valid values include ``group``, ``mitigation``, ``software``, and ``datasource``. **usage_generator.py** -`usage_ generator.py` provides the `UsageLayerGenerator` class, which is designed to allow users to +``usage_ generator.py`` provides the ``UsageLayerGenerator`` class, which is designed to allow users to generate an ATT&CK layer that scores any relevant techniques that a given input ATT&CK object has. -These objects can be any `group`, `software`, `mitigation`, or `data component`, +These objects can be any ``group``, ``software``, ``mitigation``, or ``data component``, and can be referenced by ID or by any alias when provided to the generator. **UsageLayerGenerator()** @@ -502,7 +502,7 @@ and can be referenced by ID or by any alias when provided to the generator. x = UsageLayerGenerator(source='taxii', domain='enterprise', resource=None) -The initialization function for `UsageLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where +The initialization function for ``UsageLayerGenerator``, like ``ToSVG`` and ``ToExcel``, requires the specification of where to retrieve data from (taxii server etc.). The domain can be either ``enterprise``, ``mobile``, or ``ics``, and can be pulled directly from a layer file as ``layer.domain``. The source argument tells the matrix generation tool which data source to use when building the matrix. From 7ab40f4288ee02d9846b992aa9fbb45be39cf1c6 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 13:36:58 -0400 Subject: [PATCH 126/159] testing out subheaders --- docs/attacktoexcel.rst | 6 ++++-- docs/diffinstix.rst | 3 ++- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index cfa6d16e..88ff4c8b 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -4,9 +4,11 @@ ATT&CK to Excel This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. -**Usage** +Usage +----- -**Command Line** +Command Line +----- Print full usage instructions: diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 7e4acc00..7e98c6ef 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -5,7 +5,8 @@ This folder contains a module for creating markdown, HTML, JSON and/or ATT&CK Na reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. -**Usage** +Usage +----- **Command Line** From 7a8aa265df10297141258e2a5c997d02ce50ba31 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 13:46:19 -0400 Subject: [PATCH 127/159] cleaning up attacktoexcel --- docs/attacktoexcel.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 88ff4c8b..08b1b68a 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -7,24 +7,26 @@ It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata Usage ----- -Command Line ------ +**Command Line** Print full usage instructions: .. code:: bash + python3 attackToExcel.py -h Example execution: .. code:: bash + python3 attackToExcel.py Build a excel files corresponding to a specific domain and version of ATT&CK: .. code:: bash + python3 attackToExcel -domain mobile-attack -version v5.0 @@ -33,6 +35,7 @@ Build a excel files corresponding to a specific domain and version of ATT&CK: Example execution targeting a specific domain and version: .. code-block:: python + import mitreattack.attackToExcel.attackToExcel as attackToExcel attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") @@ -121,6 +124,7 @@ These can be retrieved for use in data analysis. Example of accessing [Pandas](https://pandas.pydata.org/) DataFrames: .. code-block:: python + import mitreattack.attackToExcel.attackToExcel as attackToExcel import mitreattack.attackToExcel.stixToDf as stixToDf From 47c5aeb9ca6b831662114619494366ccfec410ea Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 13:53:05 -0400 Subject: [PATCH 128/159] cleaning up getting_started page --- docs/getting_started.rst | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 7ca57005..90a7e79d 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -25,9 +25,26 @@ you can read more about other modules in this library under "Additional Modules" More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. -`navlayers `_ A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. | Further documentation can be found `here `_ . `attackToExcel `_ | A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to `Pandas `_ DataFrames representing the dataset for use in data analysis. | Further documentation can be found `here `_. -`collections `_. A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. | Further documentation can be found `here `_. -| `diffStix `_ | Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. | Further documentation can be found `here `_.| + +.. list-table:: + :widths: 33 33 34 + :header-rows: 1 + + * - module + - description + - documentation + * - `navlayers `_ + - A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. + - Further documentation can be found `here `_ + * - `attackToExcel `_ + - A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to `Pandas `_ DataFrames representing the dataset for use in data analysis. + - Further documentation can be found `here `_ + * - `collections `_ + - A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. + - Further documentation can be found `here `_. + * - `diffStix `_ + - Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. + - Further documentation can be found `here `_ **Related MITRE Work** From 6024474417ca7372c79de5cd377d5435e51ba0ed Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 17:37:07 -0400 Subject: [PATCH 129/159] replaced bolded text with section subheaders --- docs/attacktoexcel.rst | 23 +++++++++++++++-------- docs/collections.rst | 9 ++++++--- docs/getting_started.rst | 22 ++++++++++++++-------- docs/navlayercore.rst | 8 ++++++-- docs/navlayers.rst | 30 ++++++++++++++++++++---------- 5 files changed, 61 insertions(+), 31 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 08b1b68a..1f77a72e 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -4,10 +4,11 @@ ATT&CK to Excel This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. -Usage +Usage: ----- -**Command Line** +Command Line +----- Print full usage instructions: @@ -30,7 +31,8 @@ Build a excel files corresponding to a specific domain and version of ATT&CK: python3 attackToExcel -domain mobile-attack -version v5.0 -**Module** +Module +----- Example execution targeting a specific domain and version: @@ -41,9 +43,11 @@ Example execution targeting a specific domain and version: attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") -**Interfaces** +Interfaces: +----- -**attackToExcel** +attackToExcel +----- attackToExcel provides the means by which to convert/extract the ATT&CK STIX data to Excel spreadsheets. A brief overview of the available methods follows. @@ -69,7 +73,8 @@ overview of the available methods follows. - `domain`: the domain of ATT&CK to download
`version`: optional parameter specifying which version of ATT&CK to download
`output_dir`: optional parameter specifying output directory - Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets -**stixToDf** +stixToDf +----- stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for processing. A brief overview of these methods follows. @@ -104,7 +109,8 @@ processing. A brief overview of these methods follows. - Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]` -**Spreadsheet format** +Spreadsheet format +----- The Excel representation of the ATT&CK dataset includes both master spreadsheets, containing all object types, and individual spreadsheets for each object type. @@ -116,7 +122,8 @@ A citations sheet can be used to look up the in-text citations which appear in s For domains that include multiple matrices, such as Mobile ATT&CK, each matrix gets its own named sheet. Unlike the STIX dataset, objects that have been revoked or deprecated are not included in the spreadsheets. -**Accessing the Pandas DataFrames** +Accessing the Pandas DataFrames +----- Internally, attackToExcel stores the parsed STIX data as [Pandas](https://pandas.pydata.org/) DataFrames. These can be retrieved for use in data analysis. diff --git a/docs/collections.rst b/docs/collections.rst index 04bc730a..78bf5830 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -7,7 +7,8 @@ Collections are sets of ATT&CK STIX objects, grouped for user convienence. For more information about ATT&CK collections, see the corresponding `ATT&CK documentation `_. -**Collections Scripts** +Collections Scripts +------------------------ .. list-table:: @@ -55,7 +56,8 @@ into a single `collection index `_ on ReadTheDocs. -**Install** +Install +----- To use this package, install the mitreattack-python library with [pip](https://pip.pypa.io/en/stable/): @@ -15,13 +16,15 @@ To use this package, install the mitreattack-python library with [pip](https://p Note: the library requires [python3](https://www.python.org/). -**MitreAttackData Library** +MitreAttackData Library +------------------------ The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides the ability to query the dataset for objects and their related objects. This is the main content of mitreattack-python; you can read more about other modules in this library under "Additional Modules". -**Additional Modules** +Additional Modules +------------------------ More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. @@ -47,17 +50,20 @@ More detailed information and examples about the specific usage of the additiona - Further documentation can be found `here `_ -**Related MITRE Work** +Related MITRE Work +------------------------ Go to `this link `_ for related MITRE work. -**Contributing** +Contributing +------------------------ To contribute to this project, either through a bug report, feature request, or merge request, please see the `Contributors Guide `_. -**Notice** +Notice +------------------------ Copyright 2023 The MITRE Corporation @@ -77,4 +83,4 @@ limitations under the License. This project makes use of ATT&CK® -[ATT&CK Terms of Use](https://attack.mitre.org/resources/terms-of-use/) +`ATT&CK Terms of Use `_ diff --git a/docs/navlayercore.rst b/docs/navlayercore.rst index 13cd8ddd..8bf4792b 100644 --- a/docs/navlayercore.rst +++ b/docs/navlayercore.rst @@ -20,7 +20,9 @@ object breakdown can be seen here (please note there are other fields, these are |---> demo.layer.metadata (List of Metadata instances)-------> A collection of metadata objects -**Creating Layers Programmatically** +Creating Layers Programmatically +------------------------ + With knowledge of the objects involved, as well as the additional fields (which have a 1:1 mapping with the default ATT&CK Navigator spec), it is possible to programmatically generate a layer. Below is an example of how this might be accomplished, piece by piece. @@ -142,7 +144,9 @@ produce equivalent internal layers once completed. -**Object Documentation** +Object Documentation +------------------------ + Should it be helpful, the following section provides a breakdown of the available fields and methods for each of the objects in the Core. This only includes 'public' methods and fields; there may be others used for processing and other functionality that are not documented here, though documentation does exist for these diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 1bbd27a6..385c542b 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -11,7 +11,8 @@ All scripts adhere to the MITRE ATT&CK Navigator Layer file format, but will accept legacy `version 3.0 `_ and version 4.X layers, upgrading them to version 4.3. -**Core Modules** +Core Modules +------------------------ .. list-table:: :widths: 50 50 @@ -36,7 +37,8 @@ and version 4.X layers, upgrading them to version 4.3. * - `versions `_ - Implements a basic `versions object `_ -**Manipulator Scripts** +Manipulator Scripts +------------------------ .. list-table:: :widths: 50 50 @@ -47,7 +49,8 @@ and version 4.X layers, upgrading them to version 4.3. * - `layerops `_ - Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. -**Exporter Scripts** +Exporter Scripts +------------------------ .. list-table:: :widths: 50 50 @@ -62,7 +65,8 @@ and version 4.X layers, upgrading them to version 4.3. -**Generator Scripts** +Generator Scripts +------------------------ .. list-table:: :widths: 50 50 @@ -77,7 +81,8 @@ and version 4.X layers, upgrading them to version 4.3. * - `sum_generator `_ - Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. -**Utility Modules** +Utility Modules +------------------------ .. list-table:: :widths: 50 50 @@ -94,7 +99,8 @@ and version 4.X layers, upgrading them to version 4.3. * - `svg_objects `_ - Provides raw templates and supporting functionality for generating svg objects. -**Command Line Tools** +Command Line Tools +------------------------ .. list-table:: :widths: 50 50 @@ -107,7 +113,8 @@ and version 4.X layers, upgrading them to version 4.3. * - `layerGenerator_cli.py `_ - A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. -**Layer** +Layer +------------------------ The ``Layer`` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. It is the primary interface through which other Layer-related classes defined in the core module should be used. @@ -136,7 +143,8 @@ The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found `here `_. -**Example Usage** +Example Usage +------------------------ .. code-block:: python @@ -282,7 +290,8 @@ to a local stix bundle, or if the source is set to ``remote``, in which case it The ``to_xlsx`` method exports the layer file referenced as ``layer``, as an excel file to the ``filepath`` specified. -**Example Usage** +Example Usage +------------------------ .. code-block:: python @@ -432,7 +441,8 @@ or stored to one using the ``.save_to_file(filename="path/to/file.json)`` method The ``to_svg`` method exports the layer file referenced as ``layer``, as an excel file to the ``filepath`` specified. -**Example Usage** +Example Usage +-------------- .. code-block:: python From eba7bedbc46889bb74784249b46cbce3e3f9d7f7 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 17:40:50 -0400 Subject: [PATCH 130/159] added codeblock formatting to diffstix --- docs/getting_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 97aece1c..7368d5b9 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -46,7 +46,7 @@ More detailed information and examples about the specific usage of the additiona - A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. - Further documentation can be found `here `_. * - `diffStix `_ - - Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. + - Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run ``diff_stix -h`` for full usage instructions. - Further documentation can be found `here `_ From 8c59c2526b7f0e6d4ec7537250f90fd9811bf400 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 17:45:43 -0400 Subject: [PATCH 131/159] additional improvements --- docs/getting_started.rst | 6 +++--- docs/related_work.rst | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 7368d5b9..aae213b4 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -2,8 +2,8 @@ Getting started ============================================== -This repository contains a library of Python tools and utilities for working with ATT&CK data. For more information, -see the `full documentation `_ on ReadTheDocs. +This repository houses the ATT&CK catalog's Cyber Threat Intelligence, represented in the STIX 2.0 JSON format. +Additionally, it includes a USAGE document that provides further examples for accessing and parsing our dataset using Python. Install ----- @@ -21,7 +21,7 @@ MitreAttackData Library The ``MitreAttackData`` library is used to read in and work with MITRE ATT&CK STIX 2.0 content. This library provides the ability to query the dataset for objects and their related objects. This is the main content of mitreattack-python; -you can read more about other modules in this library under "Additional Modules". +you can read more about other modules in this library under the "Additional Modules" header in the side pane. Additional Modules ------------------------ diff --git a/docs/related_work.rst b/docs/related_work.rst index 9050f901..c536fabc 100644 --- a/docs/related_work.rst +++ b/docs/related_work.rst @@ -16,7 +16,7 @@ phases of an adversary's lifecycle, and the platforms they are known to target. for understanding security risk against known adversary behavior, for planning security improvements, and verifying defenses work as expected. -https://attack.mitre.org +Visit `the ATT&CK website `_ to browse our curated knowledge base. STIX ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 15b452d9660cac18d805ceee2d86b337cc540c4a Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 17:46:37 -0400 Subject: [PATCH 132/159] additional improvements to getting started page --- docs/getting_started.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index aae213b4..fca6ff75 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -8,13 +8,14 @@ Additionally, it includes a USAGE document that provides further examples for ac Install ----- -To use this package, install the mitreattack-python library with [pip](https://pip.pypa.io/en/stable/): +To use this package, install the mitreattack-python library with `pip `_. MitreAttackData Library ------------------------ From 9500fbe302d756827c5cdcd6181ec02e80fd9df9 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 19 Jul 2023 18:45:22 -0400 Subject: [PATCH 133/159] additional improvements to getting started page part 2 --- docs/getting_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index fca6ff75..271b7ca7 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -8,7 +8,7 @@ Additionally, it includes a USAGE document that provides further examples for ac Install ----- -To use this package, install the mitreattack-python library with `pip Date: Thu, 20 Jul 2023 10:42:35 -0400 Subject: [PATCH 134/159] fixing descriptions for additional modules --- docs/attacktoexcel.rst | 2 +- docs/collections.rst | 2 +- docs/diffinstix.rst | 2 +- docs/navlayers.rst | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index 1f77a72e..c11fc9ec 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -1,7 +1,7 @@ ATT&CK to Excel ============================================== -This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. +ATT&CK to Excel contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. Usage: diff --git a/docs/collections.rst b/docs/collections.rst index 78bf5830..f6359e2a 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -2,7 +2,7 @@ Collections ============================================== -This folder contains modules and scripts for working with ATT&CK collections. +Collections contains modules and scripts for working with ATT&CK collections. Collections are sets of ATT&CK STIX objects, grouped for user convienence. For more information about ATT&CK collections, see the corresponding `ATT&CK documentation `_. diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 7e98c6ef..88c72353 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -1,7 +1,7 @@ Diff Stix ============================================== -This folder contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers +Diff Stix contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run `diff_stix -h` for full usage instructions. diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 385c542b..2343ba61 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -1,7 +1,7 @@ NavLayers ============================================== -This folder contains modules and scripts for working with ATT&CK Navigator layers. +NavLayers contains modules and scripts for working with ATT&CK Navigator layers. ATT&CK Navigator Layers are a set of annotations overlaid on top of the ATT&CK Matrix. For more about ATT&CK Navigator layers, visit the ATT&CK Navigator repository. The core module allows users to load, validate, manipulate, and save ATT&CK layers. From 12a1e063537b10423239b09219a26c711320c2fd Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 14:09:26 -0400 Subject: [PATCH 135/159] fixing syntax part 2 --- docs/stix_primer/access-attack.rst | 28 ++++++++++----------- docs/stix_primer/attack_id.rst | 2 +- docs/stix_primer/getting_revoked_object.rst | 2 +- docs/stix_primer/stix-recipes.rst | 2 +- docs/stix_primer/tactics_by_matrix.rst | 4 +-- docs/stix_primer/techniques_by_tactic.rst | 2 +- 6 files changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index dc561e58..e7c35de0 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -6,22 +6,22 @@ Accessing ATT&CK data in python There are several ways to acquire the ATT&CK data in Python. All of them will provide an object implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. -This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. +This section utilizes the `stix2 python library `_ . Please refer to the `STIX2 Python API Documentation `_ for more information on how to work with STIX programmatically. **Requirements and imports** Before installing requirements, we recommend setting up a virtual environment: 1. Create virtual environment: - - macOS and Linux: `python3 -m venv env` - - Windows: `py -m venv env` + - macOS and Linux: ``python3 -m venv env`` + - Windows: ``py -m venv env`` 2. Activate the virtual environment: - - macOS and Linux: `source env/bin/activate` - - Windows: `env/Scripts/activate.bat` + - macOS and Linux: ``source env/bin/activate`` + - Windows: ``env/Scripts/activate.bat`` **stix2** -[stix2 can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-python-stix2#installation). Imports for the recipes in this repository can be done from the base package, for example: +stix2 can be installed by following the instructions on their `repository `_ . Imports for the recipes in this repository can be done from the base package, for example: .. code-block:: python @@ -35,13 +35,13 @@ However, if you are aiming to extend the ATT&CK dataset with new objects or impl from stix2.v20 import AttackPattern -You can see a full list of the classes which have versioned imports [here](https://stix2.readthedocs.io/en/latest/api/stix2.v20.html). +You can see a full list of the classes which have versioned imports `here `_ . **taxii2client** -[taxii2-client can be installed by following the instructions on their repository](https://github.com/oasis-open/cti-taxii-client#installation). The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of `taxii2client` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. +taxii2-client can be installed by following the instructions on their `repository `_ . The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of ``taxii2client`` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. -If the TAXII Client is getting a 406 Response, make sure you are running the latest version (`pip install --upgrade stix2` or `pip install --upgrade taxii2-client`). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. +If the TAXII Client is getting a 406 Response, make sure you are running the latest version (``pip install --upgrade stix2` or `pip install --upgrade taxii2-client``). In addition, make sure you are running the 2.0 version of the client (using the `v20` import) as shown below in order to communicate with the ATT&CK TAXII 2.0 Server. .. code-block:: python @@ -58,8 +58,8 @@ Many users may opt to access the ATT&CK content via a local copy of the STIX dat **Access via FileSystemSource** -Each domain in this repo is formatted according to the [STIX2 FileSystem spec](https://stix2.readthedocs.io/en/latest/guide/filesystem.html). -Therefore you can use a `FileSystemSource` to load a domain, for example to load the enterprise-attack domain: +Each domain in this repo is formatted according to the `STIX2 FileSystem spec `_ . +Therefore you can use a ``FileSystemSource`` to load a domain, for example to load the enterprise-attack domain: .. code-block:: python @@ -70,7 +70,7 @@ src = FileSystemSource('./cti/enterprise-attack') **Access via bundle** -If you instead prefer to download just the domain bundle, e.g [enterprise-attack.json](/enterprise-attack/enterprise-attack.json), you can still load this using a MemoryStore: +If you instead prefer to download just the domain bundle, e.g `enterprise-attack.json `_ , you can still load this using a MemoryStore: .. code-block:: python @@ -134,7 +134,7 @@ The following recipe demonstrates how to access the enterprise-attack data from src = TAXIICollectionSource(collection) -For more about TAXII, please see oasis-open's [Introduction to TAXII](https://oasis-open.github.io/cti-documentation/taxii/intro). +For more about TAXII, please see oasis-open's `Introduction to TAXII `_ . **Access from Github via requests** @@ -157,7 +157,7 @@ access data on a branch of the MITRE/CTI repo (the TAXII server only holds the m **Access a specific version of ATT&CK** -ATT&CK versions are tracked on the MITRE/CTI repo using [tags](https://github.com/mitre/cti/tags). Tags prefixed with `ATT&CK-v` correspond to ATT&CK versions and tags prefixed with `CAPEC-v` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. +ATT&CK versions are tracked on the MITRE/CTI repo using `tags `_ . Tags prefixed with ``ATT&CK-v`` correspond to ATT&CK versions and tags prefixed with ``CAPEC-v`` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: diff --git a/docs/stix_primer/attack_id.rst b/docs/stix_primer/attack_id.rst index 4a449c7c..cd70cc5c 100644 --- a/docs/stix_primer/attack_id.rst +++ b/docs/stix_primer/attack_id.rst @@ -10,7 +10,7 @@ The following recipe can be used to retrieve an object according to its ATT&CK I g0075 = src.query([ Filter("external_references.external_id", "=", "G0075") ])[0] -Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as `attack-pattern` you can avoid this issue. +Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techniques and shared their technique's ID. Therefore the above method does not work properly for techniques because technique ATTT&CK IDs are not truly unique. By specifying the STIX type you're looking for as ``attack-pattern`` you can avoid this issue. .. code-block:: python diff --git a/docs/stix_primer/getting_revoked_object.rst b/docs/stix_primer/getting_revoked_object.rst index 62e92873..333cad8d 100644 --- a/docs/stix_primer/getting_revoked_object.rst +++ b/docs/stix_primer/getting_revoked_object.rst @@ -1,7 +1,7 @@ Getting a revoking object =============== -When an object is replaced by another object, it is marked with the field `revoked` and a relationship of type `revoked-by` is created where the `source_ref` is the revoked object and the `target_ref` is the revoking object. This relationship can be followed to find the replacing object: +When an object is replaced by another object, it is marked with the field ``revoked`` and a relationship of type ``revoked-by`` is created where the ``source_ref`` is the revoked object and the ``target_ref`` is the revoking object. This relationship can be followed to find the replacing object: .. code-block:: python diff --git a/docs/stix_primer/stix-recipes.rst b/docs/stix_primer/stix-recipes.rst index ef6a4cb2..aae9db25 100644 --- a/docs/stix_primer/stix-recipes.rst +++ b/docs/stix_primer/stix-recipes.rst @@ -5,7 +5,7 @@ Stix Recipes Below are example python recipes which can be used to work with ATT&CK data. They assume the existence of an object implementing the DataStore API. Any of the methods outlined in the [Accessing ATT&CK data in python](#accessing-ATTCK-Data-in-Python) section should provide an object implementing this API. -This section utilizes the [stix2 python library](https://github.com/oasis-open/cti-python-stix2). Please refer to the [STIX2 Python API Documentation](https://stix2.readthedocs.io/en/latest/) for more information on how to work with STIX programmatically. See also the section on [Requirements and imports](#requirements-and-imports). +This section utilizes the `stix2 python library `_ . Please refer to the `STIX2 Python API Documentation `_ for more information on how to work with STIX programmatically. See also the section on [Requirements and imports](#requirements-and-imports). **Getting an object** diff --git a/docs/stix_primer/tactics_by_matrix.rst b/docs/stix_primer/tactics_by_matrix.rst index d30bb285..c556e180 100644 --- a/docs/stix_primer/tactics_by_matrix.rst +++ b/docs/stix_primer/tactics_by_matrix.rst @@ -1,8 +1,8 @@ Tactics by matrix =============== -The tactics are individual objects (`x-mitre-tactic`), and their order in a matrix (`x-mitre-matrix`) is -found within the `tactic_refs` property in a matrix. The order of the tactics in that list matches +The tactics are individual objects (``x-mitre-tactic``), and their order in a matrix a (``x-mitre-matrix``) is +found within the ``tactic_refs`` property in a matrix. The order of the tactics in that list matches the ordering of the tactics in that matrix. The following recipe returns a structured list of tactics within each matrix of the input DataStore. .. code-block:: python diff --git a/docs/stix_primer/techniques_by_tactic.rst b/docs/stix_primer/techniques_by_tactic.rst index a09761ac..ee0d0bed 100644 --- a/docs/stix_primer/techniques_by_tactic.rst +++ b/docs/stix_primer/techniques_by_tactic.rst @@ -2,7 +2,7 @@ Techniques by tactic =============== Techniques are related to tactics by their kill_chain_phases property. -The `phase_name` of each kill chain phase corresponds to the `x_mitre_shortname` of a tactic. +The ``phase_name`` of each kill chain phase corresponds to the ``x_mitre_shortname`` of a tactic. .. code-block:: python From ae7eb0e531280a5a4702351400c38d243ad1dae1 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 14:41:41 -0400 Subject: [PATCH 136/159] fixing link in access attack --- docs/stix_primer/access-attack.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index e7c35de0..a55e2b7f 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -4,7 +4,7 @@ Accessing ATT&CK data in python **Accessing ATT&CK data in python** There are several ways to acquire the ATT&CK data in Python. All of them will provide an object -implementing the DataStore API and can be used interchangeably with the recipes provided in the [Python recipes](#Python-Recipes) section. +implementing the DataStore API and can be used interchangeably with the recipes provided in the `STIX recipes `_ section. This section utilizes the `stix2 python library `_ . Please refer to the `STIX2 Python API Documentation `_ for more information on how to work with STIX programmatically. From 6eb45f1c7b32377249979fbca6f21a308c1eb011 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 14:43:36 -0400 Subject: [PATCH 137/159] changing bolded to section header --- docs/navlayers.rst | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 2343ba61..f137ebc9 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -259,7 +259,8 @@ _data_ must be either a list or a dictionary of Layer objects, and is expected t out_layer6.to_file("C:\demo_layer6.json") # Save combined comment layer to file -**to_excel.py** +to_excel.py +------------------------ ``to_excel.py`` provides the ``ToExcel`` class, which is a way to export an existing layer file as an Excel spreadsheet. The ``ToExcel`` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. @@ -311,7 +312,8 @@ Example Usage t3 = ToExcel(domain='ics', source='remote', resource=workbench_url) -**to_svg.py** +to_svg.py +------------------------ ``to_svg.py`` provides the ``ToSvg`` class, which is a way to export an existing layer file as an SVG image file. The ``ToSvg`` class, like the ``ToExcel`` class, has an optional parameter for the initialization function, @@ -467,7 +469,8 @@ Example Usage t3.to_svg(layerInit=lay, filepath="demo3.svg") -**overview_generator.py** +overview_generator.py +------------------------ ``overview_generator.py`` provides the ``OverviewLayerGenerator`` class, which is designed to allow users to generate an ATT&CK layer that, on a per technique basis, has a score that corresponds to all instances @@ -499,7 +502,8 @@ If not provided, the configuration for the generator will be set to default valu The ``generate_layer`` function generates a layer, customized to the input ``object_type_name``. Valid values include ``group``, ``mitigation``, ``software``, and ``datasource``. -**usage_generator.py** +usage_generator.py +------------------------ ``usage_ generator.py`` provides the ``UsageLayerGenerator`` class, which is designed to allow users to generate an ATT&CK layer that scores any relevant techniques that a given input ATT&CK object has. @@ -542,7 +546,8 @@ Valid values include ``ATT&CK ID``, ``name``, or any known ``alias`` for ``group layer2 = handle.generate_layer(match='Adups') -**sum_generator.py** +sum_generator.py +------------------------ ``sum_generator.py`` provides the ``SumLayerGenerator`` class, which is designed to allow users to generate a collection of ATT&CK layers that, on a per technique basis, have a score that corresponds to all instances @@ -576,7 +581,8 @@ If not provided, the configuration for the generator will be set to default valu The ``generate_layer`` function generates a collection of layers, each customized to one instance of the input ``object_type_name``. Valid types include ``group``, ``mitigation``, ``software``, and ``datasource``. -**layerExporter_cli.py** +layerExporter_cli.py +------------------------ This command line tool allows users to convert a `navigator `_ layer file to either an svg image or excel file using the functionality provided by the navlayers module. @@ -617,7 +623,8 @@ entry within the navlayers module documentation. C:\Users\attack>layerExporter_cli -m svg -s taxii -l settings/config.json -o output/svg1.json output/svg2.json files/layer1.json files/layer2.json -**layerGenerator_cli.py** +layerGenerator_cli.py +------------------------ This command line tool allows users to generate `ATT&CK Navigator `_ layer files from either a specific group, software, or mitigation. Alternatively, users can generate a layer file with a From 82bf26a73812f13f0e13d1b78112bb2a9cc1d4c6 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 14:51:30 -0400 Subject: [PATCH 138/159] adding regs --- docs/navlayers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index f137ebc9..29a34448 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -25,7 +25,7 @@ Core Modules * - `gradient `_ - Implements a basic `gradient object `_ * - `layer `_ - - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. + - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section] :ref:`Layer` below. * - `layout `_ - Implements a basic `layout object `_ * - `legenditem `_ From e3f2466bef2144c137eed6b1457483caa67a4558 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 15:03:43 -0400 Subject: [PATCH 139/159] adding refs --- docs/navlayers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 29a34448..550260fa 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -25,7 +25,7 @@ Core Modules * - `gradient `_ - Implements a basic `gradient object `_ * - `layer `_ - - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section] :ref:`Layer` below. + - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding :ref:`Section` below. * - `layout `_ - Implements a basic `layout object `_ * - `legenditem `_ From 5729e3249b7ba81d42dd4a641cdf1231f5cdb9b0 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 15:19:04 -0400 Subject: [PATCH 140/159] adding more section links --- docs/navlayers.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/navlayers.rst b/docs/navlayers.rst index 550260fa..921f905c 100644 --- a/docs/navlayers.rst +++ b/docs/navlayers.rst @@ -25,7 +25,7 @@ Core Modules * - `gradient `_ - Implements a basic `gradient object `_ * - `layer `_ - - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding :ref:`Section` below. + - Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding :ref:`section` below. * - `layout `_ - Implements a basic `layout object `_ * - `legenditem `_ @@ -47,7 +47,7 @@ Manipulator Scripts * - script - description * - `layerops `_ - - Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. + - Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding :ref:`section` below. Exporter Scripts ------------------------ @@ -59,9 +59,9 @@ Exporter Scripts * - script - description * - `to_excel `_ - - Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. + - Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding `section` below. * - `to_svg `_ - - Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export. + - Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding `section`_ - - Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. + - Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding `section` below. * - `usage_generator `_ - - Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. + - Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding `section` below. * - `sum_generator `_ - - Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. + - Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding `section Date: Thu, 20 Jul 2023 15:21:33 -0400 Subject: [PATCH 141/159] fixing links --- docs/attacktoexcel.rst | 10 +++++----- docs/getting_started.rst | 2 +- docs/navlayercore.rst | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/attacktoexcel.rst b/docs/attacktoexcel.rst index c11fc9ec..d09fb62c 100644 --- a/docs/attacktoexcel.rst +++ b/docs/attacktoexcel.rst @@ -1,8 +1,8 @@ ATT&CK to Excel ============================================== -ATT&CK to Excel contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. -It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. +ATT&CK to Excel contains a module for converting `ATT&CK STIX data `_ to Excel spreadsheets. +It also provides a means to access ATT&CK data as `Pandas `_ DataFrames for data analysis. Usage: ----- @@ -76,7 +76,7 @@ overview of the available methods follows. stixToDf ----- -stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for +stixToDf provides various methods to process and manipulate the STIX data in order to create `Pandas `_ DataFrames for processing. A brief overview of these methods follows. .. list-table:: @@ -125,10 +125,10 @@ Unlike the STIX dataset, objects that have been revoked or deprecated are not in Accessing the Pandas DataFrames ----- -Internally, attackToExcel stores the parsed STIX data as [Pandas](https://pandas.pydata.org/) DataFrames. +Internally, attackToExcel stores the parsed STIX data as `Pandas `_ DataFrames. These can be retrieved for use in data analysis. -Example of accessing [Pandas](https://pandas.pydata.org/) DataFrames: +Example of accessing `Pandas `_ DataFrames: .. code-block:: python diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 271b7ca7..f6141e93 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -27,7 +27,7 @@ you can read more about other modules in this library under the "Additional Modu Additional Modules ------------------------ -More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual README files for each module linked below. +More detailed information and examples about the specific usage of the additional modules in this package can be found in the individual documents for each module linked below. .. list-table:: diff --git a/docs/navlayercore.rst b/docs/navlayercore.rst index 8bf4792b..cabb0c70 100644 --- a/docs/navlayercore.rst +++ b/docs/navlayercore.rst @@ -2,7 +2,7 @@ Layers Core =============== This subcomponent, as part of the larger navlayers module, is responsible for Layer objects. Please note, this -documentation assumes familiarity with the [ATT&CK Navigator layer format](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md). +documentation assumes familiarity with the `ATT&CK Navigator layer format `_. The main handle for this implementation is the Layer, which stores an individual instance of a LayerObj object, which further references the various sub-objects that make up a complete Layer. A visual representation of this object breakdown can be seen here (please note there are other fields, these are just the objects): From 24f3b37980a543ef22e027c5e4ff42fd190b4df5 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 15:27:05 -0400 Subject: [PATCH 142/159] fixing links and making more subheaders --- docs/collections.rst | 16 +++++++----- docs/stix_primer/access-attack.rst | 39 ++++++++++++++++++------------ 2 files changed, 34 insertions(+), 21 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index f6359e2a..6d7082e8 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -20,11 +20,12 @@ Collections Scripts * - `index_to_markdown `_ - Provides a means by which to convert a `collection index `_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below. * - `collection_to_index `_ - - Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding [section](#collection_to_index.py) below. + - Provides a means by which to convert a `collection `_ into a easy-to-share `index file `_. More information can be found in the corresponding :ref:`section` below. * - `stix_to_collection `_ - - Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding [section](#stix_to_collection.py) below. + - Provides a means by which to convert raw stix (in the form of `bundles `_) into a `collection `_. More information can be found in the corresponding :ref:`section` below. -**index_to_markdown.py** +index_to_markdown.py +------------------------ `index_to_markdown.py` provides the `IndexToMarkdown` class, which provides a way to transform an existing `collection index file `_ @@ -33,7 +34,8 @@ The `IndexToMarkdown` class is very simple, and provides a single method, `index which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown. An example of how to use the class, and method, can be found below. -**Example Usage** +Example Usage +------------------------ .. code-block:: python @@ -48,7 +50,8 @@ An example of how to use the class, and method, can be found below. print(generated_md) -**collection_to_index.py** +collection_to_index.py +------------------------ `collection_to_index.py` provides the `CollectionToIndex` class, which proves a means by which to summarize existing `collections `_ @@ -81,7 +84,8 @@ Example Usage print(output_indexC) -**stix_to_collection.py** +stix_to_collection.py +------------------------ `stix_to_collection.py` provides the `STIXToCollection` class, which proves a means by which to convert existing stix bundles into ones containing a diff --git a/docs/stix_primer/access-attack.rst b/docs/stix_primer/access-attack.rst index a55e2b7f..11aa12f9 100644 --- a/docs/stix_primer/access-attack.rst +++ b/docs/stix_primer/access-attack.rst @@ -1,14 +1,13 @@ Accessing ATT&CK data in python =============== -**Accessing ATT&CK data in python** - There are several ways to acquire the ATT&CK data in Python. All of them will provide an object implementing the DataStore API and can be used interchangeably with the recipes provided in the `STIX recipes `_ section. This section utilizes the `stix2 python library `_ . Please refer to the `STIX2 Python API Documentation `_ for more information on how to work with STIX programmatically. -**Requirements and imports** +Requirements and imports +---------------------- Before installing requirements, we recommend setting up a virtual environment: @@ -19,7 +18,8 @@ Before installing requirements, we recommend setting up a virtual environment: - macOS and Linux: ``source env/bin/activate`` - Windows: ``env/Scripts/activate.bat`` -**stix2** +stix2 +---------------------- stix2 can be installed by following the instructions on their `repository `_ . Imports for the recipes in this repository can be done from the base package, for example: @@ -37,7 +37,8 @@ However, if you are aiming to extend the ATT&CK dataset with new objects or impl You can see a full list of the classes which have versioned imports `here `_ . -**taxii2client** +taxii2client +------------- taxii2-client can be installed by following the instructions on their `repository `_ . The ATT&CK TAXII server implements the 2.0 version of the TAXII specification, but the default import of ``taxii2client`` (version 2.0.0 and above) uses the 2.1 version of the TAXII specification, which can lead to 406 responses when connecting to our TAXII server if not accounted for. @@ -48,7 +49,8 @@ If the TAXII Client is getting a 406 Response, make sure you are running the lat from taxii2client.v20 import Collection -**Access local content** +Access local content +-------------------- Many users may opt to access the ATT&CK content via a local copy of the STIX data on this repo. This can be advantageous for several reasons: @@ -56,7 +58,8 @@ Many users may opt to access the ATT&CK content via a local copy of the STIX dat - User can modify the ATT&CK content if desired - Downloaded copy is static, so updates to the ATT&CK catalog won't cause bugs in automated workflows. User can still manually update by cloning a fresh version of the data -**Access via FileSystemSource** +Access via FileSystemSource +--------------------------- Each domain in this repo is formatted according to the `STIX2 FileSystem spec `_ . Therefore you can use a ``FileSystemSource`` to load a domain, for example to load the enterprise-attack domain: @@ -68,7 +71,8 @@ Therefore you can use a ``FileSystemSource`` to load a domain, for example to lo src = FileSystemSource('./cti/enterprise-attack') -**Access via bundle** +Access via bundle +------------------ If you instead prefer to download just the domain bundle, e.g `enterprise-attack.json `_ , you can still load this using a MemoryStore: @@ -80,14 +84,16 @@ If you instead prefer to download just the domain bundle, e.g `enterprise-attack src.load_from_file("enterprise-attack.json") -**Access live content** +Access live content +------------------- Some users may instead prefer to access "live" ATT&CK content over the internet. This is advantageous for several reasons: - Always stays up to date with the evolving ATT&CK catalog - Doesn't require an initial download of the ATT&CK content, generally requires less setup -**Access from the ATT&CK TAXII server** +Access from the ATT&CK TAXII server +---------------------------------- Users can access the ATT&CK data from the official ATT&CK TAXII server. In TAXII, the ATT&CK domains are represented as collections with static IDs: @@ -136,7 +142,8 @@ The following recipe demonstrates how to access the enterprise-attack data from For more about TAXII, please see oasis-open's `Introduction to TAXII `_ . -**Access from Github via requests** +Access from Github via requests +---------------------- Users can alternatively access the data from MITRE/CTI using HTTP requests, and load the resulting content into a MemoryStore. While typically the TAXII method is more desirable for "live" access, this method can be useful if you want to @@ -155,11 +162,12 @@ access data on a branch of the MITRE/CTI repo (the TAXII server only holds the m src = get_data_from_branch("enterprise-attack") -**Access a specific version of ATT&CK** +Access a specific version of ATT&CK +---------------------- -ATT&CK versions are tracked on the MITRE/CTI repo using `tags `_ . Tags prefixed with ``ATT&CK-v`` correspond to ATT&CK versions and tags prefixed with ``CAPEC-v`` correspond to CAPEC versions. You can find more information about ATT&CK versions on the [versions of ATT&CK page](https://attack.mitre.org/resources/versions/) on the ATT&CK website. +ATT&CK versions are tracked on the MITRE/CTI repo using `tags `_ . Tags prefixed with ``ATT&CK-v`` correspond to ATT&CK versions and tags prefixed with ``CAPEC-v`` correspond to CAPEC versions. You can find more information about ATT&CK versions on the `versions of ATT&CK page `_ on the ATT&CK website. -In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the [requests method](#access-from-github-via-requests) to access a particular version of ATT&CK: +In addition to checking out the repo under the tag for a given version or downloading the STIX from github using your browser, you can also use a variation on the :ref:`requests method` to access a particular version of ATT&CK: .. code-block:: python @@ -187,7 +195,8 @@ You can get a list of ATT&CK versions programmatically using the github API: # versions = ["1.0", "2.0", ...] -**Access multiple domains simultaneously** +Access multiple domains simultaneously +---------------------- Because ATT&CK is stored in multiple domains (as of this writing, enterprise-attack, mobile-attack and ics-attack), the above methodologies will only allow you to work with a single domain at a time. While oftentimes the hard separation of domains is advantageous, occasionally it is useful to combine From 656c39991397c5894cbba823e0d9e95500a870e1 Mon Sep 17 00:00:00 2001 From: Sun Date: Thu, 20 Jul 2023 15:27:44 -0400 Subject: [PATCH 143/159] fixing links and making more subheaders part 2 --- docs/diffinstix.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 88c72353..87ec809c 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -8,7 +8,8 @@ Run `diff_stix -h` for full usage instructions. Usage ----- -**Command Line** +Command Line +----- Print full usage instructions: @@ -55,7 +56,8 @@ Example execution: diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ -**Changelog JSON format** +Changelog JSON format +----- The changelog helper script has the option to output a JSON file with detailed differences between ATT&CK releases. This is the overall structure you can expect to find in the file. @@ -128,7 +130,8 @@ A brief explanation of key pieces can be found below. - ATT&CK objects which are no longer found in the STIX data. This should almost never happen. -**Changed Objects** +Changed Objects +----- The bulk of the changelog file consists of lists of JSON objects. If you are familiar with reading the STIX format, they may look famliar, yet a little "off". From f199cc12d98c2093998cdf69ee147d78195ce261 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 21 Jul 2023 13:50:18 -0400 Subject: [PATCH 144/159] fixed formatting in diffinstix --- docs/diffinstix.rst | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/diffinstix.rst b/docs/diffinstix.rst index 87ec809c..b7c88f81 100644 --- a/docs/diffinstix.rst +++ b/docs/diffinstix.rst @@ -3,7 +3,7 @@ Diff Stix Diff Stix contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. -Run `diff_stix -h` for full usage instructions. +Run ``diff_stix -h`` for full usage instructions. Usage ----- @@ -93,8 +93,8 @@ A brief explanation of key pieces can be found below. } -* The top-level objects include information about specific domains as well as `new-contributors`, which are only found in the newer ATT&CK release. -* For domain changes, they are broken down by object type, e.g. `techniques` or `mitigations`. +* The top-level objects include information about specific domains as well as ``new-contributors``, which are only found in the newer ATT&CK release. +* For domain changes, they are broken down by object type, e.g. ``techniques`` or ``mitigations``. * The following table helps break down the change types that are currently tracked. .. list-table:: Title @@ -104,28 +104,28 @@ A brief explanation of key pieces can be found below. * - field - type - description - * - `additions` + * - ``additions`` -array[object] - ATT&CK objects which are only present in the new STIX data. - * - `major_version_changes`` + * - ``major_version_changes`` - array[object] - ATT&CK objects that have a major version change. (e.g. 1.0 → 2.0). - * - `minor_version_changes` + * - ``minor_version_changes`` - array[object] - ATT&CK objects that have a minor version change. (e.g. 1.0 → 1.1). - * - `other_version_changes` + * - ``other_version_changes`` - array[object] - array[object] | ATT&CK objects that have a version change of any other kind. (e.g. 1.0 → 1.3). These are unintended, but can be found in previous releases. - * - `patches` + * - ``patches`` - array[object] - ATT&CK objects that have been patched while keeping the version the same. - * - `revocations` + * - ``revocations`` - array[object] - ATT&CK objects which are revoked by a different object. - * - `deprecations` + * - ``deprecations`` - array[object] - ATT&CK objects which are deprecated and no longer in use, and not replaced. - * - `deletions` + * - ``deletions`` - array[object - ATT&CK objects which are no longer found in the STIX data. This should almost never happen. @@ -136,7 +136,7 @@ Changed Objects The bulk of the changelog file consists of lists of JSON objects. If you are familiar with reading the STIX format, they may look famliar, yet a little "off". That is because there are a few fields that have been added in some cases depending on what section they appear in. -For example, objects that are brand new do not have `previous_version` available to them. +For example, objects that are brand new do not have ``previous_version`` available to them. The following table lists the extra fields that can be found in objects in the changelog. .. list-table:: @@ -147,25 +147,25 @@ The following table lists the extra fields that can be found in objects in the c - Required - Type - Description - * - `changelog_mitigations` + * - ``changelog_mitigations`` - false - object - - Three lists for `shared`, `new`, and `dropped` for Mitigations that are related to a Technique between versions. - * - `changelog_detections` + - Three lists for ``shared``, ``new``, and ``dropped`` for Mitigations that are related to a Technique between versions. + * - ``changelog_detections`` - false - object - HTML rendering of a table that displays the differences between descriptions for an ATT&CK object. - * - `detailed_diff` + * - ``detailed_diff`` - false - string - A python DeepDiff object that has been JSON serialized which represents STIX changes for an ATT&CK object between releases. - * - `previous_version` + * - ``previous_version`` - false - string - If the object existed in the previous release, then it denotes the version the object was in the previous release. - * - `version_change` + * - ``version_change`` - false - string - - If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' + - If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '``old-version`` → ``new-version``' From 9c4aa4599344b92fe8b07160ea843f93d57e5f21 Mon Sep 17 00:00:00 2001 From: Sun Date: Fri, 21 Jul 2023 14:46:01 -0400 Subject: [PATCH 145/159] fixed formatting in navlayercore --- docs/navlayercore.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/navlayercore.rst b/docs/navlayercore.rst index cabb0c70..dceebc12 100644 --- a/docs/navlayercore.rst +++ b/docs/navlayercore.rst @@ -77,6 +77,7 @@ displayed below, is to create and modify instances of the core objects in the li produce equivalent internal layers once completed. .. code-block:: python + import mitreattack.navlayers as navlayers layer_example = navlayers.Layer(name="example", domain="enterprise-attack") # arguments required for every layer @@ -187,6 +188,7 @@ in the source code for them. _LayerObj().get_dict() # Export Layer as a dictionary object **Versions Object** + .. code-block:: python Versions().layer # String denoting Layer format version @@ -195,6 +197,7 @@ in the source code for them. Versions().get_dict() # Export Version data as a dictionary object **Filter Object** + .. code-block:: python Filter().domain # String denoting the domain for the Filter @@ -203,7 +206,9 @@ in the source code for them. Please note that although not documented here, there is another Filter object variant, Filterv3, which exists for backwards compatibility reasons. + **Layout Object** + .. code-block:: python Layout().layout # String denoting which layout form to use From 2faa9193c791241644827206347c85be3a9fbfe9 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 13:35:42 -0400 Subject: [PATCH 146/159] fixing navlayercore issue --- docs/navlayercore.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/navlayercore.rst b/docs/navlayercore.rst index dceebc12..3632b7dc 100644 --- a/docs/navlayercore.rst +++ b/docs/navlayercore.rst @@ -154,6 +154,7 @@ for processing and other functionality that are not documented here, though docu in the source code for them. **Layer Object** + .. code-block:: python Layer().layer # Stores the raw LayerObj file From da8eb1fa1f00b018b111f2a1a358651f7ab4e770 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 15:48:09 -0400 Subject: [PATCH 147/159] fixed code styling in collections --- docs/collections.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/collections.rst b/docs/collections.rst index 6d7082e8..cfaae6a8 100644 --- a/docs/collections.rst +++ b/docs/collections.rst @@ -27,10 +27,10 @@ Collections Scripts index_to_markdown.py ------------------------ -`index_to_markdown.py` provides the `IndexToMarkdown` class, which provides a way to transform an existing +``index_to_markdown.py`` provides the ``IndexToMarkdown`` class, which provides a way to transform an existing `collection index file `_ into a markdown file for easy of use and reference. -The `IndexToMarkdown` class is very simple, and provides a single method, `index_to_markdown`, +The ``IndexToMarkdown`` class is very simple, and provides a single method, ``index_to_markdown``, which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown. An example of how to use the class, and method, can be found below. From 69f81966fd6462aa6d6d16638667e70fe8cdcf35 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:07:43 -0400 Subject: [PATCH 148/159] fixed code styling in overview --- docs/stix_primer/overview.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/stix_primer/overview.rst b/docs/stix_primer/overview.rst index a9424b17..817e005a 100644 --- a/docs/stix_primer/overview.rst +++ b/docs/stix_primer/overview.rst @@ -2,7 +2,8 @@ Overview =============== -This is a short primer on how to manipulate STIX data from ATT&CK. While not comprehensive, it will provide you with a quick starting guide. +This is a short primer on how to manipulate STIX data from ATT&CK. While not comprehensive, it will provide you with a quick starting guide. +You can go to the `Access ATTACK `_ page to find out how to get started. .. toctree:: :maxdepth: 1 From d92852f3b701edf4d2dfc0ceee6ee988edc30411 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:40:31 -0400 Subject: [PATCH 149/159] fixed code styling in deprecated revoked page --- docs/stix_primer/deprecated_revoked.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/deprecated_revoked.rst b/docs/stix_primer/deprecated_revoked.rst index 875d19e0..bbb5bcf4 100644 --- a/docs/stix_primer/deprecated_revoked.rst +++ b/docs/stix_primer/deprecated_revoked.rst @@ -1,6 +1,6 @@ Working with deprecated and revoked objects =============== -Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either `x_mitre_deprecated` or `revoked`) noting their status. In the case of revoked objects, a relationship of type `revoked-by` is also created targeting the replacing object. +Objects that are deemed no longer beneficial to track as part of the knowledge base are marked as deprecated, and objects which are replaced by a different object are revoked. In both cases, the old object is marked with a field (either ``x_mitre_deprecated`` or ``revoked``) noting their status. In the case of revoked objects, a relationship of type ``revoked-by`` is also created targeting the replacing object. Unlike other objects in the dataset, relationships cannot be revoked or deprecated. Relationships are considered deprecated/revoked if one of the objects it is attached to is revoked or deprecated. \ No newline at end of file From 26e8a61c672ddd73854243f6d667b6348deb9570 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:41:18 -0400 Subject: [PATCH 150/159] removed special character from multiple objects page --- docs/stix_primer/multiple_objects.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/multiple_objects.rst b/docs/stix_primer/multiple_objects.rst index 77dfa923..4aface21 100644 --- a/docs/stix_primer/multiple_objects.rst +++ b/docs/stix_primer/multiple_objects.rst @@ -3,4 +3,4 @@ Getting multiple objects The recipes in this following section address how to query the dataset for multiple objects. -⚠ When working with queries to return objects based on a set of characteristics, it is likely that you'll end up with a few objects which are no longer maintained by ATT&CK. These are objects marked as deprecated or revoked. We keep these outdated objects around so that workflows depending on them don't break, but we recommend you avoid using them when possible. Please see the section [Working with deprecated and revoked objects](#Working-with-deprecated-and-revoked-objects) for more information. \ No newline at end of file +When working with queries to return objects based on a set of characteristics, it is likely that you'll end up with a few objects which are no longer maintained by ATT&CK. These are objects marked as deprecated or revoked. We keep these outdated objects around so that workflows depending on them don't break, but we recommend you avoid using them when possible. Please see the section [Working with deprecated and revoked objects](#Working-with-deprecated-and-revoked-objects) for more information. \ No newline at end of file From 77db64ae29e508a6b1c3198484630ec17b09cbb3 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:43:51 -0400 Subject: [PATCH 151/159] more syntax fixes --- docs/stix_primer/objects_by_type.rst | 2 +- docs/stix_primer/techniques_subtechniques.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/stix_primer/objects_by_type.rst b/docs/stix_primer/objects_by_type.rst index 339f69f5..641ff7c4 100644 --- a/docs/stix_primer/objects_by_type.rst +++ b/docs/stix_primer/objects_by_type.rst @@ -1,7 +1,7 @@ Objects by type =============== -See [The ATT&CK data model](#The-ATTCK-Data-Model) for mappings of ATT&CK type to STIX type. +See `The ATT&CK data model `_ for mappings of ATT&CK type to STIX type. .. code-block:: python diff --git a/docs/stix_primer/techniques_subtechniques.rst b/docs/stix_primer/techniques_subtechniques.rst index 96ae2d73..d4a7a913 100644 --- a/docs/stix_primer/techniques_subtechniques.rst +++ b/docs/stix_primer/techniques_subtechniques.rst @@ -1,7 +1,7 @@ Getting techniques or sub-techniques =============== -ATT&CK Techniques and sub-techniques are both represented as `attack-pattern` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. +ATT&CK Techniques and sub-techniques are both represented as ``attack-pattern`` objects. Therefore further parsing is necessary to get specifically techniques or sub-techniques. .. code-block:: python From ea3737ba2319bdefe12ddca1abe5167b96d36d6c Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:44:42 -0400 Subject: [PATCH 152/159] more syntax fixes --- docs/stix_primer/objects_since_date.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/objects_since_date.rst b/docs/stix_primer/objects_since_date.rst index 49454bb8..d71cbfef 100644 --- a/docs/stix_primer/objects_since_date.rst +++ b/docs/stix_primer/objects_since_date.rst @@ -25,4 +25,4 @@ Sometimes you may want to get a list of objects which have been created or modif get_modified_after(src, "2018-10-01T00:14:20.652Z") -We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to [check the list of released versions of ATT&CK](https://github.com/mitre/cti/blob/master/USAGE.md#access-a-specific-version-of-attck). +We don't recommend you use this method to detect a change to the contents of the knowledge base. For detecting an update to the overall knowledge base we recommend using requests to `check the list of released versions of ATT&CK `_ . From 7e1173f39661ee0bea47232661a913ec4b83f009 Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:45:55 -0400 Subject: [PATCH 153/159] removed duplicate entry in overview index file --- docs/stix_primer/overview.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/stix_primer/overview.rst b/docs/stix_primer/overview.rst index 817e005a..ec1943dc 100644 --- a/docs/stix_primer/overview.rst +++ b/docs/stix_primer/overview.rst @@ -24,7 +24,6 @@ You can go to the `Access ATTACK Date: Mon, 24 Jul 2023 16:52:21 -0400 Subject: [PATCH 154/159] fixed github issue link --- docs/stix_primer/remove_revoked_deprecated.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/remove_revoked_deprecated.rst b/docs/stix_primer/remove_revoked_deprecated.rst index 0de3e7d7..ab873dee 100644 --- a/docs/stix_primer/remove_revoked_deprecated.rst +++ b/docs/stix_primer/remove_revoked_deprecated.rst @@ -5,7 +5,7 @@ Revoked and deprecated objects are kept in the knowledge base so that workflows broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no longer maintained by ATT&CK. -We recommend _not_ using built-in STIX filters for removing revoked objects (e.g `Filter('revoked', '=', False)`). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See [issue #127](https://github.com/mitre/cti/issues/127) for more details. +We recommend _not_ using built-in STIX filters for removing revoked objects (e.g ``Filter('revoked', '=', False)``). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See `issue #127 `_ for more details. .. code-block:: python From bd3b13d32513e2cee3487fc0e21badd5e562173b Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 24 Jul 2023 16:53:17 -0400 Subject: [PATCH 155/159] fixed italics in removed revoked page --- docs/stix_primer/remove_revoked_deprecated.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/remove_revoked_deprecated.rst b/docs/stix_primer/remove_revoked_deprecated.rst index ab873dee..4cd21562 100644 --- a/docs/stix_primer/remove_revoked_deprecated.rst +++ b/docs/stix_primer/remove_revoked_deprecated.rst @@ -5,7 +5,7 @@ Revoked and deprecated objects are kept in the knowledge base so that workflows broken. We recommend you filter out revoked and deprecated objects from your views whenever possible since they are no longer maintained by ATT&CK. -We recommend _not_ using built-in STIX filters for removing revoked objects (e.g ``Filter('revoked', '=', False)``). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See `issue #127 `_ for more details. +We recommend `not` using built-in STIX filters for removing revoked objects (e.g ``Filter('revoked', '=', False)``). This is because the behavior of this specific filter is inconsistent depending on the method of access (using local data or accessing via the TAXII server). We recommend using the following code example to filter revoked objects instead. See `issue #127 `_ for more details. .. code-block:: python From 10ab49133ed96f60e18d851da17aaf0c44184034 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 25 Jul 2023 09:54:54 -0400 Subject: [PATCH 156/159] fixed link in getting started --- docs/getting_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index f6141e93..a09f5eb5 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -8,7 +8,7 @@ Additionally, it includes a USAGE document that provides further examples for ac Install ----- -To use this package, install the mitreattack-python library with `pip `_ : .. code:: bash From 46e6599a74257f404061e4a78b0fb05af9bf7e30 Mon Sep 17 00:00:00 2001 From: Sun Date: Tue, 25 Jul 2023 09:58:37 -0400 Subject: [PATCH 157/159] switched readme links to readthedocs links --- docs/getting_started.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index a09f5eb5..24fe676c 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -39,16 +39,16 @@ More detailed information and examples about the specific usage of the additiona - documentation * - `navlayers `_ - A collection of utilities for working with `ATT&CK Navigator `_ layers. Provides the ability to import, export, and manipulate layers. Layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images. - - Further documentation can be found `here `_ + - Further documentation can be found `here `_ * - `attackToExcel `_ - A collection of utilities for converting `ATT&CK STIX data `_ to Excel spreadsheets. It also provides access to `Pandas `_ DataFrames representing the dataset for use in data analysis. - - Further documentation can be found `here `_ + - Further documentation can be found `here `_ * - `collections `_ - A set of utilities for working with `ATT&CK Collections and Collection Indexes `_. Provides functionalities for converting and summarizing data in collections and collection indexes, as well as generating a collection from a raw stix bundle input. - - Further documentation can be found `here `_. + - Further documentation can be found `here `_. * - `diffStix `_ - Create markdown, HTML, JSON and/or ATT&CK Navigator layers reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. Run ``diff_stix -h`` for full usage instructions. - - Further documentation can be found `here `_ + - Further documentation can be found `here `_ Related MITRE Work From 22504e8a4b48866227befdf0a3c82a99d9744bd9 Mon Sep 17 00:00:00 2001 From: Sun Date: Wed, 26 Jul 2023 18:20:00 -0400 Subject: [PATCH 158/159] removed duplicate information from READMEs --- mitreattack/attackToExcel/README.md | 107 +----- mitreattack/collections/README.md | 93 +---- mitreattack/diffStix/README.md | 118 +----- mitreattack/navlayers/README.md | 518 +-------------------------- mitreattack/navlayers/core/README.md | 241 +------------ 5 files changed, 5 insertions(+), 1072 deletions(-) diff --git a/mitreattack/attackToExcel/README.md b/mitreattack/attackToExcel/README.md index b3920890..13427ad8 100644 --- a/mitreattack/attackToExcel/README.md +++ b/mitreattack/attackToExcel/README.md @@ -1,108 +1,3 @@ # ATT&CK To Excel -This folder contains a module for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. -It also provides a means to access ATT&CK data as [Pandas](https://pandas.pydata.org/) DataFrames for data analysis. - -## Usage - -### Command Line - -Print full usage instructions: - -```shell -python3 attackToExcel.py -h -``` - -Example execution: - -```shell -python3 attackToExcel.py -``` - -Build a excel files corresponding to a specific domain and version of ATT&CK: - -```shell -python3 attackToExcel -domain mobile-attack -version v5.0 -``` - -### Module - -Example execution targeting a specific domain and version: - -```python -import mitreattack.attackToExcel.attackToExcel as attackToExcel - -attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder") -``` - -## Interfaces - -### attackToExcel - -attackToExcel provides the means by which to convert/extract the ATT&CK STIX data to Excel spreadsheets. A brief -overview of the available methods follows. - -| method name | arguments | usage | -|:------------|:----------|:------| -|get_stix_data|`domain`: the domain of ATT&CK to fetch data from
`version`: optional parameter indicating which version to fetch data from (such as "v8.1"). If omitted retrieves the most recent version of ATT&CK.
`remote`: optional parameter that provides a URL of a remote ATT&CK Workbench instance to grab data from.| Retrieves the ATT&CK STIX data for the specified version and returns it as a MemoryStore object| -|build_dataframes| `src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to| Builds a Pandas DataFrame collection as a dictionary, with keys for each type, based on the ATT&CK data provided| -|write_excel| `dataframes`: pandas DataFrame dictionary (generated by build_dataframes)
`domain`: domain of ATT&CK that `dataframes` corresponds to
`version`: optional parameter indicating which version of ATT&CK is in use
`output_dir`: optional parameter specifying output directory| Writes out DataFrame based ATT&CK data to excel files| -|export| `domain`: the domain of ATT&CK to download
`version`: optional parameter specifying which version of ATT&CK to download
`output_dir`: optional parameter specifying output directory| Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets | - -### stixToDf - -stixToDf provides various methods to process and manipulate the STIX data in order to create [Pandas](https://pandas.pydata.org/) DataFrames for -processing. A brief overview of these methods follows. - -| method name | arguments | usage | -|:------------|:----------|:------| -|techniquesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX techniques from the provided data and returns corresponding Pandas DataFrames.| -|tacticsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX tactics from the provided data and returns corresponding Pandas DataFrames.| -|softwareToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX software from the provided data and returns corresponding Pandas DataFrames.| -|groupsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX groups from the provided data and returns corresponding Pandas DataFrames.| -|mitigationsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX mitigations from the provided data and returns corresponding Pandas DataFrames.| -|relationshipsToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX relationships from the provided data and returns corresponding Pandas DataFrames.| -|matricesToDf|`src`: MemoryStore or other stix2 DataSource object holding domain data
`domain`: domain of ATT&CK that `src` corresponds to | Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]`| - -## Spreadsheet format - -The Excel representation of the ATT&CK dataset includes both master spreadsheets, -containing all object types, and individual spreadsheets for each object type. -The individual type spreadsheets break out relationships (e.g procedure examples connecting groups to techniques) -into separate sheets by relationship type, while the master spreadsheet includes all relationship types in a single sheet. -Otherwise, the representation is identical. - -A citations sheet can be used to look up the in-text citations which appear in some fields. -For domains that include multiple matrices, such as Mobile ATT&CK, each matrix gets its own named sheet. -Unlike the STIX dataset, objects that have been revoked or deprecated are not included in the spreadsheets. - -## Accessing the Pandas DataFrames - -Internally, attackToExcel stores the parsed STIX data as [Pandas](https://pandas.pydata.org/) DataFrames. -These can be retrieved for use in data analysis. - -Example of accessing [Pandas](https://pandas.pydata.org/) DataFrames: - -```python -import mitreattack.attackToExcel.attackToExcel as attackToExcel -import mitreattack.attackToExcel.stixToDf as stixToDf - -# download and parse ATT&CK STIX data -attackdata = attackToExcel.get_stix_data("enterprise-attack") -techniques_data = stixToDf.techniquesToDf(attackdata, "enterprise-attack") - -# show T1102 and sub-techniques of T1102 -techniques_df = techniques_data["techniques"] -print(techniques_df[techniques_df["ID"].str.contains("T1102")]["name"]) -# 512 Web Service -# 38 Web Service: Bidirectional Communication -# 121 Web Service: Dead Drop Resolver -# 323 Web Service: One-Way Communication -# Name: name, dtype: object - -# show citation data for LOLBAS Wmic reference -citations_df = techniques_data["citations"] -print(citations_df[citations_df["reference"].str.contains("LOLBAS Wmic")]) -# reference citation url -# 1010 LOLBAS Wmic LOLBAS. (n.d.). Wmic.exe. Retrieved July 31, 2... https://lolbas-project.github.io/lolbas/Binari... -``` +If you wish to read more about the ATT&CK to Excel module, please click [here](https://mitreattack-python.readthedocs.io/en/126-docs-add-section-to-docs-for-accessing-stix/attacktoexcel.html) for more information. diff --git a/mitreattack/collections/README.md b/mitreattack/collections/README.md index 412aa48d..542ef720 100644 --- a/mitreattack/collections/README.md +++ b/mitreattack/collections/README.md @@ -1,94 +1,3 @@ # collections -This folder contains modules and scripts for working with ATT&CK collections. -Collections are sets of ATT&CK STIX objects, grouped for user convienence. -For more information about ATT&CK collections, see the corresponding -[ATT&CK documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). - -## Collections Scripts - -| script | description | -|:-------|:------------| -|[index_to_markdown](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/index_to_markdown.py)| Provides a means by which to convert a [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) into a human-readable markdown file. More information can be found in the corresponding [section](#index_to_markdown.py) below.| -|[collection_to_index](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/collection_to_index.py)| Provides a means by which to convert a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) into a easy-to-share [index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes). More information can be found in the corresponding [section](#collection_to_index.py) below.| -|[stix_to_collection](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py)| Provides a means by which to convert raw stix (in the form of [bundles](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg)) into a [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections). More information can be found in the corresponding [section](#stix_to_collection.py) below.| - -### index_to_markdown.py - -`index_to_markdown.py` provides the `IndexToMarkdown` class, which provides a way to transform an existing -[collection index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) -into a markdown file for easy of use and reference. -The `IndexToMarkdown` class is very simple, and provides a single method, `index_to_markdown`, -which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown. -An example of how to use the class, and method, can be found below. - -#### Example Usage - -```python -import json -from mitreattack.collections import IndexToMarkdown - -with open('collection_index.json', 'r') as input_file: - with open('collection_index.md', 'w') as output_file: - input_index = json.load(input_file) - generated_md = IndexToMarkdown.index_to_markdown(input_index) # Convert index to markdown - output_file.write(generated_md) -print(generated_md) -``` - -### collection_to_index.py - -`collection_to_index.py` provides the `CollectionToIndex` class, which proves a means by which to summarize existing -[collections](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) -into a single [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) file. -The `CollectionToIndex` class contains the `generate_index` function, which when provided with a name, description, root url (pointing to where the raw collections are stored), -and a list of either files, folders, or already loaded bundles in the form of dictionaries, will create a summarizing index. - -#### Example Usage - -```python -import json -from mitreattack.collections import CollectionToIndex - -output_indexA = CollectionToIndex.generate_index(name='example', description='example index', - root_url='www.example.com', - files=['/path/to/collection1.json', '/path/to/collection2.json'], - folders=None, sets=None) -output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index', - root_url='www.example.com', - files=None, folders=['/path/to/folder/with/collections'], sets=None) -with open('path/to/bundle/bundleC.json', 'r') as f: - data = json.load(f) -output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index', - root_url='www.example.com', - files=None, folders=None, sets=[data]) -print(output_indexA) -print(output_indexB) -print(output_indexC) -``` - -### stix_to_collection.py - -`stix_to_collection.py` provides the `STIXToCollection` class, which proves a means by which to convert -existing stix bundles into ones containing a -[collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) object. -The `STIXToCollection` class contains the `stix_to_collection` function, which when provided with a starter bundle, -a name, a version, and an optional description, will output a modified bundle that contains a summary collection object. - -#### Example Usage - -```python -import json -from mitreattack.collections import STIXToCollection - -with open('path/to/bundle/bundle2_0.json', 'r') as f: - data = json.load(f) -output_bundleA = STIXToCollection.stix_to_collection(bundle=data, name='collectionA', version='9.1', description="demo bundle (2.0)") - -with open('path/to/bundle/bundle2_1.json', 'r') as f: - data = json.load(f) -output_bundleB = STIXToCollection.stix_to_collection(bundle=data, name='collectionB', version='9.0', description="demo bundle (2.1)") - -print(output_bundleA) -print(output_bundleB) -``` +If you wish to read more about the collections module, please click [here](https://mitreattack-python.readthedocs.io/en/126-docs-add-section-to-docs-for-accessing-stix/collections.html) for more information. diff --git a/mitreattack/diffStix/README.md b/mitreattack/diffStix/README.md index 80453812..b33d06b2 100644 --- a/mitreattack/diffStix/README.md +++ b/mitreattack/diffStix/README.md @@ -1,119 +1,3 @@ # Diff Stix -This folder contains a module for creating markdown, HTML, JSON and/or ATT&CK Navigator layers -reporting on the changes between two versions of the STIX2 bundles representing the ATT&CK content. -Run `diff_stix -h` for full usage instructions. - -## Usage - -### Command Line - -Print full usage instructions: - -```shell -# You must run `pip install mitreattack-python` in order to access the diff_stix command -diff_stix --help -usage: diff_stix [-h] [--old OLD] [--new NEW] [--domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...]] [--markdown-file MARKDOWN_FILE] [--html-file HTML_FILE] [--html-file-detailed HTML_FILE_DETAILED] - [--json-file JSON_FILE] [--layers [LAYERS ...]] [--site_prefix SITE_PREFIX] [--unchanged] [--use-mitre-cti] [--show-key] [--contributors] [--no-contributors] [-v] - -Create changelog reports on the differences between two versions of the ATT&CK content. Takes STIX bundles as input. For default operation, put enterprise-attack.json, mobile-attack.json, and ics-attack.json bundles in 'old' and 'new' folders for the script to compare. - -options: - -h, --help show this help message and exit - --old OLD Directory to load old STIX data from. - --new NEW Directory to load new STIX data from. - --domains {enterprise-attack,mobile-attack,ics-attack} [{enterprise-attack,mobile-attack,ics-attack} ...] - Which domains to report on. Choices (and defaults) are enterprise-attack, mobile-attack, ics-attack - --markdown-file MARKDOWN_FILE - Create a markdown file reporting changes. - --html-file HTML_FILE - Create HTML page from markdown content. - --html-file-detailed HTML_FILE_DETAILED - Create an HTML file reporting detailed changes. - --json-file JSON_FILE - Create a JSON file reporting changes. - --layers [LAYERS ...] - Create layer files showing changes in each domain expected order of filenames is 'enterprise', 'mobile', 'ics', 'pre attack'. If values are unspecified, defaults to output/January_2023_Updates_Enterprise.json, - output/January_2023_Updates_Mobile.json, output/January_2023_Updates_ICS.json, output/January_2023_Updates_Pre.json - --site_prefix SITE_PREFIX - Prefix links in markdown output, e.g. [prefix]/techniques/T1484 - --unchanged Show objects without changes in the markdown output - --use-mitre-cti Use content from the MITRE CTI repo for the -old data - --show-key Add a key explaining the change types to the markdown - --contributors Show new contributors between releases - --no-contributors Do not show new contributors between releases - -v, --verbose Print status messages -``` - -Example execution: - -```shell -diff_stix -v --show-key --html-file output/changelog.html --html-file-detailed output/changelog-detailed.html --markdown-file output/changelog.md --json-file output/changelog.json --layers output/layer-enterprise.json output/layer-mobile.json output/layer-ics.json --old path/to/old/stix/ --new path/to/new/stix/ -``` - -## Changelog JSON format - -The changelog helper script has the option to output a JSON file with detailed differences between ATT&CK releases. -This is the overall structure you can expect to find in the file. -A brief explanation of key pieces can be found below. - -```JSON -{ - "enterprise-attack": { - "techniques": { - "additions": [], - "major_version_changes": [], - "minor_version_changes": [], - "other_version_changes": [], - "patches": [], - "revocations": [], - "deprecations": [], - "deletions": [], - }, - "software": {}, - "groups": {}, - "campaigns": {}, - "mitigations": {}, - "datasources": {}, - "datacomponents": {} - }, - "mobile-attack": {}, - "ics-attack": {}, - "new-contributors": [ - "Contributor A", - "Contributor B", - "Contributor C" - ] -} -``` - -* The top-level objects include information about specific domains as well as `new-contributors`, which are only found in the newer ATT&CK release. -* For domain changes, they are broken down by object type, e.g. `techniques` or `mitigations`. -* The following table helps break down the change types that are currently tracked. - -| Field | Type | Description | -|-------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| `additions` | array[object] | ATT&CK objects which are only present in the new STIX data. | -| `major_version_changes` | array[object] | ATT&CK objects that have a major version change. (e.g. 1.0 → 2.0). | -| `minor_version_changes` | array[object] | ATT&CK objects that have a minor version change. (e.g. 1.0 → 1.1). | -| `other_version_changes` | array[object] | ATT&CK objects that have a version change of any other kind. (e.g. 1.0 → 1.3). These are unintended, but can be found in previous releases. | -| `patches` | array[object] | ATT&CK objects that have been patched while keeping the version the same. | -| `revocations` | array[object] | ATT&CK objects which are revoked by a different object. | -| `deprecations` | array[object] | ATT&CK objects which are deprecated and no longer in use, and not replaced. | -| `deletions` | array[object] | ATT&CK objects which are no longer found in the STIX data. This should almost never happen. | - -### Changed Objects - -The bulk of the changelog file consists of lists of JSON objects. -If you are familiar with reading the STIX format, they may look famliar, yet a little "off". -That is because there are a few fields that have been added in some cases depending on what section they appear in. -For example, objects that are brand new do not have `previous_version` available to them. -The following table lists the extra fields that can be found in objects in the changelog. - -| Field | Required | Type | Description | -|----------------------------|----------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `changelog_mitigations` | false | object | Three lists for `shared`, `new`, and `dropped` for Mitigations that are related to a Technique between versions. | -| `changelog_detections` | false | object | Three lists for `shared`, `new`, and `dropped` for Detections that are related to a Technique between versions. || `description_change_table` | false | string | HTML rendering of a table that displays the differences between descriptions for an ATT&CK object. | -| `detailed_diff` | false | string | A python DeepDiff object that has been JSON serialized which represents STIX changes for an ATT&CK object between releases. | -| `previous_version` | false | string | If the object existed in the previous release, then it denotes the version the object was in the previous release. | -| `version_change` | false | string | If the object existed in the previous release and was changed in the current release, then a descriptive string in the format '`old-version` → `new-version`' | +If you wish to read more about the collections module, please click [here](https://mitreattack-python.readthedocs.io/en/126-docs-add-section-to-docs-for-accessing-stix/diffinstix.html) for more information. \ No newline at end of file diff --git a/mitreattack/navlayers/README.md b/mitreattack/navlayers/README.md index 31bb93d0..d02dd82e 100644 --- a/mitreattack/navlayers/README.md +++ b/mitreattack/navlayers/README.md @@ -1,519 +1,3 @@ # navlayers -This folder contains modules and scripts for working with ATT&CK Navigator layers. -ATT&CK Navigator Layers are a set of annotations overlaid on top of the ATT&CK Matrix. -For more about ATT&CK Navigator layers, visit the ATT&CK Navigator repository. -The core module allows users to load, validate, manipulate, and save ATT&CK layers. -A brief overview of the components can be found below. -All scripts adhere to the MITRE ATT&CK Navigator Layer file format, -[version 4.3](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_3.md), -but will accept legacy [version 3.0](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv3.md) -and version 4.X layers, upgrading them to version 4.3. - -| script | description | -|:-------|:------------| -| [filter](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/filter.py) | Implements a basic [filter object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#filter-object-properties). | -| [gradient](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/gradient.py) | Implements a basic [gradient object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#gradient-object-properties). | -| [layer](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/layer.py) | Provides an interface for interacting with core module's layer representation. A further breakdown can be found in the corresponding [section](#Layer) below. | -| [layout](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/layout.py) | Implements a basic [layout object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#layout-object-properties). | -| [legenditem](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/legenditem.py) | Implements a basic [legenditem object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#legenditem-object-properties). | -| [metadata](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/metadata.py) | Implements a basic [metadata object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#metadata-object-properties). | -| [technique](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/technique.py) | Implements a basic [technique object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#technique-object-properties). | -| [versions](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/versions.py) | Implements a basic [versions object](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md#versions-object-properties).| - -### Manipulator Scripts - -| script | description | -|:-------|:------------| -| [layerops](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/manipulators/layerops.py) | Provides a means by which to combine multiple ATT&CK layer objects in customized ways. A further breakdown can be found in the corresponding [section](#layerops.py) below. | - -### Exporter Scripts - -| script | description | -|:-------|:------------| -| [to_excel](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_excel.py) | Provides a means by which to export an ATT&CK Layer to an excel file. A further breakdown can be found in the corresponding [section](#to_excel.py) below. | -| [to_svg](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/to_svg.py) | Provides a means by which to export an ATT&CK layer to an svg image file. A further breakdown can be found in the corresponding [section](#to_svg.py) below. This file also contains the `SVGConfig` object that can be used to configure the SVG export.| - -### Generator Scripts - -| script | description | -|:-------|:------------| -| [overview_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/overview_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes, on a per technique basis, all instances of a given ATT&CK object type that reference/utilize each technique. A further explanation can be found in the corresponding [section](#overview_generator.py) below. | -| [usage_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/usage_generator.py)| Provides a means by which to generate an ATT&CK Layer that summarizes the techniques associated with a given ATT&CK object. A further explanation can be found in the corresponding [section](#usage_generator.py) below. | -| [sum_generator](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/generators/sum_generator.py)| Provides a means by which to generate a collection of ATT&CK Layers, one for each object in a given ATT&CK object class, that summarizes the coverage of that object. A further explanation can be found in the corresponding [section](#sum_generator.py) below. | - -### Utility Modules - -| script | description | -|:-------|:------------| -| [excel_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/excel_templates.py) | Provides a means by which to convert a matrix into a clean excel matrix template. | -| [matrix_gen](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/matrix_gen.py) | Provides a means by which to generate a matrix from raw data, either from the ATT&CK TAXII server, from a local STIX Bundle, or from an ATT&CK Workbench instance (via url). | -| [svg_templates](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_templates.py) | Provides a means by which to convert a layer file into a marked up svg file. | -| [svg_objects](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/exporters/svg_objects.py) | Provides raw templates and supporting functionality for generating svg objects. | - -### Command Line Tools - -| script | description | -|:-------|:------------| -| [layerExporter_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerExporter_cli.py) | A commandline utility to export Layer files to excel or svg formats using the exporter tools. Run with `-h` for usage. | -| [layerGenerator_cli.py](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/layerGenerator_cli.py) | A commandline utility to generate Layer files that correspond to various and collections of various stix objects. Run with `-h` for usage. | - -## Layer - -The `Layer` class provides format validation and read/write capabilities to aid in working with ATT&CK Navigator Layers in python. -It is the primary interface through which other Layer-related classes defined in the core module should be used. -The Layer class API and a usage example are below. -The class currently supports version 3 and 4 of the ATT&CK Layer spec, and will upgrade version 3 layers into compatible version 4 ones whenever possible. - -| method [x = Layer()]| description | -|:-------|:------------| -| `x.from_str(_input_)` | Loads an ATT&CK layer from a string representation of a json layer. | -| `x.from_dict(_input_)` | Loads an ATT&CK layer from a dictionary. | -| `x.from_file(_filepath_)` | Loads an ATT&CK layer from a file location specified by the _filepath_. | -| `x.to_file(_filepath_)` | Saves the current state of the loaded ATT&CK layer to a json file denoted by the _filepath_. | -| `x.to_dict()` | Returns a representation of the current ATT&CK layer object as a dictionary. | -| `x.to_str()` | Returns a representation of the current ATT&CK layer object as a string representation of a dictionary. | - -Examples on how to create a layer programmatically, as opposed to loading it from an existing medium, can be found -[here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/core/README.md). - -### Example Usage - -```python -example_layer3_dict = { - "name": "example layer", - "version": "3.0", - "domain": "mitre-enterprise" -} - -example_layer4_dict = { - "name": "layer v4.3 example", - "versions" : { - "attack": "8", - "layer" : "4.3", - "navigator": "4.4.4" - }, - "domain": "enterprise-attack" -} - -example_layer_location = "/path/to/layer/file.json" -example_layer_out_location = "/path/to/new/layer/file.json" - -from mitreattack.navlayers.core import Layer - -layer1 = Layer(example_layer3_dict) # Create a new layer and load existing data -layer1.to_file(example_layer_out_location) # Write out the loaded layer to the specified file - -layer2 = Layer() # Create a new layer object -layer2.from_dict(example_layer4_dict) # Load layer data into existing layer object -print(layer2.to_dict()) # Retrieve the loaded layer's data as a dictionary, and print it - -layer3 = Layer() # Create a new layer object -layer3.from_file(example_layer_location) # Load layer data from a file into existing layer object -``` - -### layerops.py - -`Layerops.py` provides the `LayerOps` class, which is a way to combine layer files in an automated way, using user defined lambda functions. -Each LayerOps instance, when created, ingests the provided lambda functions, and stores them for use. -An existing `LayerOps` class can be used to combine layer files according to the initialized lambda using the process method. -The breakdown of this two step process is documented in the table below, while examples of both the list and dictionary modes of operation can be found below. - -#### LayerOps() - -```python -x = LayerOps(score=score, comment=comment, enabled=enabled, colors=colors, metadata=metadata, name=name, desc=desc, default_values=default_values) -``` - -Each of the _inputs_ takes a lambda function that will be used to combine technique object fields matching the parameter. -The one exception to this is _default_values_, which is an optional dictionary argument containing default values -to provide the lambda functions if techniques of the combined layers are missing them. - -##### .process() Method - -```python -x.process(data, default_values=default_values) -``` - -The process method applies the lambda functions stored during initialization to the layer objects in _data_. -_data_ must be either a list or a dictionary of Layer objects, and is expected to match the format of the lambda equations provided during initialization. -`default_values` is an optional dictionary argument that overrides the currently stored default values with new ones for this specific processing operation. - -#### Example Usage - -```python -from mitreattack.navlayers.manipulators.layerops import LayerOps -from mitreattack.navlayers.core.layer import Layer - -demo = Layer() -demo.from_file("C:\Users\attack\Downloads\layer.json") -demo2 = Layer() -demo2.from_file("C:\Users\attack\Downloads\layer2.json") -demo3 = Layer() -demo3.from_file("C:\Users\attack\Downloads\layer3.json") - -# Example 1) Build a LayerOps object that takes a list and averages scores across the layers -lo = LayerOps(score=lambda x: sum(x) / len(x), - name=lambda x: x[1], - desc=lambda x: "This is an list example") # Build LayerOps object -out_layer = lo.process([demo, demo2]) # Trigger processing on a list of demo and demo2 layers -out_layer.to_file("C:\demo_layer1.json") # Save averaged layer to file -out_layer2 = lo.process([demo, demo2, demo3]) # Trigger processing on a list of demo, demo2, demo3 -visual_aid = out_layer2.to_dict() # Retrieve dictionary representation of processed layer - -# Example 2) Build a LayerOps object that takes a dictionary and averages scores across the layers -lo2 = LayerOps(score=lambda x: sum([x[y] for y in x]) / len([x[y] for y in x]), - colors=lambda x: x['b'], - desc=lambda x: "This is a dict example") # Build LayerOps object, with lambda -out_layer3 = lo2.process({'a': demo, 'b': demo2}) # Trigger processing on a dictionary of demo and demo2 -dict_layer = out_layer3.to_dict() # Retrieve dictionary representation of processed layer -print(dict_layer) # Display retrieved dictionary -out_layer4 = lo2.process({'a': demo, 'b': demo2, 'c': demo3})# Trigger processing on a dictionary of demo, demo2, demo3 -out_layer4.to_file("C:\demo_layer4.json") # Save averaged layer to file - -# Example 3) Build a LayerOps object that takes a single element dictionary and inverts the score -lo3 = LayerOps(score=lambda x: 100 - x['a'], - desc= lambda x: "This is a simple example") # Build LayerOps object to invert score (0-100 scale) -out_layer5 = lo3.process({'a': demo}) # Trigger processing on dictionary of demo -print(out_layer5.to_dict()) # Display processed layer in dictionary form -out_layer5.to_file("C:\demo_layer5.json") # Save inverted score layer to file - -# Example 4) Build a LayerOps object that combines the comments from elements in the list, with custom defaults -lo4 = LayerOps(score=lambda x: '; '.join(x), - default_values= { - "comment": "This was an example of new default values" - }, - desc= lambda x: "This is a defaults example") # Build LayerOps object to combine descriptions, defaults -out_layer6 = lo4.process([demo2, demo3]) # Trigger processing on a list of demo2 and demo0 -out_layer6.to_file("C:\demo_layer6.json") # Save combined comment layer to file -``` - -## to_excel.py - -`to_excel.py` provides the `ToExcel` class, which is a way to export an existing layer file as an Excel spreadsheet. -The `ToExcel` class has an optional parameter for the initialization function, that tells the exporter what data source to use when building the output matrix. -Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or retrieving data from an ATT&CK Workbench instance. - -### ToExcel() - -```python -x = ToExcel(domain='enterprise', source='taxii', resource=None) -``` - -The `ToExcel` constructor takes domain, server, and resource arguments during instantiation. -The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. -The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the official ATT&CK Taxii Server (`cti-taxii`) when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that -it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path -to a local stix bundle, or if the source is set to `remote`, in which case it should be the url of a ATT&CK workbench instance. - -### .to_xlsx() Method - -```python -x.to_xlsx(layerInit=layer, filepath="layer.xlsx") -``` - -The `to_xlsx` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. - -#### Example Usage - -```python -from mitreattack.navlayers import Layer -from mitreattack.navlayers import ToExcel - -lay = Layer() -lay.from_file("path/to/layer/file.json") -# Using taxii server for template -t = ToExcel(domain=lay.layer.domain, source='taxii') -t.to_xlsx(layerInit=lay, filepath="demo.xlsx") -# Using local stix data for template -t2 = ToExcel(domain='mobile', source='local', resource='path/to/local/stix.json') -t2.to_xlsx(layerInit=lay, filepath="demo2.xlsx") -# Using remote ATT&CK Workbench instance for template -workbench_url = 'localhost:3000' -t3 = ToExcel(domain='ics', source='remote', resource=workbench_url) -``` - -## to_svg.py - -`to_svg.py` provides the `ToSvg` class, which is a way to export an existing layer file as an SVG image file. -The `ToSvg` class, like the `ToExcel` class, has an optional parameter for the initialization function, -that tells the exporter what data source to use when building the output matrix. -Valid options include using live data from cti-taxii.mitre.org, using a local STIX bundle, or utilizing a remote ATT&CK Workbench instance. - -### ToSvg() - -```python -x = ToSvg(domain='enterprise', source='taxii', resource=None, config=None) -``` - -The `ToSvg` constructor, just like the `ToExcel` constructor, takes domain, server, and resource arguments during instantiation. -The domain can be either `enterprise` or `mobile`, and can be pulled directly from a layer file as `layer.domain`. -The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. -The `config` parameter is an optional `SVGConfig` object that can be used to configure the export as desired. -If not provided, the configuration for the export will be set to default values. - -### SVGConfig() - -```python -y = SVGConfig(width=8.5, height=11, headerHeight=1, unit="in", showSubtechniques="expanded", - font="sans-serif", tableBorderColor="#6B7279", showHeader=True, legendDocked=True, - legendX=0, legendY=0, legendWidth=2, legendHeight=1, showLegend=True, showFilters=True, - showAbout=True, showDomain=True, border=0.104) -``` - -The `SVGConfig` object is used to configure how an SVG export behaves. -The defaults for each of the available values can be found in the declaration above, and a brief explanation for each field is included in the table below. -The config object should be provided to the `ToSvg` object during instantiation, but if values need to be updated on the fly, -the currently loaded configuration can be interacted with at `ToSvg().config`. -The configuration can also be populated from a json file using the `.load_from_file(filename="path/to/file.json")` method, -or stored to one using the `.save_to_file(filename="path/to/file.json)` method. - -| attribute| description | type | default value | -|:-------|:------------|:------------|:------------| -| width | Desired SVG width | number | 8.5 | -| height | Desired SVG height | number | 11 | -| headerHeight | Desired Header Block height | number | 1 | -| unit | SVG measurement units (qualifies width, height, etc.) - "in", "cm", "px", "em", or "pt"| string | "in" | -| showSubtechniques | Display form for subtechniques - "all", "expanded" (decided by layer), or "none" | string | "expanded" | -| font | What font style to use - "serif", "sans-serif", or "monospace" | string | "sans-serif" | -| tableBorderColor | Hex color to use for the technique borders | string | "#6B7279" | -| showHeader | Whether or not to show Header Blocks | bool | True | -| legendDocked | Whether or not the legend should be docked | bool | True | -| legendX | Where to place the legend on the x axis if not docked | number | 0 | -| legendY | Where to place the legend on the y axis if not docked | number | 1 | -| legendWidth | Width of the legend if not docked | number | 2 | -| legendHeight | Height of the legend if not docked | number | 1 | -| showLegend | Whether or not to show the legend | bool | True | -| showFilters | Whether or not to show the Filter Header Block | bool | True | -| showDomain | Whether or not to show the Domain and Version Header Block | bool | True | -| showAbout | Whether or not to show the About Header Block | bool | True | -| border | What default border width to use | number | 0.104 | - -### .to_svg() Method - -```python -x.to_svg(layerInit=layer, filepath="layer.svg") -``` - -The `to_svg` method exports the layer file referenced as `layer`, as an excel file to the `filepath` specified. - -#### Example Usage - -```python -from mitreattack.navlayers import Layer -from mitreattack.navlayers import ToSvg, SVGConfig - -lay = Layer() -lay.from_file("path/to/layer/file.json") -# Using taxii server for template -t = ToSvg(domain=lay.layer.domain, source='taxii') -t.to_svg(layerInit=lay, filepath="demo.svg") -#Using local stix data for template - -conf = SVGConfig() -conf.load_from_file(filename="path/to/poster/config.json") - -t2 = ToSvg(domain='mobile', source='local', resource='path/to/local/stix.json', config=conf) -t2.to_svg(layerInit=lay, filepath="demo2.svg") - -workbench_url = "localhost:3000" -t3 = ToSvg(domain='enterprise', source='remote', resource=workbench_url, config=conf) -t3.to_svg(layerInit=lay, filepath="demo3.svg") -``` - -## overview_generator.py - -`overview_generator.py` provides the `OverviewLayerGenerator` class, which is designed to allow users to -generate an ATT&CK layer that, on a per technique basis, has a score that corresponds to all instances -of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. - -### OverviewLayerGenerator() - -```python -x = OverviewLayerGenerator(source='taxii', domain='enterprise', resource=None) -``` - -The initialization function for `OverviewLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where -to retrieve data from (taxii server etc.). -The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. -The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. -If not provided, the configuration for the generator will be set to default values. - -### .generate_layer() - -```python -x.generate_layer(obj_type=object_type_name) -``` - -The `generate_layer` function generates a layer, customized to the input `object_type_name`. -Valid values include `group`, `mitigation`, `software`, and `datasource`. - -## usage_generator.py - -`usage_ generator.py` provides the `UsageLayerGenerator` class, which is designed to allow users to -generate an ATT&CK layer that scores any relevant techniques that a given input ATT&CK object has. -These objects can be any `group`, `software`, `mitigation`, or `data component`, -and can be referenced by ID or by any alias when provided to the generator. - -### UsageLayerGenerator() - -```python -x = UsageLayerGenerator(source='taxii', domain='enterprise', resource=None) -``` - -The initialization function for `UsageLayerGenerator`, like `ToSVG` and `ToExcel`, requires the specification of where -to retrieve data from (taxii server etc.). -The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. -The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. -If not provided, the configuration for the generator will be set to default values. - -### .generate_layer() - -```python -x.generate_layer(match=object_identifier) -``` - -The `generate_layer` function generates a layer, customized to the input `object_identifier`. -Valid values include `ATT&CK ID`, `name`, or any known `alias` for `group`, `mitigation`, `software`, and `data component` objects within the selected ATT&CK data. - -#### Example Usage - -```python -from mitreattack.navlayers import UsageLayerGenerator - -handle = UsageLayerGenerator(source='taxii', domain='enterprise') - -layer1 = handle.generate_layer(match='G0018') -layer2 = handle.generate_layer(match='Adups') -``` - -## sum_generator.py - -`sum_generator.py` provides the `SumLayerGenerator` class, which is designed to allow users to -generate a collection of ATT&CK layers that, on a per technique basis, have a score that corresponds to all instances -of the specified ATT&CK object type (group, mitigation, etc.), and a comment that lists all matching instance. -Each one of the generated layers will correspond to a single instance of the specified ATT&CK object type. - -### SumLayerGenerator() - -```python -x = SumLayerGenerator(source='taxii', domain='enterprise', resource=None) -``` - -The initialization function for `SumGeneratorLayer`, like `ToSVG` and `ToExcel`, requires the specification of where -to retrieve data from (taxii server etc.). -The domain can be either `enterprise`, `mobile`, or `ics`, and can be pulled directly from a layer file as `layer.domain`. -The source argument tells the matrix generation tool which data source to use when building the matrix. -`taxii` indicates that the tool should utilize the `cti-taxii` server when building the matrix, -while the `local` option indicates that it should use a local bundle, and the `remote` option indicates that it should utilize a remote ATT&CK Workbench instance. -The `resource` argument is only required if the source is set to `local`, in which case it should be a path to a local stix bundle, -or if the source is set to `remote`, in which case it should be the url of an ATT&CK Workbench instance. -If not provided, the configuration for the generator will be set to default values. - -### .generate_layer() - -```python -x.generate_layer(layers_type=object_type_name) -``` - -The `generate_layer` function generates a collection of layers, each customized to one instance of the input `object_type_name`. -Valid types include `group`, `mitigation`, `software`, and `datasource`. - -## layerExporter_cli.py - -This command line tool allows users to convert a [navigator](https://github.com/mitre-attack/attack-navigator) -layer file to either an svg image or excel file using the functionality provided by the navlayers module. -Details about the SVG configuration json mentioned below can be found in the -[SVGConfig](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md#svgconfig) -entry within the navlayers module documentation. - -```commandline -C:\Users\attack>layerExporter_cli -h -usage: layerExporter_cli [-h] -m {svg,excel} [-s {taxii,local,remote}] - [--resource RESOURCE] -o OUTPUT [OUTPUT ...] - [-l LOAD_SETTINGS] [-d WIDTH HEIGHT] - input [input ...] - -Export an ATT&CK Navigator layer as a svg image or excel file - -positional arguments: - input Path(s) to the file to export - -optional arguments: - -h, --help show this help message and exit - -m {svg,excel}, --mode {svg,excel} - The form to export the layers in - -s {taxii,local,remote}, --source {taxii,local,remote} - What source to utilize when building the matrix - --resource RESOURCE Path to the local resource if --source=local, or url - of an ATT&CK Workbench instance if --source=remote - -o OUTPUT [OUTPUT ...], --output OUTPUT [OUTPUT ...] - Path(s) to the exported svg/xlsx file - -l LOAD_SETTINGS, --load_settings LOAD_SETTINGS - [SVG Only] Path to a SVG configuration json to use - when rendering - -d WIDTH HEIGHT, --size WIDTH HEIGHT - [SVG Only] X and Y size values (in inches) for SVG - export (use -l for other settings) - -C:\Users\attack>layerExporter_cli -m svg -s taxii -l settings/config.json -o output/svg1.json output/svg2.json files/layer1.json files/layer2.json -``` - -## layerGenerator_cli.py - -This command line tool allows users to generate [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) -layer files from either a specific group, software, or mitigation. Alternatively, users can generate a layer file with a -mapping to all associated groups, software, or mitigations across the techniques within ATT&CK. - -```commandline -C:\Users\attack>layerGenerator_cli -h -usage: layerGenerator_cli [-h] - (--overview-type {group,software,mitigation,datasource} | --mapped-to MAPPED_TO | --batch-type {group,software,mitigation,datasource}) - [-o OUTPUT] [--domain {enterprise,mobile,ics}] - [--source {taxii,local,remote}] - [--resource RESOURCE] - -Generate an ATT&CK Navigator layer - -optional arguments: - -h, --help show this help message and exit - --overview-type {group,software,mitigation,datasource} - Output a layer file where the target type is - summarized across the entire dataset. - --mapped-to MAPPED_TO - Output layer file with techniques mapped to the given - group, software, mitigation, or data component. Argument - can be name, associated group/software, or ATT&CK ID. - --batch-type {group,software,mitigation,datasource} - Output a collection of layer files to the specified - folder, each one representing a different instance of - the target type. - -o OUTPUT, --output OUTPUT - Path to the output layer file/directory - --domain {enterprise,mobile,ics} - Which domain to build off of - --source {taxii,local,remote} - What source to utilize when building the layer files - --resource RESOURCE Path to the local resource if --source=local, or url - of an ATT&CK Workbench instance if --source=remote - -C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --mapped-to S0065 --output generated_layer.json -C:\Users\attack>layerGenerator_cli --domain mobile --source taxii --overview-type mitigation --output generated_layer2.json -C:\Users\attack>layerGenerator_cli --domain ics --source taxii --batch-type software -C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --overview-type datasource --output generated_layer3.json -``` +If you wish to read more about the Navlayers module, please click [here](https://mitreattack-python.readthedocs.io/en/126-docs-add-section-to-docs-for-accessing-stix/navlayers.html) for more information. \ No newline at end of file diff --git a/mitreattack/navlayers/core/README.md b/mitreattack/navlayers/core/README.md index 98e101b2..a73068c3 100644 --- a/mitreattack/navlayers/core/README.md +++ b/mitreattack/navlayers/core/README.md @@ -1,242 +1,3 @@ # Layers Core -This subcomponent, as part of the larger navlayers module, is responsible for Layer objects. Please note, this -documentation assumes familiarity with the [ATT&CK Navigator layer format](https://github.com/mitre-attack/attack-navigator/blob/develop/layers/LAYERFORMATv4_1.md). -The main handle for this implementation is the Layer, which stores an individual instance of a LayerObj object, -which further references the various sub-objects that make up a complete Layer. A visual representation of this -object breakdown can be seen here (please note there are other fields, these are just the objects): -``` -demo (Layer instance) <------------------------------------------------> The container for a layer object - |---> demo.layer (_LayerObj instance)--------------------------------> The raw layer object itself - |---> demo.layer.version (Versions instance)-----------------> A versions object - |---> demo.layer.filters (Filter instance)-------------------> A filter object - |---> demo.layer.layout (Layout instance)--------------------> A layout object - |---> demo.layer.techniques (List of Technique instances)----> A collection of technique objects - |---> demo.layer.gradient (Gradient instance)----------------> A gradient object - |---> demo.layer.legendItems (List of LegendItem instances)--> A collection of legend item objects - |---> demo.layer.metadata (List of Metadata instances)-------> A collection of metadata objects -``` -## Creating Layers Programmatically -With knowledge of the objects involved, as well as the additional fields (which have a 1:1 mapping with the -default ATT&CK Navigator spec), it is possible to programmatically generate a layer. Below is an example of -how this might be accomplished, piece by piece. - -```python -import mitreattack.navlayers as navlayers - -layer_example = navlayers.Layer() -layer_example.from_dict(dict(name="example", domain="enterprise-attack")) # arguments required for every layer - -# configure the versions object -layer_example.layer.versions = dict(layer="4.2", attack="9.1", navigator="4.2") - -# set a description -layer_example.layer.description = "This is a demonstration of how to set up a layer piece by piece" - -# configure the "filters" object -layer_example.layer.filters = dict(platforms=['macOS']) # platforms can be provided during initialization -layer_example.layer.filters.platforms = ['Windows'] # or separately - -# configure the 'sorting' setting -layer_example.layer.sorting = 3 # 0: sort ascending alphabetically by technique name -# 1: sort descending alphabetically by technique name -# 2: sort ascending by technique score -# 3: sort descending by technique score - -# configure the layout object -layer_example.layer.layout = dict(layout="side", - showID=True, - showName=True, - showAggregateScores=True, - countUnscored=True, - aggregateFunction="sum") # average, sum, max, min - -# configure whether or not to hide disabled techniques -layer_example.layer.hideDisabled = True -# configure the gradient object -layer_example.layer.gradient = dict(minValue=-100, maxValue=100, - colors=["#DAF7A6", "#FFC300", "#FF5733", "#C70039", "#900C3F", "#581845"]) -# configure collection of legend items -layer_example.layer.legendItems = [dict(label='A', color='#DAF7A6'), dict(label='B', color='#581845')] -# configure collection of metatdata values -layer_example.layer.metadata = [dict(name='example metadata', value='This is an example')] -# create listing of techniques in this layer -layer_example.layer.techniques = [dict(techniqueID='T1000', tactic='privilege-escalation', score=15, color='#AABBCC'), - dict(techniqueID='T1000.1', tactic='privilege-escalation', score=1, comment='Demo')] -``` -This first example utilizes the native dictionary form for initializing the layer. This approach is similar to the -method used by the automated import process, but may not be the most intuitive for users. An alternative method, -displayed below, is to create and modify instances of the core objects in the library. Please note, these two examples -produce equivalent internal layers once completed. - -```python -import mitreattack.navlayers as navlayers - -layer_example = navlayers.Layer(name="example", domain="enterprise-attack") # arguments required for every layer -layer_build = layer_example.layer # short handle to make the rest of this example easier to read - -# configure the versions object -versions_obj = navlayers.Versions() -versions_obj.layer = "4.2" -versions_obj.attack = "9.1" -versions_obj.navigator = "4.2" -layer_build.versions = versions_obj - -# set a description -layer_build.description = "This is a demonstration of how to set up a layer piece by piece" - -# configure the "filters" object -filter_obj = navlayers.core.Filter(domain="enterprise-attack") -filter_obj.platforms = ['Windows'] -layer_build.filters = filter_obj - -# configure the 'sorting' setting -layer_build.sorting = 3 # 0: sort ascending alphabetically by technique name -# 1: sort descending alphabetically by technique name -# 2: sort ascending by technique score -# 3: sort descending by technique score - -# configure the layout object -layout_obj = navlayers.core.Layout() -layout_obj.layout = "side" -layout_obj.showID = True -layout_obj.showName = True -layout_obj.showAggregateScores = True -layout_obj.countUnscored = True -layout_obj.aggregateFunction = "sum" # average, sum, max, min -layer_build.layout = layout_obj - -# configure whether or not to hide disabled techniques -layer_build.hideDisabled = True - -# configure the gradient object -gradient_obj = navlayers.core.Gradient(colors=["#DAF7A6", "#FFC300", "#FF5733", "#C70039", "#900C3F", "#581845"], - minValue=-100, maxValue=100) -layer_build.gradient = gradient_obj - -# configure collection of legend items -legend_item_obj_a = navlayers.core.LegendItem(label='A', color='#DAF7A6') -legend_item_obj_b = navlayers.core.LegendItem(label='B', color='#581845') -list_of_legend_items = [legend_item_obj_a, legend_item_obj_b] -layer_build.legendItems = list_of_legend_items - -# configure collection of metatdata values -metadata_object = navlayers.core.Metadata(name='example metadata', value='This is an example') -layer_build.metadata = [metadata_object] - -# create listing of techniques in this layer -technique_obj_a = navlayers.core.Technique(tID='T1000') -technique_obj_a.tactic = 'privilege-escalation' -technique_obj_a.score = 15 -technique_obj_a.color = '#AABBCC' -technique_obj_b = navlayers.core.Technique(tID='T1000.1') -technique_obj_b.tactic = 'privilege-escalation' -technique_obj_b.score = 1 -technique_obj_b.comment = "Demo" -layer_build.techniques = [technique_obj_a, technique_obj_b] - -``` - -### Object Documentation -Should it be helpful, the following section provides a breakdown of the available fields and methods for -each of the objects in the Core. This only includes 'public' methods and fields; there may be others used -for processing and other functionality that are not documented here, though documentation does exist for these -in the source code for them. - -#### Layer Object -```python - Layer().layer # Stores the raw LayerObj file - Layer().strict # Determines whether or not to be strict about loading files - Layer().from_str() # Initializes data from a string - Layer().from_dict() # Initializes data from a dictionary - Layer().from_file() # Initializes data from a file - Layer().to_file() # Exports the layer data to a file - Layer().to_dict() # Exports the layer data to a dictionary - Layer().to_str() # Exports the layer data to a string - ``` -#### LayerObj Object -```python - _LayerObj().versions # Link to a Versions object instance - _LayerObj().name # The Name for the Layer - _LayerObj().description # A description string for the Layer - _LayerObj().domain # The domain for the Layer - _LayerObj().filters # Link to a Filter object instance - _LayerObj().sorting # An integer denoting which sorting form to use - _LayerObj().layout # Link to a Layout object instance - _LayerObj().hideDisabled # Bool determining whether or not to show disabled techniques - _LayerObj().techniques # List of links to Technique objects - _LayerObj().gradient # Link to Gradient object - _LayerObj().legendItems # List of links to LegendItems objects - _LayerObj().showTacticRowBackground # Bool determining whether or not to show a background for tactics - _LayerObj().tacticRowBackground # Color code for tactic background - _LayerObj().selectTechniquesAcrossTactics # Bool determining whether or not to select cross-tactic - _LayerObj().selectSubtechniquesWithParent # Bool determining whether or not to select subtechniques - _LayerObj().metadata # List of links to Metadata items - _LayerObj().get_dict() # Export Layer as a dictionary object -``` -#### Versions Object -```python - Versions().layer # String denoting Layer format version - Versions().__attack # String denoting ATT&CK version - Versions().navigator # String denoting Navigator version - Versions().get_dict() # Export Version data as a dictionary object -``` -#### Filter Object -```python - Filter().domain # String denoting the domain for the Filter - Filter().platforms # String denoting platforms within this filter - Filter().get_dict() # Export Filter data as a dictionary object -``` -Please note that although not documented here, there is another Filter object variant, Filterv3, which exists -for backwards compatibility reasons. -#### Layout Object -```python - Layout().layout # String denoting which layout form to use - Layout().showID # Bool denoting whether or not to show technique IDs - Layout().showName # Bool denoting whether or not to show technique names - Layout().showAggregateScores # Bool denoting whether or not to utilize Aggregate scores - Layout().countUnscored # Bool denoting whether ot not to count unscored techniques as 0s for Aggregates - Layout().aggregateFunction # A enum integer denoting which aggregate function to utilize - # 1 - Average, 2 - min, 3 - max, 4 - sum - Layout().get_dict() # Export Layout data as a dictionary object - Layout().compute_aggregate() # Compute the aggregate score for a technique and it's subtechniques -``` -#### Technique Object -```python - Technique().techniqueID # String denoting the technique's ID - Technique().tactic # String denoting the technique's tactic - Technique().comment # String denoting any comments - Technique().enabled # Bool denoting if the technique is enabled - Technique().score # Integer denoting technique score - Technique().aggregateScore # Integer denoting pre-configured aggregate score - Technique().color # String denoting manually configured color code - Technique().metadata # List of links to metadata objects - Technique().showSubtechniques # Bool denoting whether or not to show subtechniques - Technique().get_dict() # Export Technique data as a dictionary object -``` -#### Gradient Object -```python - Gradient().colors # Array of colors (string codes) over which the gradient is to be calculated - Gradient().minValue # Integer denoting minimum viable value on the gradient - Gradient().maxValue # Integer denoting maximum viable value on the gradient - Gradient().compute_color() # Calculate the appropriate color for a given score on the gradient - Gradient().get_dict() # Export Gradient data as a dictionary object -``` -#### LegendItem Object -```python - LegendItem().label # String denoting the label for this Legend Item' item - LegendItem().color # String denoting the color code for the Legend Item - LegendItem().get_dict() # Export Legend Item data as a dictionary object -``` -#### Metadata/Metadiv Object -```python - Metadata().name # String denoting metadata keypair name - Metadata().value # String denoting metadata keypair value - Metadata().get_dict() # Export metadata data as a dictionary object -``` -```python - Metadiv().name # Always set to "DIVIDER" - Metadiv().value # Bool denoting active or not - Metadiv().get_dict() # Export metadiv as a dictionary object -``` -A `Metadiv` object is simply a modified version of a `Metadata` object used as a visual divider. \ No newline at end of file +If you wish to read more about the Nav Layers Core submodule, please click [here](https://mitreattack-python.readthedocs.io/en/126-docs-add-section-to-docs-for-accessing-stix/navlayercore.html) for more information. \ No newline at end of file From 8436e84cda4dc03d20d1dfc105a84b88b7ec18ec Mon Sep 17 00:00:00 2001 From: Sun Date: Mon, 18 Sep 2023 11:52:25 -0400 Subject: [PATCH 159/159] fixing link --- docs/stix_primer/attack_id.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stix_primer/attack_id.rst b/docs/stix_primer/attack_id.rst index cd70cc5c..eea08d73 100644 --- a/docs/stix_primer/attack_id.rst +++ b/docs/stix_primer/attack_id.rst @@ -21,4 +21,4 @@ Note: in prior versions of ATT&CK, mitigations had 1:1 relationships with techni Filter("type", "=", "attack-pattern") ])[0] -The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see [Removing revoked and deprecated objects](#Removing-revoked-and-deprecated-objects). +The old 1:1 mitigations causing this issue are deprecated, so you can also filter them out that way — see `Removing revoked and deprecated objects `_ . \ No newline at end of file