Home

Required Elasticsearch mapping for indexes

Elasticsearch needs a template so it can import the converted JSON data from the scripts the right way. The easiest way to do this is by applying the template to all the newly created indexes in the future. Do this by making the following HTTP PUT request (TIP: Kibana -> Dev tools -> Console):

PUT _template/template_osint
{
    "template" : "*",
    "settings" : {
        "number_of_shards" : 1,
        "index.mapping.total_fields.limit": 7000
    },
    "mappings" : {
           "_default_": {
        "properties": {
          "ip": {
            "type": "ip"
          },
          "ip_int": {
            "type": "long"
          },
          "ipinfo.location": {
            "properties": {
              "geo": {
                "type": "geo_point"
              }
            }
          },
          "shodan.location": {
            "properties": {
              "geo": {
                "type": "geo_point"
              }
            }
          },
          "ipinfo.whois.person.last_modified": {
           "type": "text"
          },
          "ipinfo.whois.last_modified": {
           "type": "text"
          },
          "shodan.opts.bitcoin.handshake.nonce": {
           "type": "text"
          }
        },
        "dynamic_templates": [
        {
          "censys_timestamps_as_date": {
            "match_mapping_type": "string",
            "path_match":   "censys.*.timestamp",
            "mapping": {
              "type": "date",
              "format": "yyyy-MM-dd HH:mm:ss zzz"
            }
          }
        }
      ]
      }
    }
}

Mapping explanation

• The "ip" field will be used as ID. Every IP element will contain intel about an IP address from difference OSINT sources;
• "index.mapping.total_fields.limit" will increase the maximum amount of possible fields of the index, so elements will not be lost;
• “ipinfo.location.geo” will be a geo_point so you can create worldmaps with it in Kibana;
• The other mappings will make sure Elasticsearch saves specific datafields as the right datatype.