Merge pull request #47 from nautobot/release-v0.1.7

Release v0.1.7

.travis.yml

@@ -53,20 +53,20 @@ jobs:
         - "invoke flake8"
         - "invoke pylint"
 
-    # - stage: "deploy-github"
-    #   before_script:
-    #     - "pip install poetry"
-    #   script:
-    #     - "poetry version $TRAVIS_TAG"
-    #     - "poetry build"
-    #   deploy:
-    #     provider: "releases"
-    #     api_key: "$GITHUB_AUTH_TOKEN"
-    #     file_glob: true
-    #     file: "dist/*"
-    #     skip_cleanup: true
-    #     "on":
-    #       tags: true
+    - stage: "deploy-github"
+      before_script:
+        - "pip install poetry"
+      script:
+        - "poetry version $TRAVIS_TAG"
+        - "poetry build"
+      deploy:
+        provider: "releases"
+        api_key: "$GITHUB_AUTH_TOKEN"
+        file_glob: true
+        file: "dist/*"
+        skip_cleanup: true
+        "on":
+          tags: true
 
     - stage: "deploy-pypi"
       before_script:

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## v0.1.7 - 2021-07-27
+
+### Added
+
+- #42:
+    - Add stack trace to job log on exception
+    - IMAP and GMail notification sources now support a `source_header` configuration parameter to allow for cases where `From` is not the relevant header to inspect.
+
+
+### Fixed
+
+- #42:
+    - Avoid an exception if some Providers do not have a populated `emails_circuit_maintenance` value
+    - `extract_email_source()` now correctly handles email addresses containing dash characters.
+    - Avoid an exception on processing a non-multipart email payload
+    - Don't try to create a `RawNotification` if no `raw_payload` could be extracted from the notification.
+
 ## v0.1.6 - 2021-07-14
 
 ### Added

README.md

@@ -43,21 +43,19 @@ PLUGINS_CONFIG = {
 
 ## Usage
 
-All the plugin configuration is done via UI, under the **Plugins** tab, in the **Circuit Maintenance** sections.
-
 ### 1. Define source emails per Provider
 
-Each Circuit **Provider**, that we would like to track via the Circuit Maintenance plugin, requires at least one email address under the `Custom Fields` -> `Emails for Circuit Maintenance plugin` section.
+In the Nautobot UI, under **Circuits -> Providers**, for each Provider that we would like to track via the Circuit Maintenance plugin, we must configure at least one email source address (or a comma-separated list of addresses) in the **`Custom Fields -> Emails for Circuit Maintenance plugin** field.
 
-These are the source email addresses that the plugin will check and use to classify each notification for each specific provider.
+These are the source email addresses that the plugin will detect and will use to classify each notification for each specific provider.
 
 ### 2. Configure Notification Sources
 
 Notification Sources are defined in two steps:
 
-#### 2.1 Define Notification Sources in `configuration.py`
+#### 2.1 Define Notification Sources in `nautobot_config.py`
 
-In the `PLUGINS_CONFIG`, under the `nautobot_circuit_maintenance` key, we should define the Notification Sources that will be able later in the UI, where you will be able to **validate** if the authentication credentials provided are working fine or not.
+In `nautobot_config.py`, in the `PLUGINS_CONFIG`, under the `nautobot_circuit_maintenance` key, we should define the Notification Sources that will be able later in the UI, where you will be able to **validate** if the authentication credentials provided are working fine or not.
 
 There are two mandatory attributes (other keys are dependent on the integration type, and will be documented below):
 
@@ -66,58 +64,68 @@ There are two mandatory attributes (other keys are dependent on the integration
 
 > Currently, only IMAP and HTTPS (accounts.google.com) integrations are supported as URL scheme
 
-##### IMAP
+##### 2.1.1 IMAP
 
-There are 2 extra attributes:
+There are 2 extra required attributes:
 
 - `account`: Identifier (i.e. email address) to use to authenticate.
 - `secret`: Password to IMAP authentication.
 
 > Gmail example: [How to setup Gmail with App Passwords](https://support.google.com/accounts/answer/185833)
 
+There is also one optional attribute:
+
+- `source_header`: Specify a particular email header to use to identify the source of a particular notification and assign it to the appropriate provider. If unset, `From` will be used, but if your emails are not received directly from the provider but instead pass through a mailing list or alias, you might need to set this to a different value such as `X-Original-Sender` instead.
+
 ```py
 PLUGINS_CONFIG = {
     "nautobot_circuit_maintenance": {
         "notification_sources": [
             {
                 "name": "my custom name",
-                "account": os.environ.get("CM_NS_1_ACCOUNT", ""),
-                "secret": os.environ.get("CM_NS_1_SECRET", ""),
-                "url": os.environ.get("CM_NS_1_URL", ""),
+                "account": os.getenv("CM_NS_1_ACCOUNT", ""),
+                "secret": os.getenv("CM_NS_1_SECRET", ""),
+                "url": os.getenv("CM_NS_1_URL", ""),
+                "source_header": os.getenv("CM_NS_1_SOURCE_HEADER", "From"),  # optional
             }
         ]
     }
 }
 ```
 
-##### Gmail API integrations
+##### 2.1.2 Gmail API integrations
+
+There are 2 extra required attributes:
 
-There are 2 extra attributes:
+- `account`: Identifier (i.e. email address) to access via OAuth or to impersonate as service account.
+- `credentials_file`: JSON file containing all the necessary data to identify the API integration (see below).
 
-- `account`: Identifier (i.e. email address) to use to impersonate as service account.
-- `credentials_file`: JSON file containing all the necessary data to identify the API integration.
+There is also one optional attribute:
+
+- `source_header`: Specify a particular email header to use to identify the source of a particular notification and assign it to the appropriate provider. If unset, `From` will be used, but if your emails are not received directly from the provider but instead pass through a mailing list or alias, you might need to set this to a different value such as `X-Original-Sender` instead.
 
 ```py
 PLUGINS_CONFIG = {
     "nautobot_circuit_maintenance": {
         "notification_sources": [
             {
                 "name": "my custom name",
-                "account": os.environ.get("CM_NS_1_ACCOUNT", ""),
-                "credentials_file": os.environ.get("CM_NS_1_CREDENTIALS_FILE", ""),
-                "url": os.environ.get("CM_NS_1_URL", ""),
+                "account": os.getenv("CM_NS_1_ACCOUNT", ""),
+                "credentials_file": os.getenv("CM_NS_1_CREDENTIALS_FILE", ""),
+                "url": os.getenv("CM_NS_1_URL", ""),
+                "source_header": os.getenv("CM_NS_1_SOURCE_HEADER", "From"),  # optional
             }
         ]
     }
 }
 ```
 
-To enable Gmail API access, there are some common steps for either Service Account and OAuth authentication:
+To enable Gmail API access, there are some common steps for both Service Account and OAuth authentication:
 
 1. Create a **New Project** in [Google Cloud Console](https://console.cloud.google.com/).
 2. Under **APIs and Services**, enable **Gmail API** for the selected project.
 
-**Service Account**
+###### 2.1.2.1 Service Account
 
 To create a [Service Account](https://support.google.com/a/answer/7378726?hl=en) integration:
 
@@ -126,7 +134,7 @@ To create a [Service Account](https://support.google.com/a/answer/7378726?hl=en)
 5. With Super Admin rights, open the [Google Workspace admin console](https://admin.google.com). Navigate to **Security**, **API controls**, and select the **Manage Domain Wide Delegation** at the bottom of the page.
 6. Add a new API client and paste in the Client ID copied earlier. In the **OAuth scopes** field add the scopes `https://www.googleapis.com/auth/gmail.readonly` and `https://mail.google.com/`. Save the new client configuration by clicking _Authorize_.
 
-**OAuth**
+###### 2.1.2.2 OAuth
 
 To create a [OAuth 2.0](https://developers.google.com/identity/protocols/oauth2/web-server) integration:
 
@@ -136,12 +144,17 @@ To create a [OAuth 2.0](https://developers.google.com/identity/protocols/oauth2/
 
 > For OAuth integration, it's recommendable that, at least the first time, you run a manual **Validate** of the Notification Source to complete the OAuth authentication workflow, identifying your Google credentials.
 
+> Typically the `url` setting to configure in your `nautobot_config.py` for use with OAuth integration will be `"https://accounts.google.com/o/oauth2/auth"`.
+
+
 #### 2.2 Add `Providers` to the Notification Sources
 
 In the Circuit Maintenance plugin UI section, there is a **Notification Sources** button (yellow) where you can configure the Notification Sources to track new circuit maintenance notifications from specific providers.
 
 Because the Notification Sources are defined by the configuration, you can only view and edit `providers`, but not `add` or `delete` new Notification Sources via UI or API.
 
+> Note that for emails from a given Provider to be processed, you must *both* define a source email address(es) for that Provider (Usage section 1, above) *and* add that provider to a specific Notification Source as described in this section.
+
 ### 3. Run Handle Notifications Job
 
 There is an asynchronous task defined as a **Nautobot Job**, **Handle Circuit Maintenance Notifications** that will connect to the emails sources defined under the Notification Sources section (step above), and will fetch new notifications received since the last notification was fetched.

nautobot_circuit_maintenance/__init__.py

@@ -1,5 +1,5 @@
 """Init for Circuit Maintenance plugin."""
-__version__ = "0.1.6"
+__version__ = "0.1.7"
 from django.conf import settings
 from django.db.models.signals import post_migrate
 from django.utils.text import slugify

nautobot_circuit_maintenance/handle_notifications/handler.py

@@ -3,7 +3,6 @@
 import traceback
 from typing import Union
 from django.conf import settings
-from django.db import OperationalError
 from circuit_maintenance_parser import ParsingError, init_provider
 from nautobot.circuits.models import Circuit, Provider
 from nautobot.extras.jobs import Job
@@ -183,6 +182,7 @@ def process_raw_notification(logger: Job, notification: MaintenanceNotification)
         )
         return None
 
+    raw_payload = b""
     for raw_payload in notification.raw_payloads:
         parser = init_provider(raw=raw_payload, provider_type=notification.provider_type)
         if not parser:
@@ -192,18 +192,20 @@ def process_raw_notification(logger: Job, notification: MaintenanceNotification)
         try:
             parsed_notifications = parser.process()
             break
-        except ParsingError as exc:
-            tb_str = traceback.format_exception(etype=type(exc), value=exc, tb=exc.__traceback__)
-            logger.log_debug(message=f"Parsing failed for notification {notification.subject}:.\n{tb_str}")
-        except Exception as exc:
-            tb_str = traceback.format_exception(etype=type(exc), value=exc, tb=exc.__traceback__)
+        except ParsingError:
+            tb_str = traceback.format_exc()
+            logger.log_debug(message=f"Parsing failed for notification `{notification.subject}`:\n```\n{tb_str}\n```")
+        except Exception:
+            tb_str = traceback.format_exc()
             logger.log_debug(
-                message=f"Unexpected exception while parsing notification {notification.subject}.\n{tb_str}"
+                message=f"Unexpected exception while parsing notification `{notification.subject}`.\n```\n{tb_str}\n```"
             )
     else:
         parsed_notifications = []
-        raw_payload = b""
-        logger.log_warning(message=f"Parsed failed for all the raw payloads for '{notification.subject}'.")
+        logger.log_warning(message=f"Parsed failed for all the raw payloads for `{notification.subject}`.")
+        # Carry on with the last raw_payload in the list, if any
+        if not raw_payload:
+            return None
 
     if isinstance(raw_payload, str):
         raw_payload = raw_payload.encode("utf-8")
@@ -217,7 +219,7 @@ def process_raw_notification(logger: Job, notification: MaintenanceNotification)
             sender=notification.sender,
             source=NotificationSource.objects.filter(name=notification.source).last(),
         )
-    except OperationalError as exc:
+    except Exception as exc:
         logger.log_warning(message=f"Raw notification '{notification.subject}' not created because {str(exc)}")
         return None
 
@@ -282,7 +284,7 @@ def run(self, data=None, commit=None):
                 return raw_notification_ids
 
             for notification in notifications:
-                self.log_info(message=f"Processing notification {notification.subject}.")
+                self.log_info(message=f"Processing notification `{notification.subject}`.")
                 raw_id = process_raw_notification(self, notification)
                 if raw_id:
                     raw_notification_ids.append(raw_id)