What is the best way to integrate Rollbar? (Global initialisation)

[mjmare]

I want to enable notifications to Rollbar.io whenever an exception occurs in my pipelines. The RB python integration (https://docs.rollbar.com/docs/python) needs a global initialisation. I'm unsure what the best location is for this initialisation?
My current implementation is in the pipeline hooks. Is that considered good practice?
At initialisation time I need access to the credentials in order to get the auth token for RB.

This is my implementation, which works so far.

class ProjectHooks:

    def init_rollbar(self):
        session = get_current_session()
        context = session.load_context()
        credentials = context.config_loader.get('*credentials.yml')
        rb_token = credentials['rollbar']['token']
        rb_environment = 'development' if context.env == 'local' else 'production'
        rollbar.init(rb_token,
                     environment=rb_environment,
                     locals={'enabled': True}
                     )


    @hook_impl
    def register_catalog(
            self,
            catalog: Optional[Dict[str, Dict[str, Any]]],
            credentials: Dict[str, Dict[str, Any]],
            load_versions: Dict[str, str],
            save_version: str,
            journal: Journal,
    ) -> DataCatalog:

        cat = DataCatalog.from_config(
            catalog, credentials, load_versions, save_version, journal
        )

        self.init_rollbar()

        return cat

    @hook_impl
    def on_pipeline_error(self, error, run_params, pipeline, catalog):
        rb_extra = {'pipeline': run_params["pipeline_name"]}
        rollbar.report_exc_info(sys.exc_info(), extra_data=rb_extra)

I tried to move the call roller_init to the init method of the Hooks class, but when that is called there is no session yet.

Is this the "Kedro way"? Suggestions?

[datajoely]

Hi @mjmare you're very close! You should be using hooks for this, but the right way is to create your own set of custom hooks. We have instructions on how to do this here and also I have an example project that may be helpful

The before_pipeline_run hook spec gives you pretty much everything you need here. You get access to the instantiated catalog, environment name and other things. I don't think we give you the credentials this way, but you can either read the YAML file yourself or handle that another way.

[mjmare]

@datajoely So you mean move the call to init_rollbar to the before_pipeline_run implementation (and maybe move it to its own Hook class), right?
What I hadn't realised is that even if you call a ton of pipelines (strung together in, say, the default pipeline) the before_pipeline_run is only called once. This single pipeline is the one name passed as an argument to Kedro run. Is that correct?

[datajoely]

So yes there is actually a bit of nuance when we use the word pipeline here. You can have many registered pipelines in your pipeline registry, but at runtime only one pipeline is run because we collapse all Pipeline objects into a single unified one (there is no requirements for the nodes to even be connected).

By default running kedro run is the same as kedro run --pipeline="__default__" any permutations you do to that with different run configurations still only create a single 'runtime pipeline'

[mjmare]

Yes, I did not get that nuance (of the single pipeline) from the docs. Only after running in the debugger did I came to that conclusion.