datahub-template

This project shows how easy it is to integrate the datahub schema registry for use in spark and prophecy. It features two main functions:

1. browse datasets - Browse and select existing datasets listed in datahub, synchronizing schema and other metadata.
2. sync dataset - If you want to save dataset details to datahub, simply click "Sync To Datahub" to sync schema, descriptions and other metadata to datahub.

Requirements

1. A deployed instance of datahub, See here for a guide to deploy datahub.
2. A datahub bearer token to authenticate requests, See here for a guide to generate authentication tokens.

Getting started

1. Specify datahub url and token:

Edit either in prophecy or in github: project/gems/prophecysamples_datahubtemplate/gems/DatahubTable.py

Set these to appropriate values:

    DATAHUB_BASE_URL = "<INSERT_DATAHUB_URL>"
    DATAHUB_TOKEN = "<INSERT_DATAHUB_TOKEN>"

2. Create a DatahubTable source/target.

3. Browse existing tables saved to Datahub.

4. Sync changes when there's an update.