Snowflake Data Marketplace?

[sfc-gh-phoran]

CMS is already a Snowflake customer, and for those insurers/providers that are also on Snowflake, using the Data Cloud as the platform for the exchange makes a lot more sense than a file-based transfer architecture.
https://www.snowflake.com/data-marketplace/

A) Snowflake supports XML/JSON/Parquet natively (as well as structured relational tables), so this could be built with a standard relational model or as semi-structured data formats, and each data provider could simple "share" their Snowflake databases/tables/views with CMS - and vice versa.
B) All participants would have real-time live access to data, rather than deal with the delays and staleness inherent in file-based transfers.
C) The option exists to host this as a Private (invite-only) Exchange, or a dataset on the public Snowflake data marketplace.

[BobSyracuse]

@sfc-gh-phoran Hello, you do realize that the mandate requires insurers/providers to provide these 3 file types are to be "made available to the public without restrictions that would impede the re-use of that information". So these files can not be hosted as Private (invite-only). Also the files are not transferred to CMS so CMS being on Snowflake does not provide any value.

[sfc-gh-phoran]

Snowflake Data Marketplace does not "impede the re-use" of information. It fosters re-use. It removes impediments to re-use.

[BobSyracuse]

@sfc-gh-phoran So Snowflake Data Marketplace is a commercial service that will allow insurers/providers to host their 3 file types to be publicly available without restriction, including registration, for the public consumptions of the data. In your documentation on the Marketplace? In the documentation https://docs.snowflake.com/en/user-guide/data-marketplace-intro.html#how-do-i-access-the-snowflake-data-marketplace-to-browse-listings it appears that you must be registered and signed in to use the Marketplace which does not conform to the ruling
_

To access the Snowflake Marketplace as a data consumer:

Log in to the Snowflake classic Web Interface.

Click Data Marketplace in the top menu. The Data Marketplace opens in a new tab. You can sign in using your Snowflake credentials.

_

[sfc-gh-phoran]

Snowflake customers can provide free access to any of their sharable datasets - it's called a "Reader Account". It still requires the consumer to login and authenticate to the Snowflake UI to execute queries, but there's no charge to the consumer. Any dataset available on the Marketplace would also available to Reader Accounts for free.

[jonberkery-work]

I have been researching "Snowflake\Snowflake-like" products. If we went this route, I would "front" the API with a web application. The file consumer would hit web application (requiring no log-in), the web application would access the API with a "reader account" login, retrieve the data, and start streaming it to the file consumer. This is how I would meet the requirements of the rule stating "no login required" for the file consumer.

[sfc-gh-phoran]

I think this is a great architecture. Snowflake is in Private Preview with our own REST-based API that could serve as the backend, rather than requiring a direct JDBC or nodeJS connection from the web front-end.

[garysolebc]

Completely agree that this is the file based solution is the wrong solution for this problem. But the Snowflake solution does not really help much more partly because a lot of the pricing can often be contextual and dynamically determined. A standard based API based solution would probably be the best option.

[ericsilvertx]

I believe the objective of the law is to provide the data with no need of authorization hence why the file based approach for a JSON file.

[sfc-gh-phoran]

API solutions are notoriously bad at large file extract use-cases (IMO). They're much more appropriate for smaller, more focused result sets.

[garysolebc]

The Provider Directory approach for Interoperability is an API without any authorization. So creating a unauthorized API would not be a problem.
The API would not be doing a large file extract - rather, given a Provider, Billing Code and any other information - currently as provided in the file, a collection of data could be provided. This would allow for a more dynamic access of data being looked for and more relevant. In some cases context is important, an API would allow requests to create context such as provider full price.
We are having major challenges figuring out how to create a static price in an environment where we cannot ask for some form of context.
Pretty much most of the files we are going to create will not fit on a standard laptop. Each file we are likely to produce will be in the high hundred Gb size to low Tb size. And we will create many of these and it will not be clear as to how or why these files exist in this way. As a consumer of the data you would need to try and figure out under which context to go after which file, wait for a huge file to download, then parse for the information you are seeking. In many cases a particular file will have the nearly the exact same data as another file produced by the same payer. Only context will help determine what you need.

[BobSyracuse]

@garysolebc I agree completely! The plus is for the Transparency ruling will be the Cost Estimator that could be used for the public and the Advanced EOB that would allow enough context for a plan member and provider to understand the cost ramifications.

[ericsilvertx]

Can you share how you were able estimate the size of a same data file? I have read ok another discussion that there was a spreadsheet model to help compute a size estimate.

…

On Fri, Jul 30, 2021, 3:34 PM Paul Horan ***@***.***> wrote: I think this is a great architecture. Snowflake is in Private Preview with our own REST-based API that could serve as the backend, rather than requiring a direct JDBC or nodeJS connection. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#153 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHEDVXXJS5P5IBAWL4X7QILT2MEENANCNFSM5A2LXYJA> .