Question: What is the current state of data lineage?

[jsnowacki]

I went through the current information about data lineage and it is hard to grasp the current state of the feature. Roadmap vaguely indicates that the feature is ongoing. Some issues like #163 are closed, some like #69 are still open, and throughout documentation some mentions of lineage are done, mostly with Apache Atlas, but not clear information is given.

My current understanding is that the development, especially one based on native graph database approach is still ongoing.

My main questions are:

1. When the lineage will be usable/ready?
2. Can we somehow setup and test it out currently?
3. Which metadata backend/proxy we need to use to have the lineage working? (it is not clear if the current gremlin-based backends are general or there are some technology specific, like Neo4j, features only)
4. Do Atlas lineage is or is not useable/visible in the frontend?

[verdan]

Hi @jsnowacki , sorry for all the confusion. We usually update the community via Slack, and through the community meetings. The best way to get the response quickly is indeed Slack 🙂
To answer your questions:

When the lineage will be usable/ready?

The list-based view of lineage is already available in the Amundsen and is being used by a few companies.
This is where you need to enable this in the frontend: frontend config
In terms of backend, you can inject your own 3rd party lineage tool via Metadata or you can use databuilder to ingest your own lineage in Neo4j. More details about the implementation in this commit

Can we somehow setup and test it out currently?

Again, the list-based lineage you can already use for production systems. The graph-based is currently in progress, but please feel free to checkout this branch verdan/lineage-graph-v1.

Which metadata backend/proxy we need to use to have the lineage working? (it is not clear if the current gremlin-based backends are general or there are some technology specific, like Neo4j, features only)

For now, Amundsen supports native lineage for Neo4j, but Atlas is a WIP (this branch: feat/atlas_lineage)

Do Atlas lineage is or is not useable/visible in the frontend?

This is a Work In Progress. branch: feat/atlas_lineage. @mgorsk1 should be able to respond more accurately about the current state of this implementation.

[mgorsk1]

@jsnowacki as for Atlas lineage the implementation is already finished and available on feat/atlas_lineage branch if you want to take a look. I have one outstanding PR (atlas dashboard support) to merge and this will be the next in line.

[jsnowacki]

@verdan and @mgorsk1 thanks a lot for the answers! I'll look into it and come back to you if there are any more questions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[verdan]

@jsnowacki please feel free to reopen if you have any further questions here.

[tooptoop4]

@verdan where is lineage-graph-v1 branch?

[cheji02]

@mgorsk1, has feat/atlas_lineage branch been merged to the main branch? also is there a good instruction for customer build/deploy (with configuration change)? Thanks!