Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream vertex edges off of disk -- (HDFS solution? -- ImmutableBytes solution?) #50

Open
okram opened this issue Nov 1, 2012 · 3 comments

Comments

@okram
Copy link
Contributor

okram commented Nov 1, 2012

solve the super node problem for a vertex (can't always assume we can hold in memory)

@BrynCooke
Copy link

So perhaps a memory mapped FaunusVertex implementation?

This could kick in once the number of edges reach a large size.

@okram
Copy link
Contributor Author

okram commented Jan 31, 2014

If you could get the FaunusEdge lists backed by a HDFS inputstream, that would be cool.

@BrynCooke
Copy link

I had a good look at this - but am getting cold feet.
The idea was to stash the FaunusEdges in MapDB and use PantomReferences to detect when the store needed to be updated. For this to work the FaunusEdge needs to be separated in to a wrapper and data objects and the edge lists replaced by a cache implementation.
However, the underlying Faunus code is complex and I am worried that I may find that this doesn't work after a lot of effort. Sorry, I'm not confident at taking this to its conclusion right now. Willing to discuss further though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants