What are the machine requirements for terminusDB?

[ghost]

I'd like to load an N-triples or Turtle dump of wikidata which weighs about 120GB bzipped. The file hasn't been downloaded yet (in progress), but before loading in the DB, I'd like to know the machine requirements - especially memory.

Can terminusDB handle a 120GB without requiring the same or more memory?

[GavinMendelGleason]

At the moment loading 120GB at once is unlikely to work well unless you have around a terabyte of RAM. It may be possible to run it in batches as several commits, perhaps 10GB chunks within around 64GB memory machine.

We'd be interested in trying to load this data with you as a project.

[ghost]

Thank you for the response. Unfortunately, I'm just a private person unaffiliated with the wikipedia  project. The only machineni possess has 64 GB of RAM, so I was hoping because terminusdb is written in Rus, that it would be able to load wikidata within those limits.  You could probably contact the wikidata project to see if they'd be interested in using terminus dB over blazegraph. Blazegraph seems to require far less memory though despite being written in Java. https://github.com/blazegraph/database/wiki/Hardware_Configuration

[GavinMendelGleason]

You don't have to be affiliated, both are open source!

The ultimate memory footprint is likely to be much lower in TerminusDB, but currently our ingestion process will probably take a significant amount. However, as I said before, if it is phased in chunks, it's quite possible that you could do it in under 64GB.

[hpvd]

The ultimate memory footprint is likely to be much lower in TerminusDB, but currently our ingestion process will ...

would it be possible to share some details on this?

I know from other in memory databases, the most memory is used when periodically taking a snapshot for backup.
Is this the same for terminusDB or is data persistence/backup achieved using another method?

[GavinMendelGleason]

With the latest concat keyword which we are adding to TerminusDB you can ingest a data product which is near to the same size as memory resolving @LoveIsGrief 's issue @hpvd

This works by loading in parallel to multiple databases and then merging them using the concat into a single database. The operations are performed by streaming the indexes into a new index.

Currently this is in a feature branch, but we will be merging this week.

[GavinMendelGleason]

I think on average we require about half of what Blazegraph requires. I'm currently doing a benchmark which will show the comparison.

[hpvd]

@GavinMendelGleason many thanks for this fast and detailed and positive answer!
Looking forward for the benchmark!

[hpvd]

looks like the possibility of merging using concat was released with
TerminusDB Server v11.1.0 released at Jul 10, 2023

Add ability to merge base layers with the concat command

https://github.com/terminusdb/terminusdb/releases/tag/v11.1.0