Skip to content

RDF (.nq) to DHT : Strategy for converting / querying very large .dht files on rather low end hardware #171

Answered by D063520
davidandreoletti asked this question in Q&A
Discussion options

You must be logged in to vote

NOTE: HDT does not support quads, neither the java nor the c++ version. If this is a no go, then currently do not use HDT. See #3 and https://github.com/JulianRei/hdtq-java (this is not code to use in production currently).
Q1: currently not, even if we are working on a compression that is better. With 16Gb of RAM you can compress a 16Gb of nt file in one chunk (more or less)
Q2: no. Note that the jena implemention is a bit buggy and might not optimize the queries super well. You can check out this though https://github.com/the-qa-company/qEndpoint
Q3: no, the HDT is only mapped and you need 3% of the data size in memory as a minimum requirement. So for 100Gb of HDT file, 3GB of memory.
N…

Replies: 2 comments 4 replies

Comment options

You must be logged in to vote
1 reply
@davidandreoletti
Comment options

Answer selected by D063520
Comment options

You must be logged in to vote
3 replies
@davidandreoletti
Comment options

@davidandreoletti
Comment options

@ate47
Comment options

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
3 participants