elias-fano

Implementing a semi-index is very easy if you have implementations of Elias-Fano and balanced parentheses data structures.

Encoding-wise, Elias-Fano requires exactly 2 + ceil(log(2^32 / n)) bits, so it would provably fit in the memory budget.

Elias-Fano monotone sequences unfortunately do need a bit more space than just a list of Rice codes.

Elias-Fano is now the main Facebook indexing algorithm and it is slowly percolating to Lucene (look in the sources).

For Java I recommend Sux4j [1] that has a very good implementation of Elias-Fano, but I think that balanced parentheses are very primitive.

Partitioned Elias-Fano indexes may be a superior choice in contrast to the sparse codec in terms of rank and compression, and probably less so for select and code complexity.

If I were coding a static approximate membership query structure, I wouldn't use either bloom filters or GCS, I'd use an Elias-Fano-coded sequence of fingerprints.

> We haven't discussed other solutions like Partitioned Elias-Fano indexes or Tree-Encoded Bitmaps.

Using Elias-Fano coding (which I believe is an instance of Golomb coding), the write position will never overtake the read position, so the merge can be done in-place.

The higher bits of the Elias-Fano encoding happen to be dense and hence we can use the previous dense implementation on that higher bits and then combine it with the lower bits.

For one, space utilization seems to be a serious issue: bits per posting are more than double than the Elias-Fano indexes in their experiments, and furthermore, they claim that they used the Java Partitioned Elias-Fano implementation of MG4J, but MG4J does not have one!

It only has an Elias-Fano index; since they get the same space from PEF and MG4J, this means that either they are using a random ordering of the document identifiers (otherwise they'd get another 2x improvement from PEF with proper docid ordering) or they're misusing the PEF code.

Creation speed is also quite important, do you know how "Partitioned Elias-Fano" performs there?

Elias-fano are overestimated without any proof or comparison benchmarks.