duckdb

Maybe the OP is thinking of reading/writing to DuckDB native format files.

If you are already on Ducklake, Duckdb will use that for getting the list of relevant objects anyway.

Parquet is static append only, so DuckDB has no problems with those living on S3.

Love this, nestled in among the posts about forking httpx and similarity search in DuckDB.

I was thinking of using it with Duckdb as well but seems it would be of limited benefit.

As an aside, there's now Lance data format support in DuckDB through their extension.

DuckDB exposes Postgres SQL, and most coding LLMs have been trained on that.

A team may use Trino for federated queries, DuckDB for embedded analytics, StarRocks for low-latency serving, and Athena for pay-per-scan workloads on cold data.

With the help of Claude and DuckDB, I analyzed it and here is what I found.

Unfortunately, even NFS or SMB are not sufficiently FS-like for DuckDB.

DuckDB + Parquet is a way nicer angle than what I had in mind.

I built something on top of DuckDB last year but it never got deployed.

“The data is stored as monthly Parquet files sorted by item ID, making it straightforward to query with DuckDB, load with the datasets library, or process with any tool that reads Parquet.” Out of curiosity, I tinkered with it using Claude to see trends and patterns (I did find a few embarrassing things about me!).