packfiles

While there are a lot of references to packfiles online, surprisingly, the actual format of packfiles was rather underdocumented.

But when you go in packfiles the content is actually diff'ed because it compresses so well.

Most resources just mention that they exist, and describe how to use `git verify-pack` to inspect a packfile, without explaining how to parse packfiles and apply deltas.

* More frequent synchronizations (peers always online) => More immediacy => Smaller packfiles => Higher probability that multiple peers want the same packfiles.

* Less frequent synchronizations (peers go offline regularly) => Less immediacy => Bigger packfiles => Lower probability that multiple peers want the same packfiles.

The disadvantage here is that any given file in the repo could be stored in an ever increasing number of packfiles.

* Higher popularity => More commits => More permutations of packfiles => Lower probability that multiple peers want the same packfiles (and stronger trends toward small/inefficient packfiles).

Git does do delta compression in packfiles, but the fundamental primitives lack deltas.

That said, I am planning to add versioning and I am looking at adopting git's packfiles for that purpose.

You're planning to deflate, applying deltas, huge packfiles with 4G objects in Javascript?

The delta'ing doesn't happen until later, if and when packfiles are constructed.

But I think the infohash should actually be shared, packfiles are pretty deterministic in practice.

This ("Unpacking Git packfiles") was a CTF challenge a few weeks ago (at the Haxpo CTF in Amsterdam), except we weren't given the original repository, we only got a pcap dump of the traffic.

Peers don't have to seed packfiles the way we're used to for "traditional" seeding of movies or music; these packfiles actually represent transition from one commit to another (instead of "all content from beginning to now"), so they are inherently ephemeral.

Git may handle _compression_ in terms of deltas (see 'Packfiles' in [2]), but logically, a commit should be thought of as equivalent to the state of all files that are being tracked.