CoRecursive: Coding Stories · Adam Gordon Bell - Software Developer

Story: From Project Management to Data Compression Innovator

·59 min·5 clips

Jan built LZ4, decompressing at one gigabyte per second, and knew it could revolutionize data storage.

Adam Gordon-Bell profiles Yan Collet, who spent his career as a project manager organizing Excel spreadsheets and coordinating government military communication contracts in Paris, while building some of the fastest data compression software ever written as a side project. Yan's origin story begins with an HP-48 graphing calculator in the early 1990s, where he attempted to compress a game he had been writing since secondary school. The compression work gradually consumed his attention entirely — he stopped touching the game and kept improving the compressor. After years of learning assembly on the calculator, he moved to PC development and discovered an online forum for compression enthusiasts that required answering trivia questions to join, providing him with a community and frame of reference for the first time. He describes the forum as critical to sustaining multi-year effort in isolation. His first open-source release encountered immediate competition from Google's Snappy, released just one week earlier with significant corporate backing and press attention. Rather than retreating, Yan doubled down, systematically studying why his algorithm behaved the way it did and improving it percentage point by percentage point over six months until LZ4 was measurably faster than Snappy across all benchmarks. LZ4's first commercial adoption came through indie game developers needing fast decompression on constrained hardware, then Unity, then Hadoop, and eventually the Large Hadron Collider. Getting LZ4 merged into the Linux kernel took a year — driven by engineers at LG, rejected multiple times on naming convention grounds — but Yan describes the moment it landed as the closest thing to being knighted by a community that otherwise had no ceremony for it. Recognizing that LZ4 was too lightweight to displace Zlib for users who needed higher compression ratios, Yan spent years learning more advanced entropy encoding techniques and eventually invented a new entropy coder called Finite State Entropy, which had not been done in roughly forty years. He combined this with what he had learned from LZ4 to create ZStandard, which compresses comparably to Zlib but at three times the speed. This attracted attention from every major technology company simultaneously, arriving at the same moment Yan learned he was about to become a father — forcing him to decide between his safe Paris career and accepting one of the Silicon Valley offers to work on compression full time.

As heard by us

A hobbyist obsession becomes hidden infrastructure, with enough technical bite to reward patient listeners.

Jan Collette's arc reads like a software career that only looks improbable in hindsight: a project manager in Paris, worn down by corporate life, goes on to create LZ4 and Zstandard, compression algorithms that now sit inside databases, operating systems, file systems, and more.

Read the full review in PlayNext →

Why you'd press play

For the listener who likes improbable career arcs and the hidden plumbing of software.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from CoRecursive: Coding Stories