| date | permalink | header_image | title | description | author | tags | ||
|---|---|---|---|---|---|---|---|---|
2020-06-09 |
/2020-06-09-libp2p-in-2020/ |
/header-image-libp2p.png |
libp2p in 2020 |
The libp2p Team |
|
If you’re doing anything in the decentralized, peer-to-peer space, you’ve probably heard of libp2p - a modular networking stack for peer-to-peer applications. libp2p consists of many modular libraries from which p2p network developers can select and reuse just the protocols they need, while making it easy to upgrade and interoperate between applications. This helps web3 developers get up and running faster, build more resilient decentralized applications, and utilize advanced features like decentralized publish-subscribe and a distributed hash table.
What makes libp2p different from the networking stack of today is its focus on transport agnosticism, modularity, and portable encodings (like multiaddr). These properties make libp2p the networking layer of choice for most new dweb projects, blockchains, and peer-to-peer applications. Read more about why projects are choosing to build on libp2p, or watch the recent talk from Tech Lead Raul Kripalani at DevCon5.
The past year has seen amazing improvements to libp2p with the protocol quickly becoming the de facto web3 networking layer for a huge swath of projects.
Probably the most significant news on that front is the official adoption of libp2p as the networking layer for the Ethereum 2.0 Network Specification. Ethereum 2.0 is the next generation of Ethereum, with a large and thriving network of clients, dapps, and developers. A big focus for libp2p in 2019 was leveling up stability, scalability, and ease of use to meet the requirements of such a large scale blockchain.
libp2p now has 7 language implementations:
- go-libp2p by Protocol Labs
- rust-libp2p by the Web3 Foundation
- js-libp2p by Protocol Labs
- cpp-libp2p by Soramitsu
- jvm-libp2p by Harmony, PegaSys, and Web3 Labs
- nim-libp2p by Status.im
- py-libp2p by the Ethereum Foundation
<nbsp;>
The other major new blockchain building on libp2p is Filecoin which plans to launch their public storage and retrieval network this summer. With over 7.5 PB of storage already live across an active community of testnet miners, a big focus for libp2p in 2019 was landing the performance and hardening improvements to meet Filecoin’s needs.
In addition to Filecoin and Ethereum 2, there are now more than two dozen projects building directly on libp2p, including: Polkadot and Parity Substrate, Nervos, Status.im, Paratii, 0x, Coda, Enigma, Flow, Harmony, Golem, Keep Network and Validity Labs. Groups like Matrix.org are also in the early stages of adding libp2p support to their applications, helping their users benefit from the resilience and openness of p2p networking.
These newcomers to the libp2p ecosystem are a big deal - but they actually still make up less than half of all active libp2p nodes! The vast majority of libp2p end users come from dapps, tools, and projects using libp2p through IPFS, the InterPlanetary File System!
Millions of end users and over 100,000 libp2p nodes participate in the IPFS network every week. Organizations like Opera, Ethereum, Brave, Netflix, Microsoft, and Cloudflare have joined the IPFS ecosystem, and hundreds of nascent community-owned projects like Audius, Fleek, Textile, Qri, Pinata, OpenBazaar, Berty, Infura and 3Box are building their businesses with IPFS.
The focus for the year was hardening and evolving the stack - and we blew that out of the water! Here are some of the features the core team and our community have contributed to libp2p:
- In 2019, we added AutoRelay, a feature that ensures any user can launch a libp2p node easily without having to open ports on their router or do anything technically complicated. Relay nodes will handle the relaying of traffic. Coming soon is an automatic upgrade to direct connections from relayed connections.
- We added better connection management, with decaying tags and connection gating.
- We built and released Gossipsub, wrote a new explainer, and later enhanced with security extensions including peer scoring, opportunistic grafting, peer exchange, and other features.
- Specs improvement in 2019 has made a night-and-day difference for cross language compatibility and the ease of implementing libp2p in new languages.
- We simplified the go-libp2p codebase by centralizing abstractions and interfaces under the go-libp2p-core module, decommissioning 12+ repos along the way. Further codebase simplification is on the roadmap for 2020.
- A huge focus for the year was the Async Await refactor in js-libp2p - enabling the use of streaming iterables instead of pullstreams, along with some interface refactors.
- The libp2p core team has launched a massive documentation overhaul. Check out docs site and the new Getting Started guide.
- We’ve completed a major DHT improvement project. Queries to the DHT are much faster, thus better supporting DHT-intensive applications like IPFS’s name resolver.
- Support for Noise handshakes has been spec’ed, and is now available in all 7 implementations of libp2p.
- We’ve continued to update our QUIC implementation to keep pace with new releases of the QUIC standard. We’ve also made progress on implementing QUIC in js.
- We introduced certified peer records to enhance foundational protocol security. They’re used by gossipsub and the DHT, and are available to application protocols too.
- We have prototyped an introspection protocol, with a reference implementation in go-libp2p. We’ll soon release a framework and UI shell that will interoperate with all libp2p implementations.
Releases
- go-libp2p had 19 releases including v0.9.5.
- js-libp2p had 41 releases including v0.28.0.
- rust-libp2p had 10 releases including v0.19.1.
- cpp-libp2p had 2 releases including v0.0.1-p0.
- py-libp2p had 6 releases including v0.1.5.
- jvm-libp2p had 3 releases including v0.2.0.
We saw huge growth in the libp2p team and community this year, with presentations and workshops at a variety of conferences including EthCC 2, Web3 2019, EthBerlin Zwei, IPFS Camp, ReadyLayerOne, DevCon V (talks: state of libp2p, gossipsub), and more.
The libp2p project also launched a development grants program. The initial kickoff for the program at EthBerlinZwei led to several significant features being implemented by the community, including Noise handshakes in Go and Lua Wireshark dissectors. The bounties program is a great way to get involved in the libp2p project. Check it out on GitHub!
Throughout the first half of the year, our top focus was supporting the needs of current and in-flight adopters, especially IPFS, Eth2, and Filecoin. So far, we have delivered on the following.
To support IPFS and all our other dapp users out there, we’ve improved libp2p’s content routing and connectivity to make finding and sharing content in distributed networks fast and reliable. This meant we needed to dive deep into major systems like the libp2p DHT, providing subsystem, NAT traversal, and more to optimize and refactor for performance. We’ve been collaborating closely with Protocol Labs’ IPFS and Testground teams to make this a reality. Read more about our progress in the IPFS 0.5 launch post, and our path to get there in the Road to the New DHT post!
What is Testground you ask? Only the coolest new testing tool for large-scale p2p networks! When starting our network upgrade plans, it was clear that the scope and complexity of changes we’d need to make to core IPFS network functions would be impossible to optimize without powerful testing tools to simulate each change to network performance at scale. We couldn’t find a tool with the scale and configurability we needed, so we built the testing tool we needed, and made it modular and system agnostic just like libp2p so other p2p networks can take advantage of this new OSS infrastructure too!
For our large blockchain users, the focus of Q1 was stability, scalability, and security hardening. We designed and implemented a new version for the Gossipsub protocol, v1.1. This new P2P PubSub router includes several security extensions that add a protection to Sybil, eclipse, and spam attacks.
This work is of extreme importance as our adopters (e.g., Filecoin and Ethereum 2.0) require a secure messaging layer to distribute their time sensitive and valuable data, without being vulnerable to malicious actors.
You can find the specification and implementation on GitHub. We will be releasing a report that fully describes how we identified the attack vectors, developed the motivations and went through implementing, testing, evaluating under multiple network conditions and got it audited. Stay tuned.
Both of these workstreams (content routing performance and gossipsub hardening) continue to be top priorities with IPFS having launched go-ipfs 0.5 (their largest network upgrade in years), and Ethereum and Filecoin gearing up for Q3 mainnet launches, as well as other adopters such as Polkadot (who went live recently!), Flow, Coda, 0x, Matrix, Keep Network, Enigma, Harmony, Golem, Validity Labs, etc., reaching new milestones.
Furthermore, in H2 we plan to work on protocol negotiation enhancements, a more efficient connection boostrapping flow, an implementation alignment framework (including a long-awaited conformance test kit), code refactors, and a lot more!
Protocol Labs’s small and scrappy libp2p core team transformed into a true force of nature with some great additions last year. Our doors will continue to be open for great people from both inside and outside of the ecosystem.
In particular, we’re looking for a new Project Lead to help grow and steward the libp2p project and ecosystem to new heights in 2020. Check out the role description for more specific details and information on where you can apply. In particular, we’re looking for candidates with proven leadership experience, a passion for p2p tech, and an interest in scaling the libp2p community as it becomes critical infrastructure for new, large users such as Ethereum (with launch of the Beacon chain) and Filecoin.
Excited about building the networking stack for a more resilient, upgradeable, open network for humanity’s knowledge? We’d love to get in touch! Apply, or just shoot [email protected] a quick email with your thoughts or questions!
2019 was a fantastic, high growth year for the libp2p project and 2020 is already well on track to be even more exciting. Huge thank you to the thousands of community members who contributed to all the new language implementations, feature improvements, performance speedups, and amazing projects bringing libp2p directly to end users.
Thank you for your support! We’re excited to continue working with you to build libp2p.
References: