ACDE #206: Public Testnets are Not for Breaking

The podcast discusses the significant issues stemming from the Pectra upgrade on the Holeshki testnet, which has been deemed one of the worst upgrades in Ethereum's public testnet history. This incident marks the first complete failure of a long-lived public testnet due to an upgrade, leaving only the Sepolia testnet available for public testing. The failure occurred when three out of five execution layer client teams misconfigured the deposit contract address, leading to a chain split among validators. Currently, only about 10% of validators are correctly building the Holeshki blockchain, complicating the path to finalization, which requires at least 66% validator participation.

Developers are working on a recovery plan proposed by Prism developer POTUS, which involves ensuring all validators use updated client software and attest to the correct version of the Holeshki testnet. This plan includes a mass slashing event to remove inactive validators, aiming to increase active participation and help finalize the correct head of the Holeshki chain. However, the lengthy slashing process makes Holeshki unsuitable for testing the Pectra upgrade, necessitating reliance on Sepolia and other testing infrastructures.

The Holeshki testnet has been crucial for the Lido team, but with its unavailability, they face limitations with Sepolia's permissioned validator set. Discussions are underway to establish a new, shorter-lived Pectra development net, raising concerns about setup time and its impact on the Pectra mainnet activation timeline. The initial 30-day buffer between the Sepolia upgrade and Ethereum mainnet activation may be extended due to these challenges, potentially delaying the Pectra mainnet activation to late April or early May.

The decision to bundle the Sepolia and Holeshki releases has faced criticism, especially as ongoing bug discoveries in Pectra DevNet 6 could require re-releasing client software. The speaker emphasizes that public testnets should not be broken and advocates for prioritizing safety over speed in upgrades. Georgios, leading the Reth client, believes the Holeshki incident should not delay Pectra or future planning for the Fusaka upgrade, but the speaker calls for a pause to learn from the experience.

The conversation highlights the need for major applications like Lido and decentralized exchanges to support the upcoming Pectra upgrade effectively. Participants suggest that developers consider slowing the pace of upgrades to prioritize safety, reflecting a cautious approach in light of recent incidents. Anticipation builds for updates regarding the outcomes of the Revime and Holeshki incidents, which are expected to clarify testing substitutes for Pectra and their implications for the mainnet timeline. There is a recommendation to potentially delay the Pectra mainnet timeline by a few weeks to ensure thorough preparation.