It has been a little over six weeks since we launched the Crosslink incentivized feature net, where participants can join the network as miners or finalizers and stake to earn rewards. We wanted to provide an update on how the network has been progressing, the issues we’ve encountered so far, what we’ve learned from them, and what we are currently focused on improving.
As expected at this stage, the network is still immature and has been extremely rough around the edges. Participants have encountered crashes, syncing issues, stalled finality, and a variety of other bugs and edge cases across multiple iterations of the network, particularly around block production, syncing behavior, and network recovery. While that has understandably been frustrating at times for some participants, others have embraced the challenge by helping diagnose issues, coordinate recovery efforts, and better understand how the system behaves in practice. As a result, the feature net has already provided valuable insights into real-world network behavior, and overall we are encouraged by the progress being made.
Rather than repeatedly resetting the network every time a major issue occurs and only practicing under ideal conditions, our goal is to work through these problems in a live environment whenever possible. The kinds of failures, outages, and coordination challenges we are encountering are precisely the kinds of situations a production network must be able to survive. We believe working through them now will ultimately lead to a stronger network.
Stalled BFT
The most significant issue the network is currently facing is a stalled finality layer. Stalls are perhaps the most controversial aspect of BFT systems. Rather than risking transaction rollbacks, a protocol will halt finalization if it can no longer safely reach consensus, choosing safety over availability. This is the opposite side of the tradeoff made by the Nakamoto PoW that Zcash currently uses, which prioritizes availability and continued block production even under adverse conditions, but accepts the possibility of transaction rollbacks through chain reorganizations. Put simply, when network conditions become unfavorable, PoW tends to produce rollbacks while BFT systems tend to produce stalls.
In Crosslink, a BFT stall causes finality to stop progressing while PoW continues producing blocks. This can happen if enough stake goes offline. At this stage, it is still unclear whether the network will recover naturally as finalizers come back online or whether participants will ultimately decide that some form of coordinated recovery action is necessary.
As developers, it is often tempting to simply reset the network when it encounters a difficult state, much like rebooting a laptop when something goes wrong. However, because stalls are a known and expected behavior of BFT systems, and because the feature net is designed to expose real-world operational challenges, we believe there is greater value in understanding and working through these situations. Our focus is on improving the robustness of the software and providing better analytics, monitoring, and diagnostic tools. Ultimately, our goal is to give participants the ability to detect, analyze, coordinate, and recover from these kinds of scenarios themselves, in order to ensure that any future deployment is more resilient under real-world conditions.
To help participants better understand the state of the network, we recently released new visualization and diagnostic tooling focused on finalizer activity and network health. The tooling allows users to identify which finalizers appear online or offline based on recent voting activity, view aggregate online and offline stake, inspect individual finalizer status, and better interpret the current state of the BFT network. Recent updates also introduced additional filtering and the visualization of finalizers’ direct connection status, made possible by peer-to-peer topology improvements.
A major goal of this work is to ensure that participants have the information necessary to make informed decisions themselves rather than relying on Shielded Labs to dictate outcomes. One of the core ideas behind the Crosslink architecture is that different participants in the system have different responsibilities:
-
Wallets protect users.
-
Miners include transactions and produce blocks.
-
Finalizers enforce finality.
-
Stakers select good finalizers.
-
Users slash stakers if they fail to do their job.
To that end, we are developing recovery tooling that could allow users to coordinate an emergency network upgrade if the broader network concludes that action is necessary, including mechanisms that could socially slash stake delegated to persistently inactive finalizers. The operational and coordination challenges involved here are exactly the sort of issues the feature net is intended to surface before any production deployment is considered.
What’s Next
When we originally introduced the incentivized feature net, the plan was for development to progress through a series of “seasons,” where each new season would effectively operate as a new network with updated software, fresh consensus history, and improved functionality. The idea was that participants could continue earning rewards while porting over their finalizer identities and transitioning between progressively more mature networks.
However, based on the issues we have encountered so far, we have decided to move away from the idea of regularly restarting the network and launching entirely new seasons on a predetermined schedule. As mentioned earlier in this post, we believe there is significant value in working through failures and recovery scenarios in a live environment rather than repeatedly resetting the network and starting over. While that approach can be slower and more painful in the short term, we believe it will produce a stronger network and give us a much deeper understanding of how the system behaves under real-world conditions.
Going forward, the plan is to continue iterating on the existing network through regular software updates that improve stability, fix bugs, expand tooling, and introduce more mature functionality over time, only restarting the network when deemed essential for development [1]. We will continue to provide regular updates on network progress, major issues, new tooling, and changes to the feature net as development continues.
Reward Distribution
Rather than tying incentives to discrete “seasons,” rewards will instead be distributed on a rolling six week cadence. Before each payout period begins, we will announce how much ZEC has been allocated for that six week period.
We will also be announcing the payout process for the first six weeks shortly. Because this is the initial distribution period, there will likely be a short delay before rewards are processed as we finalize the tooling and operational workflow. The current plan is for participants to submit a feature net wallet viewing key along with a mainnet Orchard address where rewards should be sent. As originally specified, rewards will be based on cTAZ earned through mining and staking activity during the payout period, and not total cTAZ holdings.
Participants will have a limited claim window to submit the required information and receive rewards. Any unclaimed ZEC remaining after that period may be used as discretionary rewards for contributors who went above and beyond in testing, debugging, reporting issues, or otherwise helping improve the network. As previously mentioned, Shielded Labs will not be a recipient of ZEC-based rewards from the incentivized feature net.
Conclusion
We appreciate everyone who has continued participating despite the instability and the inevitable frustrations that come with testing software at this stage of development. The feedback, failures, and edge cases encountered over the past six weeks have already significantly improved our understanding of the system and will help guide the next phase of development.
Despite the challenges, many participants have continued providing valuable feedback, testing edge cases, reporting bugs, developing tooling, and helping improve the network in meaningful ways. We especially want to thank @zk_nd3r and @kenbak for being consistently engaged and going above and beyond by building block explorers and contributing important infrastructure. We would also like to recognize @dismad, @pacu, Orchard Guardian, and Tom Z for their valuable feedback and contributions throughout the first six weeks of the feature net.
We also understand that the current state of the network can make participation difficult or frustrating for some users. There is absolutely no expectation that everyone participate at this early stage. Participants are free to step away and rejoin later as the network becomes more stable and mature. As previously mentioned, the amount of ZEC allocated to the incentivized feature net is expected to increase over time as the network and tooling continue to improve. We also expect the network to continue running for an extended period of time, likely through 2026 and into 2027. In addition, we plan to continue hosting workshops and community calls on a regular basis, approximately once every six weeks, and there will continue to be plenty of opportunities for participants to get involved.
Footnotes
[1] While we are moving away from planned seasonal resets, we may still reset the network when doing so allows us to simplify the codebase or make significant design improvements that would otherwise break compatibility. Backwards compatibility is not a primary goal at this stage, but it will become increasingly important as we move closer to production readiness.