Solana Validator Performance Report

This article details our Solana validator performance, key validator characteristics, and switch from cloud to bare metal infrastructure -- a major change that has resulted in more performant operations. As emphasized in our recent Ethereum Validator Performance Report, Coinbase is deeply committed to security, supporting the broader network, and fostering an open dialogue with the community.

Our Solana Validators at a Glance

• Staked to Coinbase validators: 40.04M SOL → 10.6% of total staked SOL (as of 6/1/25)

• Validator distribution: 6 countries and 2 bare metal providers 

• Client diversity: 2 clients (Jito, Paladin), with Frankendancer coming soon

• Skip rate: 0.05% (from 5/2/25-6/1/25)

• Running the latest and greatest hardware (i.e., 5th Generation AMD EPYC CPUs)

• Built-in proprietary double signing protections that ensure validator keys are never used on more than one machine at a time

• Capped stake concentration to de-risk operations -- with a goal of eventually moving all validators out of the “superminority”, the smallest number of validators on the Solana blockchain that collectively hold enough stake to control >33% of the network's total voting power

• 100% bare metal

Validator Distribution

Coinbase’s Solana validators are distributed across multiple geographic locations, prioritizing high-performance regions, such as Europe. This supports network decentralization and reduces the risk of simultaneous failures if local outages occur. For redundancy and disaster recovery purposes, each validator has a corresponding backup server in a separate location, adding an extra layer of security.

We operate validators in: 

• Amsterdam

• Frankfurt

• London

• New York City

• Singapore

• Tokyo

We also distribute our validators between two bare metal providers: Latitude.sh and Teraswitch. Diversifying across vendors helps further compartmentalize local outages (at the provider level) and mitigate provider-level failures. It also strengthens decentralization by avoiding concentration of activity with any single provider.

Client Diversity

Client diversity is important for network resilience (avoiding single points of failure), reward stability, and strengthening decentralization. As part of our steadfast commitment to protecting customer assets, our Solana validators currently support two distinct clients with a third on the way: 

• Jito: ~92%

• Paladin: ~8%

• Frankendancer: Coming soon

We are in the process of integrating Firedancer -- known as “Frankendancer” in its current iteration. We are working with Jump Crypto to ensure a secure mainnet rollout once it has been sufficiently battle-tested. Frankendancer’s early voting latency results look very promising, with the latest releases achieving competitive vote credits without any vote boosting mods.

Skip Rate

Validator skip rate measures the percentage of time a validator fails to produce blocks for assigned slots. Missed slots can lead to lower block rewards, with higher skip rates indicating reduced validator performance and reliability. To maximize block rewards, skip rates must be kept low.

Since switching to bare metal, our skip rate (average across our four public validators) has significantly outperformed the network average. Below is our skip rate for the period between 5/2/25 to 6/1/25.

Source: Solana Compass

Moving to Bare Metal

Due to the sheer scale of blockchain nodes under management, all of Coinbase’s staking architecture was originally built on managed Kubernetes clusters distributed across Amazon Web Services (AWS) and Google Cloud Platform (GCP). This setup provided clear advantages: enterprise-grade security, compliance, operational efficiency, and premium support from major cloud providers. While a cloud configuration has worked well for the majority of networks we support, it soon became clear that Solana presented unique challenges. In 2023, the network’s high throughput and tight performance requirements began to expose the limitations of cloud infrastructure.

In October 2024, we completed a full migration of all Solana validators to bare metal. The move required replacing managed Kubernetes with more granular control using Ansible. Single-tenant bare metal enabled us to directly tune BIOS settings, Linux kernel parameters, NVMe formatting, CPU affinity, and more. We also gained access to newer-generation AMD CPUs with high single-core clock speeds, plus Gen4/5 enterprise-grade NVMe storage -- both critical for validator performance. 

Rather than creating proprietary architecture, we embraced the community’s "paved road" approach -- hardware recommendations, performance benchmarks, tuning guides, and best practices. This dramatically reduced our operational burden and made us more adaptable to ongoing network changes. Aligning with the broader ecosystem gave us a stronger, faster, and more resilient deployment model than we could have built in isolation.

Shifting from AWS EC2 to bare metal servers has resulted in significant performance improvements. Key performance metrics include reductions in skip rates (indicating more reliable block production) and vote latency (speeding up network consensus). Enhancements to replay times and catchup times show more efficient data synchronization and faster node recovery as well.

Bare Metal Configuration

To optimize validator performance, we ran extensive tests that focused on decreasing block replay times via kernel and BIOS optimizations. Replay time is critical for validator performance. High replay latency, often due to CPU or disk bottlenecks, can cause a validator to miss votes and slots. If a validator is stuck replaying blocks, it increases vote latency and may skip its assigned leader slots, consequently forfeiting inflation, block, and Jito MEV rewards. The results of our testing are the optimizations below.

After embracing the Solana community’s paved road approach in our switch to bare metal, we are happy to now be able to share some of our own best practices with the community. 

CPU Optimizations

• BIOS Configuration:

  • Hyperthreading/Simultaneous Multithreading Disabled

  • Global C-State Disabled

  • Data Fabric (DF) C-State Disabled

  • Max Package Power Limit (PPL)

  • Max Thermal Design Power (TDP)

  • Power Determinism Mode

  • Hypervisor Disabled

• “amd_pstate” driver “active”: This AMD-specific driver enables hardware-managed CPU frequency scaling on modern AMD CPUs. In “active” mode, the CPU autonomously adjusts frequency based on performance and power hints, outperforming the older acpi_cpufreq driver in both responsiveness and efficiency.

• “cpu/scaling_governor” “performance” Mode: “performance” mode nudges the CPU to prefer higher performance, allowing frequent boosting while still enabling dynamic frequency adjustments.

Disk and File System Optimizations

• Separate NVME Drives for snapshots, ledger, and accounts: Isolating these onto separate drives alleviates disk bottlenecks, enhancing I/O throughput and parallel reads/writes.

• XFS Filesystem: Designed for high-throughput, parallel I/O workloads, XFS demonstrated superior performance over ext4 in our benchmarks.

• noatime: atime, or “access time updates”, is a legacy Linux setting for updating file access times, and setting noatime eliminates these unnecessary disk reads and writes.

Upcoming Developments

We are continuously improving our staking infrastructure to provide the best performance and industry-leading security for our customers. Below are some new implementations in the pipeline for our Solana validators.  

Slashing Precautions

Slashing has yet to be implemented on Solana, but it is expected. To prepare for this, we are proactively taking steps to de-risk and harden our systems. We plan to rebalance stake over time such that validators are not in the “superminority”, or at the very least maintain a stake concentration below 1%. We have a bespoke system for double signing protection that is built for bare metal deployments; we are testing and continuing to iteratively upgrade it. This system ensures two machines can never be active with the same validator keypairs at once.

(Near) Zero-Downtime Deployments

We are taking a cautious and deliberate approach to implementing hot-swapping techniques for near zero-downtime validator deployments. This will help further improve performance. While Solana supports validator key rotation between active nodes, this process carries inherent risk of double signing -- which will be consequential for stakers once slashing is introduced. To eliminate any possibility of double signing and ensure that identical validator keypairs are never present on multiple machines, we are developing a near-zero-downtime deployment mechanism that performs strict preflight checks and key transition validations. This mechanism will deeply integrate with our existing double signing protection system.

DoubleZero

Solana’s core philosophy of “increase bandwidth, reduce latency” (IBRL) can only be fully realized if it is addressed from multiple angles. One of the most promising technologies helping advance this goal is DoubleZero, which addresses latency at the network infrastructure layer. DoubleZero provides a high-performance, fiber-based backbone that directly links data centers. Validators connected to this network effectively “bypass” the traditional internet, achieving significantly lower latency and more consistent throughput.

We have already integrated our Solana testnet validators with DoubleZero’s testnet and are preparing for the upcoming mainnet launch. In addition to latency and throughput improvements, we are excited about DoubleZero’s potential to promote geographic decentralization. Today, global validator distribution is clustered in Europe due the region’s favorable voting latency. This creates a self-reinforcing dynamic that makes it increasingly difficult for other regions to remain competitive. DoubleZero has the potential to level the playing field, enabling regions at the edge of the network to achieve competitive latency and become viable validator locations.

Alpenglow

Alpenglow, the upcoming change to Solana’s consensus mechanism, will be a complete revision of the existing consensus protocol. This change will do away with Proof of History (PoH), Tower BFT, and onchain voting while bringing significant performance improvements, like faster finality. Alpenglow will undoubtedly reshape Solana’s validator operation dynamics. We will be keeping a pulse on these changes and adapting as needed to continuously provide our customers with the most secure risk-adjusted yield possible.

How do you set up your machines? Let’s continue the conversation on X -- DMs are open for all things Solana staking -- s4bs94.

Special thanks to the following individuals for rolling up their sleeves and helping us get to where we are today: ferric (stakeware.xyz) Gabriel Hicks (Phase Labs) nasmithan (Backpack) Tim Garcia (Solana Foundation) Ben Hawkins (Solana Foundation) Dan Albert (Solana Foundation) brooks (Anza)