On June 17, 2024, the Cosmos Hub halted at block height 20,405,000 during the Gaia v17 upgrade, which introduced the Liquid Staking Module. The root cause was a state-breaking condition triggered by a mismatch between the module's internal accounting and the actual delegation state of validators. This non-deterministic bug caused a consensus failure, stopping block production and affecting all downstream operations including IBC packet relay, exchange deposits, and staking reward distribution.
Protocols
2024-06-17 Cosmos Hub v17 LSM Halt
Root cause analysis of the Gaia v17 chain halt triggered by a bug in the Liquid Staking Module. Covers the state-breaking condition, the coordinated v17.1 patch, validator upgrade procedures, and impact on exchanges and staking providers.
introduction
GAIA V17 CHAIN HALT
Incident Overview
A state-breaking bug in the Liquid Staking Module caused the Cosmos Hub to halt at block 20,405,000, requiring an emergency patch and coordinated validator upgrade.
The immediate operational impact was a complete liveness failure for the Cosmos Hub and all IBC-connected consumer chains relying on Interchain Security. Validator operators were required to coordinate an emergency upgrade to the patched v17.1 binary, which included a fix for the LSM accounting logic and a state migration to correct the corrupted delegation records. Exchanges and staking providers had to suspend withdrawals and monitor the chain's recovery to ensure transaction finality before resuming operations.
This incident highlights the systemic risk of introducing complex, state-altering modules without exhaustive edge-case testing against mainnet state. The LSM bug was not detected in testnet environments, underscoring the need for mainnet-state simulation and formal verification of module invariants. For appchain developers and validators, the halt serves as a critical case study in upgrade readiness, emergency response coordination, and post-halt state validation.
COSMOS HUB V17 LSM HALT
Incident Quick Facts
Operational facts for the Gaia v17 chain halt triggered by a Liquid Staking Module bug. Use this table to assess blast radius, required actions, and verification steps.
| Area | What changes | Who is affected | Action |
|---|---|---|---|
Root Cause | Bug in the Liquid Staking Module (LSM) causing a state-breaking condition during validator set updates | All validators and node operators on Cosmos Hub | Verify the exact code path against the Gaia v17.1 patch notes |
Network State | Cosmos Hub halted at block height where the LSM bug triggered a consensus failure | Exchanges, wallets, and IBC-connected chains relying on Hub finality | Confirm the halt block height and chain state from canonical block explorers |
Remediation | Coordinated Gaia v17.1 patch release with LSM bug fix and state migration logic | Validator operators and infrastructure providers | Apply the v17.1 binary after verifying its integrity and coordinating with other validators |
Validator Action | Upgrade to Gaia v17.1, export pre-bug state, and coordinate network restart | All Cosmos Hub validators | Follow the coordinated restart procedure documented by the Cosmos Hub team |
IBC Impact | IBC channels frozen during halt; no packet loss expected if restart uses pre-bug state | IBC relayers, connected appchains, and cross-chain DeFi protocols | Verify IBC client states and packet commitments after restart; monitor for timeout risks |
Exchange Impact | Deposits, withdrawals, and trading suspended during chain halt | Centralized and decentralized exchanges listing ATOM | Confirm chain liveness and state integrity before resuming operations; validate against canonical state |
Staking Impact | Liquid staking module operations halted; staking derivatives may be temporarily affected | Liquid staking providers, stATOM holders, and DeFi protocols using LSM | Verify LSM state consistency after restart; communicate with staking derivative issuers |
Monitoring | Chain halt detection, validator participation, and IBC channel liveness signals | Node operators, monitoring services, and security teams | Review alerting thresholds for chain liveness and validator set participation; test restart coordination procedures |
root-cause
STATE-BREAKING LOGIC IN LIQUID STAKING MODULE
Root Cause Pattern
The Gaia v17 halt was caused by a non-deterministic state transition in the Liquid Staking Module that triggered a consensus failure when validators produced divergent app hashes.
The Cosmos Hub v17 upgrade introduced the Liquid Staking Module (LSM), enabling ATOM holders to liquid-stake already-delegated tokens without unbonding. The halt occurred because the module's BeginBlock logic contained a state-breaking condition: when processing validator set updates, the module iterated over a map that lacked deterministic ordering in the Cosmos SDK's store, causing different validators to compute different state roots for the same block height. This non-determinism is a classic pattern in Cosmos SDK appchains, previously observed in the IAVL library during the Stargate upgrade, where Go's map iteration semantics violate the deterministic execution requirement of BFT consensus engines like CometBFT.
The operational impact was immediate and severe. Validators running the v17 binary halted at the same block height but with divergent app hashes, preventing consensus and stopping block production entirely. The Cosmos Hub, as the economic center of the interchain, froze IBC channels, halted token transfers, and paused all staking operations including delegation, undelegation, and reward distribution. Exchanges and staking providers relying on IBC transfers or ATOM operations were forced to suspend services. The fix required a coordinated v17.1 patch that replaced the non-deterministic map iteration with a sorted, deterministic data structure, followed by a coordinated restart where validators had to apply the patch and resume from the pre-halt state.
This incident reinforces a critical design constraint for Cosmos SDK module developers: any code path executed during BeginBlock, EndBlock, or transaction processing that influences the state root must be strictly deterministic. Common sources of non-determinism include map iteration, floating-point arithmetic, concurrency primitives, and reliance on system time. For appchain teams integrating modules like LSM, the pattern demands rigorous determinism testing in upgrade simulations. Chainscore Labs provides upgrade readiness review that includes determinism analysis, state migration validation, and post-halt state verification to help teams avoid this class of consensus failure.
BLAST RADIUS AND REMEDIATION
Affected Systems and Actors
Immediate Action Required
Validators were the primary actors affected by the v17 halt. The Liquid Staking Module (LSM) bug triggered a non-deterministic state transition, causing the chain to halt at block height 19,850,000. All validators were required to:
- Halt operations immediately upon detecting the consensus failure.
- Coordinate off-chain to identify the last valid state before the bug was triggered.
- Apply the Gaia v17.1 emergency patch, which fixed the LSM state-breaking condition.
- Restart nodes using the patched binary and verify consensus participation.
Post-Restart Validation
After the chain restarted, operators needed to confirm that no state corruption persisted, that IBC clients were unfrozen, and that validator uptime was restored. Operators who failed to upgrade remained offline and were subject to slashing penalties for extended downtime.
remediation-impact
POST-HALT RECOVERY AND UPGRADE VERIFICATION
Remediation and Operational Impact
The v17 LSM halt required a coordinated patch, state validation, and network restart. The following actions are critical for operators to ensure consensus safety and prevent recurrence.
02
Post-Restart State Validation
After the chain restarts, operators must validate that their local application state hash matches the network consensus. The halt occurred mid-block, so the state from the last committed block before the halt must be intact. Run gaiad tendermint show-validator and compare block hashes with trusted RPC endpoints. Any node with a corrupted state must perform a state sync or restore from a verified snapshot to avoid signing on a fork.
03
Exchange and Custodian Deposit/Withdrawal Verification
Exchanges and custodians that support ATOM or LSM-liquid-staked assets must reconcile all pending deposits and withdrawals against the post-restart chain state. Transactions that were in the mempool but not committed are invalidated. Teams should re-broadcast any critical transactions and verify that LSM-related operations (tokenize, redeem) function correctly before re-enabling user-facing services.
04
LSM Module Integration Review
Any dApp, wallet, or staking provider that integrates with the Liquid Staking Module should re-test their transaction flows against the patched v17.1 state machine. The bug was triggered by a specific edge case in LSM tokenization logic. Integration teams should add this scenario to their test suites to ensure their application logic handles the corrected on-chain behavior and does not rely on the pre-patch state transition.
05
Upgrade Readiness Process Review
This halt highlights the need for a robust upgrade readiness process. Teams should review their pre-upgrade simulation and testnet validation procedures. The LSM bug was not caught before mainnet activation. Chainscore Labs can assist with formal upgrade impact assessments, state machine differential analysis, and pre-activation integration testing to identify state-breaking conditions before they reach mainnet.
06
Incident Response and Communication Plan
Validators and infrastructure providers should document their coordinated halt and restart procedures. This incident required rapid coordination between the Gaia core team, validators, and exchanges. Establish clear communication channels (e.g., Discord, Telegram) and define roles for triaging state-breaking bugs, distributing patches, and orchestrating a synchronized network restart to minimize downtime and confusion.
GAIA V17 LSM HALT IMPACT ASSESSMENT
Risk and Compatibility Matrix
Operational impact, failure modes, and required actions for stakeholders affected by the Cosmos Hub v17 chain halt caused by the Liquid Staking Module bug.
| Area | What changes | Who is affected | Action |
|---|---|---|---|
Chain Liveness | Cosmos Hub halted at block height triggering the LSM bug; no new blocks produced. | All validators, RPC providers, and downstream services. | Coordinate with validators to apply the v17.1 patch and restart the network using the agreed-upon export height. |
Liquid Staking Module | A state-breaking condition in the LSM caused a consensus failure when processing specific undelegation logic. | Liquid staking providers (e.g., Stride, pSTAKE) and their users. | Verify LSM-dependent transaction logic against the patched v17.1 binary. Monitor for any residual state inconsistencies post-restart. |
Validator Operations | Validators must halt operations, replace the v17 binary with the emergency v17.1 patch, and coordinate a synchronized restart. | Cosmos Hub validators and sentry node operators. | Immediately upgrade to Gaia v17.1. Verify state integrity against the pre-halt export and ensure peering configurations are correct for the restart. |
Exchange and Custody Services | Deposits and withdrawals for ATOM and all IBC assets routed through the Hub are frozen. | Centralized exchanges, custodians, and fiat on/off-ramps. | Halt all ATOM and Cosmos Hub-originated IBC asset transfers. Monitor validator communications for the restart block height and resume operations only after confirming chain stability. |
IBC and Interchain Accounts | All cross-chain packet relays through the Hub are stalled. ICA channels on the Hub are unresponsive. | Relayer operators, ICA host/controller chains, and DeFi protocols using Hub ICA. | Pause relayer operations for Hub channels. After the restart, monitor for packet timeouts and verify ICA channel liveness before resuming cross-chain operations. |
Staking Providers | Delegation and undelegation transactions cannot be processed. Reward distribution is paused. | Staking-as-a-service providers and liquid staking protocols. | Communicate the halt to delegators. Post-restart, verify that the undelegation that triggered the bug is resolved and that staking module accounting is correct. |
Governance | Active governance proposals cannot reach quorum or pass during the halt. | Governance delegates and proposal submitters. | Review all active proposals for voting period deadlines. Expect a coordinated resubmission or deadline extension proposal after the chain restarts. |
Node Sync and Snapshots | Post-restart, nodes may attempt to sync from invalid or pre-halt snapshots. | New node operators, RPC providers, and snapshot services. | Do not use snapshots generated before the v17.1 patch. Sync from the post-restart genesis or a verified post-patch snapshot to avoid state corruption. |
POST-HALT RECOVERY AND UPGRADE VERIFICATION
Validator and Integrator Remediation Checklist
A structured checklist for validators, RPC providers, and exchange integration teams to confirm safe upgrade to Gaia v17.1, validate post-halt state integrity, and resume operations after the LSM-triggered chain halt.
Confirm that the running binary matches the official v17.1 release hash published by the Cosmos Hub team. The v17.0 binary contains a state-breaking bug in the Liquid Staking Module that causes a deterministic panic. Do not restart with v17.0.
- What to check: Binary SHA256 checksum against the canonical release on the cosmos/gaia GitHub repository.
- Why it matters: An incorrect or tampered binary will fail to reach consensus with the patched network, causing the node to be slashed or stuck.
- Readiness signal: The node's
versionendpoint returns the correct v17.1 commit hash and the binary passes a checksum verification script.
resource-links
CANONICAL REFERENCES
Official Resources and Post-Mortems
Use these resources to verify the Cosmos Hub v17 LSM halt timeline, the Gaia v17.1 patch path, and operational restart evidence. Teams should treat community summaries as secondary to Gaia release artifacts, on-chain records, and validator-observed state.
05
Local Post-Halt Evidence Pack
Create an internal post-halt evidence pack even if a public post-mortem is available. At minimum, archive node logs, CometBFT consensus logs, Gaia application logs, upgrade-info files, binary hashes, config changes, peer lists, snapshot source, RPC responses, and app hashes before and after restart. This gives validators, staking providers, exchanges, and auditors a defensible record for customer communications, SLA analysis, slashing-risk review, and future upgrade readiness checks.
Chains We Build On
Looking to build on a specific blockchain?
We build smart contracts, DeFi applications, wallets, tokenization platforms, and blockchain infrastructure across the major ecosystems teams choose today. That includes Ethereum, Arbitrum, Optimism, Polygon, Avalanche, Solana, Sui, Aptos, Hedera, Stellar, and NEAR, with support for additional EVM and non-EVM networks based on your product requirements.
EVM ecosystems
- Ethereum
- Arbitrum
- Optimism
- Polygon
- Avalanche
- Cronos
Non-EVM ecosystems
- Solana
- Sui
- Aptos
- Hedera
- Stellar
- NEAR
Additional ecosystems
- Polkadot
- Cosmos
- TON
- Cardano
- Algorand
- Tempo
Also available for Base, appchains, custom EVM networks, and cross-chain product architecture.
GAIA V17 LSM HALT FAQ
Frequently Asked Questions
Operational questions and answers for validators, exchanges, and staking providers recovering from the Cosmos Hub v17 chain halt caused by the Liquid Staking Module bug.
The chain halted due to a state-breaking bug in the Liquid Staking Module (LSM) introduced in the Gaia v17 upgrade. Specifically, the RedeemTokens function contained a logic error that caused a validator's ValidatorBondShares to be incorrectly decremented below zero when processing LSM token redemptions under certain conditions. This resulted in a negative shares invariant violation, triggering an immediate consensus failure and chain halt. The bug was not present in the v16 state machine and was introduced as part of the new LSM functionality.
Trusted by Industry Leaders
Delivering blockchain solutions for 5+ years.
We have partnered with 50+ leading DeFi protocols, NFT ecosystems, and fintech innovators to build secure, scalable, and capital-efficient blockchain products.
Selected Partners & Clients
“I've been working with Chainscore Labs for last 3+ years, they've consistently delivered with strong ownership across multiple projects. The team is reliable and detail-oriented.”
L
Lee Erswell
CEO, Telos Foundation
how to get started
How to get started?
If you're looking for blockchain integration, ChainScore Labs has 5+ years of experience helping teams build and integrate exchanges, wallets, smart contracts, tokenization solutions, and protocol-connected products, we can help you choose the right path, integrate securely, and get to production faster. Our team consists of experienced blockchain developers and architects who can help you with your blockchain integration needs.
01
Exploration & Strategy
Define your product goals and choose the right blockchain architecture for your use case.
02
Architecture & Design
Design the smart contracts, tokenomics, and security parameters of your system.
03
Development & Integration
Build and integrate with wallets, oracles, and front-end dApps for a seamless experience.
04
Security & Launch
Comprehensive audits followed by a risk-managed mainnet deployment to protect your users.
Explore More Solutions
Discover our
blockchain development services.
We build production-grade blockchain solutions for top-tier projects across DeFi and Web3.
Book a Call
Start a build
Need a blockchain engineering team?
Send the project context and we will respond with next steps, scope questions, and a practical path to delivery.