Block-Height Activation Monitoring Guide

introduction

CONSENSUS FAILURE AND SLASHING RISK

The Cost of Missing a Block-Height Activation

A single missed upgrade height triggers an immediate consensus failure, resulting in downtime penalties, potential slashing, and emergency intervention for validators and infrastructure operators.

For a Cosmos SDK-based chain, a planned upgrade activated at a specific block height is a hard, deterministic deadline. The x/upgrade module's Plan is written to on-chain state by a passed governance proposal. When the network reaches the designated height, the CometBFT consensus engine halts and requires every validator to restart their node with the new, agreed-upon binary. Missing this height is not a performance degradation; it is an immediate and total failure to participate in consensus. A node running the old binary will panic with a UPGRADE_NEEDED error, stop signing blocks, and fall out of the active validator set.

The operational cost is immediate and financial. A validator that fails to upgrade on time stops earning block rewards and transaction fees for the duration of the outage. More critically, on chains using the Cosmos SDK's default slashing logic, downtime is penalized. The validator incurs a DowntimeJailDuration and a slashing penalty proportional to the SlashFractionDowntime parameter. For a validator with a significant delegation, this represents a direct loss of staked principal for both the operator and their delegators. The reputational damage often outweighs the financial penalty, as delegators monitor reliability metrics and may redelegate away from operators with a history of missing critical upgrades.

The blast radius extends beyond a single validator. Infrastructure providers such as exchanges, custodians, and wallet services that run full nodes to index transactions or broadcast user operations will also experience an outage if their nodes are not upgraded. This can halt user deposits and withdrawals, creating a significant customer service incident. A robust monitoring and alerting system that programmatically queries the current_plan via the Cosmos SDK's gRPC or REST endpoints is not a best practice—it is a core infrastructure requirement. Chainscore Labs can design, implement, and test a custom upgrade monitoring and alerting system, ensuring that a validator's on-call team is paged well before the critical height, and can execute a pre-rehearsed upgrade procedure to avoid any consensus failure.

UPGRADE DETECTION AND ALERTING

Monitoring Quick Facts

Key operational facts for programmatically detecting a scheduled upgrade and preventing a consensus failure on a Cosmos SDK chain.

Area	What changes	Who is affected	Action
Upgrade Plan	An on-chain governance proposal passes, setting a future 'Plan' with a name and block height.	Validator operators, RPC providers, indexers	Query the upgrade module's 'current_plan' endpoint to detect a new scheduled upgrade.
Upgrade Height	The chain will halt at the specified height and await a patched binary with the correct upgrade handler.	Validator operators, sentry node operators	Monitor the node's current block height against the plan's height. Set an alert for 1000 blocks before the halt.
Consensus Failure	At the upgrade height, nodes running an old binary will log 'UPGRADE "%s" NEEDED at height: %d' and stop participating in consensus.	Validator operators	Configure log monitoring to trigger a critical alert on the 'UPGRADE NEEDED' string to detect a missed upgrade immediately.
Binary Readiness	A new binary release containing the required upgrade handler must be installed and restarted before the halt height.	Validator operators, infrastructure teams	Automate the detection of a new tagged release in the chain's repository and begin the deployment pipeline.
Cosmovisor Automation	Cosmovisor can automatically swap the binary at the halt height if the new binary is placed in the correct directory.	Validator operators	Verify the 'DAEMON_ALLOW_DOWNLOAD_BINARIES' and 'DAEMON_RESTART_AFTER_UPGRADE' settings and pre-stage the binary.
IBC Client Expiry	A halted chain cannot send IBC packets, risking light client expiry on counterparty chains if the halt is prolonged.	Relayer operators, counterparty chain validators	Monitor IBC client trust periods and prepare to submit a client update immediately after the chain restarts.
Post-Upgrade Verification	After restart, the node must produce blocks, and critical endpoints must be reachable.	Validator operators, RPC providers, wallets	Automate a health check that queries the node's status, latest block, and a sample balance endpoint after the upgrade.

technical-context

THE UPGRADE MODULE LIFECYCLE

How the Upgrade Module Schedules and Halts the Chain

A technical breakdown of the Cosmos SDK `x/upgrade` module's role in coordinating a deterministic, governance-triggered chain halt for planned network upgrades.

The Cosmos SDK x/upgrade module is the on-chain coordination mechanism that transforms a passed governance proposal into a deterministic, network-wide halt. When a SoftwareUpgradeProposal passes, it writes a Plan to the module's state, specifying a name and a block height. This Plan is the single source of truth that every full node queries to determine when to gracefully shut down, ensuring that all validators stop at the exact same block, a prerequisite for a coordinated state migration and binary swap.

Operationally, the module's BeginBlocker logic is the critical path. At the start of each block, the module checks if the current block height matches the scheduled Plan.Height. If it does, the x/upgrade module panics the application, causing the node to halt with a UPGRADE_NEEDED consensus failure. This is not a crash but a designed, controlled stop. The node will refuse to proceed until the operator replaces the binary with a new version that contains an UpgradeHandler registered for the Plan.Name. This handler executes the required state migrations before the chain resumes, making the module the linchpin of the entire hard fork process.

For infrastructure teams, the Plan object is the primary monitoring target. The x/upgrade module exposes a GET /cosmos/upgrade/v1beta1/current_plan gRPC and REST endpoint. A null response means no upgrade is scheduled. A non-null response provides the exact height and name an operator needs to prepare. Alerting systems should poll this endpoint and trigger a critical alert if the current node version does not have a registered handler for the scheduled Plan.Name or if the upgrade height is within a configurable window, such as 10,000 blocks. This transforms the upgrade from a surprise event into a scheduled maintenance operation, preventing downtime and slashing.

AFFECTED ACTORS AND OPERATIONAL REQUIREMENTS

Who Needs Upgrade Height Monitoring

Validator Operators

Validators face the most acute risk from missed upgrade heights. Failure to install the correct binary and restart before the first post-upgrade block results in a UPGRADE_NEEDED consensus failure, causing the node to halt and the validator to stop signing blocks. This leads to immediate downtime slashing and, if prolonged, potential jailing and tombstoning.

Operational Checklist:

Monitor the upgrade module's Plan via cosmos query upgrade plan.
Set an alert for when plan.height - current_height < 1000.
Automate binary download and swap using Cosmovisor's auto-download feature.
Verify the upgrade handler name in the Plan matches the expected binary.
After the halt, confirm the node is producing blocks on the new version before clearing alerts.

Chainscore can design and deploy a custom monitoring daemon that queries the upgrade plan, verifies the binary checksum, and alerts via PagerDuty before the halt height.

implementation-impact

BUILDING A FAIL-SAFE UPGRADE DETECTION SYSTEM

Monitoring Architecture Components

A robust monitoring stack for block-height upgrades requires specific, composable components that detect the on-chain plan, alert on consensus-critical thresholds, and verify binary readiness. Each component addresses a distinct failure mode in the upgrade lifecycle.

On-Chain Plan Poller

A service that queries the Cosmos SDK upgrade module's /cosmos/upgrade/v1beta1/current_plan endpoint at a high frequency. This component is the first line of defense, detecting a new SoftwareUpgradeProposal that has passed and is scheduled. The poller must parse the plan.height and plan.name fields to extract the exact activation block and required upgrade handler. Operators should configure this poller to trigger a PagerDuty alert the moment a previously unseen plan appears, providing maximum lead time before the halt height is reached.

Consensus Failure Alert Trigger

A critical alert rule that monitors a validator's signed blocks. If the node reaches the upgrade height without the correct patched binary, CometBFT will broadcast a UPGRADE_NEEDED consensus failure and stop signing. The monitoring system must detect this state immediately by tracking the node's signing activity and consensus round steps via the /status and /dump_consensus_state endpoints. A zero block-signing rate after the target height is a definitive signal that the operator has missed the upgrade and is now accruing downtime slashing penalties.

Binary Version Verifier

A pre-upgrade check that compares the running binary's version against the expected release for the scheduled upgrade handler. This component should be integrated into the monitoring pipeline to run automatically after a new plan is detected. It queries the node's /abci_info endpoint to retrieve the current AppVersion and cross-references it with the upgrade handler name from the on-chain plan. A mismatch must generate a blocking alert, preventing the operator from assuming readiness and risking a consensus failure at the halt height.

Block Height Countdown Dashboard

A visualization layer that calculates the estimated time remaining until the upgrade halt based on the current block time and the target height from the on-chain plan. This component transforms raw block numbers into an operational countdown, allowing infrastructure teams to schedule their maintenance windows precisely. The dashboard should display the current network block height, the target upgrade height, the delta, and a dynamically updating ETA. This is the primary situational awareness tool for coordinating a validator set's manual upgrade actions.

Governance Lifecycle Tracker

A monitoring feed that watches the full lifecycle of a SoftwareUpgradeProposal before it becomes an active plan. This component tracks the proposal's deposit, voting period, and tallying phases by querying the /cosmos/gov/v1/proposals endpoint. Alerting on a newly submitted upgrade proposal gives operators days or weeks of advance notice, rather than waiting for the plan to be scheduled. This is essential for teams that need to coordinate binary compilation, security review, and testnet rehearsal before the on-chain plan is locked in.

Post-Upgrade Health Validator

An automated verification suite that runs immediately after the node restarts with the new binary. This component confirms that the node has caught up to the tip of the chain, is signing new blocks, and that its IBC connections are not expired. It queries the /health endpoint, checks the node's catching_up status, and verifies that the number of connected peers is above a safe threshold. A failure in this post-upgrade check must trigger an immediate escalation, as it may indicate a state migration error or a network partition.

BLOCK-HEIGHT ACTIVATION MONITORING

Failure Mode and Consequence Matrix

Operational failure modes for validator and infrastructure teams that fail to programmatically detect an upgrade height, monitor the upgrade module's plan, or respond to the UPGRADE_NEEDED consensus failure.

Failure Mode	Trigger Condition	Immediate Consequence	Affected Actors	Preventive Action
Missed upgrade height detection	Node operator relies on manual block explorer checks or social media announcements instead of polling the /cosmos/upgrade/v1beta1/current_plan endpoint	Node does not halt at the upgrade height; continues on the old binary and produces blocks that are rejected by the upgraded network	Validators, RPC node operators, sentry node operators	Implement a cron job or monitoring agent that polls the upgrade module's plan endpoint every block and triggers an alert when plan.height is within a configurable threshold
UPGRADE_NEEDED consensus failure not alerted	Node halts at the upgrade height but the operator has no alerting on the 'UPGRADE_NEEDED' module in consensus logs or Prometheus metrics	Validator is offline and misses blocks indefinitely; no human is paged to perform the binary swap and restart	Validator operators, infrastructure teams, on-call SREs	Configure a Prometheus alert on the cosmos_sdk_upgrade_need_upgrade metric or a log-based alert for the 'UPGRADE NEEDED' message; integrate with PagerDuty or equivalent on-call system
Incorrect upgrade handler name in plan	The on-chain plan.name does not match the upgrade handler registered in the locally installed binary	Node halts at the upgrade height but the upgrade handler fails to execute; node may panic or produce an app hash mismatch after restart	Validators, full node operators	Verify that the plan.name in the current_plan query response exactly matches the handler name in the release notes for the required binary version; automate a pre-upgrade check script
Binary swap not completed before halt time	Operator detects the upgrade height but delays downloading, verifying, and installing the new binary until after the halt	Validator is offline for the duration of the manual intervention; extended downtime risks slashing and missed rewards	Validators, especially those without automated deployment pipelines	Pre-download and verify the binary checksum well before the upgrade height; use a configuration management tool or containerized deployment to stage the new binary and swap it automatically at halt time
Cosmovisor misconfiguration	Cosmovisor is enabled but the DAEMON_ALLOW_DOWNLOAD_BINARIES flag is set to false and the binary is not manually placed in the upgrades directory, or the upgrade-info.json is missing	Cosmovisor halts the node but cannot find or execute the new binary; node remains offline until manual intervention	Validators using Cosmovisor for automated upgrades	Test the Cosmovisor upgrade flow on a testnet or staging node; verify that the binary is placed in the correct upgrades/<handler-name>/ directory and that upgrade-info.json is present if required
State sync or snapshot dependency after halt	Operator restarts the node with the new binary but relies on state sync or a snapshot that is not yet available for the upgraded network	Node cannot join the upgraded network until a snapshot is produced by another operator; extended downtime	Validators that do not maintain a full local state	Ensure the node has a complete local state before the upgrade halt; do not rely on state sync or external snapshots as the sole recovery method for an upgrade restart
IBC client expiry during downtime	Validator is offline for an extended period after the upgrade halt, and a counterparty chain's IBC client expires due to missed updates	IBC connection to the counterparty chain is frozen; requires a governance proposal or client update to recover	Validators, relayer operators, counterparty chain teams	Minimize downtime by automating the upgrade procedure; monitor IBC client expiry windows and ensure relayers are operational immediately after the node restarts

OPERATIONAL READINESS

Upgrade Monitoring Implementation Checklist

A step-by-step implementation guide for infrastructure teams to build a robust monitoring and alerting system for block-height-activated upgrades. This checklist ensures validators detect the upgrade plan, track block progression, and avoid the UPGRADE_NEEDED consensus failure that results in downtime and slashing.

What to check: The active SoftwareUpgradeProposal or governance-triggered Plan stored in the x/upgrade module.

Why it matters: The canonical upgrade height and handler name are set here. Relying on social coordination or block explorers alone is a single point of failure.

Implementation:

Poll the cosmos.upgrade.v1beta1.Query/CurrentPlan gRPC endpoint or the gov/v1/proposals/{id} REST endpoint for the passed proposal.
Parse the plan.height and plan.name fields.
Set an initial alert if no plan exists when one is expected, or if the plan's height is in the past (indicating a missed upgrade).
Confirmation signal: A non-null Plan with a future height and a recognized handler name matching the expected upgrade release.

resource-links

COSMOS SDK AND IBC

Canonical Resources

Use these sources to verify upgrade plans, activation heights, node behavior, and operational tooling before relying on alerts or automation. Teams should treat on-chain state and official repositories as the final authority for a scheduled block-height activation.

Cosmos SDK x/upgrade module documentation

The Cosmos SDK upgrade module is the primary protocol mechanism behind scheduled block-height activations on SDK chains. Operators should use the module documentation to understand how SoftwareUpgradeProposal plans are stored, how the application halts at the planned height when the binary is not prepared, and how upgrade handlers are expected to run. Monitoring systems should query the on-chain upgrade plan rather than relying only on forum posts or release notes, because the chain state is what determines the actual activation condition.

EXPLORE

Cosmos SDK x/upgrade source code

For teams building production alerting, the source implementation is useful for confirming exact module behavior around upgrade plan storage, handler registration, and halt conditions. Validator SRE teams should review the code path that leads to the UPGRADE_NEEDED failure and verify that their automation distinguishes an expected upgrade halt from an unexpected liveness incident. This is also the right reference when appchain teams customize upgrade handling or need to audit whether a migration is deterministic across validators.

EXPLORE

Cosmovisor operational tooling

Cosmovisor is commonly used by Cosmos SDK validators to manage binary swaps at scheduled upgrade heights. Operators should use the official tooling reference to validate directory layout, upgrade binary naming, environment variables, and restart behavior before mainnet activation. A monitoring guide should not assume Cosmovisor is correctly configured: alerts should still confirm the current block height, the active upgrade name, binary availability, and post-upgrade node health after the process restarts.

EXPLORE

CometBFT RPC documentation

Block-height activation monitoring depends on reliable node and consensus telemetry. CometBFT RPC endpoints such as status, block, consensus state, and health checks are practical inputs for detecting whether a validator is approaching the upgrade height, has halted as expected, or is falling behind peers. Infrastructure teams should test RPC access from their monitoring network, apply authentication or network controls where needed, and avoid depending on a single public endpoint for alert-critical height calculations.

EXPLORE

Cosmos Hub Gaia releases

For Cosmos Hub upgrades, Gaia release notes are the operational source for binary versions, upgrade names, and release-specific instructions. Validators, exchanges, indexers, and wallet infrastructure teams should compare the on-chain upgrade plan against the relevant Gaia release before scheduling automation. Do not infer compatibility from version numbers alone; confirm the expected upgrade handler, build target, dependency changes, and any post-upgrade checks documented by the maintainers.

EXPLORE

Cosmos chain registry

The Cosmos chain registry is useful for discovering chain metadata, public endpoints, chain IDs, asset information, and ecosystem references when building multi-chain upgrade monitors. It should not replace direct on-chain queries for the active upgrade plan, but it can help monitoring teams bootstrap endpoint inventories and route alerts to the correct network context. Operators should validate any registry-derived RPC or API endpoint before using it in production alerting.

EXPLORE

Chains We Build On

Looking to build on a specific blockchain?

We build smart contracts, DeFi applications, wallets, tokenization platforms, and blockchain infrastructure across the major ecosystems teams choose today. That includes Ethereum, Arbitrum, Optimism, Polygon, Avalanche, Solana, Sui, Aptos, Hedera, Stellar, and NEAR, with support for additional EVM and non-EVM networks based on your product requirements.

Learn More

EVM ecosystems

Ethereum
Arbitrum
Optimism
Polygon
Avalanche
Cronos

Non-EVM ecosystems

Solana
Sui
Aptos
Hedera
Stellar
NEAR

Additional ecosystems

Polkadot
Cosmos
TON
Cardano
Algorand
Tempo

Also available for Base, appchains, custom EVM networks, and cross-chain product architecture.

UPGRADE MONITORING FAQ

Frequently Asked Questions

Common operational questions for infrastructure teams building alerting systems to detect and respond to scheduled protocol upgrades on Cosmos SDK chains.

The canonical signal is the existence of an active Plan in the x/upgrade module. Query the cosmos.upgrade.v1beta1.Query/CurrentPlan gRPC endpoint or the upgrade/current_plan REST endpoint. A non-null response containing name and height fields confirms a governance-passed, scheduled upgrade. Monitoring this endpoint is more reliable than parsing governance proposals because it reflects the state after a SoftwareUpgradeProposal has passed and the plan has been set. If the plan is null, no upgrade is currently scheduled, regardless of any active proposals.

Trusted by Industry Leaders

Delivering blockchain solutions for 5+ years.

We have partnered with 50+ leading DeFi protocols, NFT ecosystems, and fintech innovators to build secure, scalable, and capital-efficient blockchain products.

Selected Partners & Clients

“I've been working with Chainscore Labs for last 3+ years, they've consistently delivered with strong ownership across multiple projects. The team is reliable and detail-oriented.”

Lee Erswell

CEO, Telos Foundation

how to get started

How to get started?

If you're looking for blockchain integration, ChainScore Labs has 5+ years of experience helping teams build and integrate exchanges, wallets, smart contracts, tokenization solutions, and protocol-connected products, we can help you choose the right path, integrate securely, and get to production faster. Our team consists of experienced blockchain developers and architects who can help you with your blockchain integration needs.

Exploration & Strategy

Define your product goals and choose the right blockchain architecture for your use case.

Architecture & Design

Design the smart contracts, tokenomics, and security parameters of your system.

Development & Integration

Build and integrate with wallets, oracles, and front-end dApps for a seamless experience.

Security & Launch

Comprehensive audits followed by a risk-managed mainnet deployment to protect your users.

Learn More

Explore More Solutions

Discover our
blockchain development services.

We build production-grade blockchain solutions for top-tier projects across DeFi and Web3.

Book a Call

DeFi Protocol Development

End-to-end development of custom DeFi protocols including core smart contracts and system architecture.

DEX & Exchange Development

Building high-performance decentralized exchanges with optimized liquidity and advanced trading logic.

Lending & Borrowing Platforms

Scalable credit markets and money protocols built for production-grade liquidity and security.

Block-Height Activation Monitoring Guide

The Cost of Missing a Block-Height Activation

Monitoring Quick Facts

How the Upgrade Module Schedules and Halts the Chain

Who Needs Upgrade Height Monitoring

Validator Operators

Monitoring Architecture Components

On-Chain Plan Poller

Consensus Failure Alert Trigger

Binary Version Verifier

Block Height Countdown Dashboard

Governance Lifecycle Tracker

Post-Upgrade Health Validator

Failure Mode and Consequence Matrix

Upgrade Monitoring Implementation Checklist

Canonical Resources

Cosmos SDK x/upgrade module documentation

Cosmos SDK x/upgrade source code

Cosmovisor operational tooling

CometBFT RPC documentation

Cosmos Hub Gaia releases

Cosmos chain registry

Looking to build on a specific blockchain?

Frequently Asked Questions

Delivering blockchain solutions for 5+ years.

How to get started?

Exploration & Strategy

Architecture & Design

Development & Integration

Security & Launch

Discover our blockchain development services.

DeFi Protocol Development

DEX & Exchange Development

Lending & Borrowing Platforms

Need a blockchain engineering team?

Discover our
blockchain development services.