Database Backup, Restore, and Disaster Recovery

Implement continuous WAL archiving to enable point-in-time recovery (PITR) for PostgreSQL databases. This allows restoration to any moment before a failure, minimizing data loss to seconds rather than hours. Indexers must configure archive_command to ship WAL segments to a separate, durable storage location. Without PITR, a database corruption event forces a full re-index from genesis, which for high-volume subgraphs can mean weeks of downtime and lost query fees. Chainscore can validate your WAL archiving pipeline and recovery procedures.

A backup that has never been restored is not a backup. Indexers must schedule regular pg_basebackup operations and, critically, perform automated restore tests in an isolated environment. This validates backup integrity, measures recovery time objectives (RTO), and ensures the operational team is practiced in the procedure. Common failure points include missing WAL segments, incorrect permissions, or insufficient disk space on the restore target. Chainscore can design a full backup validation framework and runbook tailored to your indexer stack.

Maintain a geographically separate disaster recovery (DR) environment with a warm standby database using streaming replication. In the event of a primary site failure, operators can promote the standby and redirect graph-node instances. The DR plan must account for DNS cutover for query endpoints, reconfiguration of indexer-agent to point to the new database, and state channel continuity to avoid payment disputes. Without a documented and tested failover procedure, a data center outage becomes a prolonged service disruption. Chainscore can review your DR architecture and failover automation.

Implement proactive monitoring on backup processes, not just database uptime. Critical metrics include WAL archive lag, last successful base backup timestamp, replication slot status, and available storage capacity on backup targets. Silent backup failures are a leading cause of unrecoverable data loss. Alert thresholds should trigger before storage exhaustion or replication lag exceeds the acceptable recovery point objective (RPO). Chainscore can build Prometheus alerting rules and Grafana dashboards specifically for indexer database backup health.

A clear, tested runbook is essential for responding to database corruption, accidental data deletion, or hardware failure. The runbook must define decision trees: when to attempt PITR versus failing over to a standby, who has authority to initiate recovery, and how to communicate status to delegators and gateway users. Post-recovery steps should include verifying subgraph indexing has resumed correctly and that query responses match on-chain state. Chainscore can help develop and tabletop-test incident response procedures specific to indexer database failures.

Indexer databases storing full indexed state can grow to multiple terabytes. Backup retention policies must balance recovery needs against storage costs. Define separate retention periods for base backups and WAL archives based on the maximum acceptable recovery window. Consider tiered storage with object storage for older backups. A common mistake is retaining only the latest base backup, which is useless if corruption existed before it was taken. Chainscore can model storage costs and design retention policies aligned with your recovery objectives.

Maintain an internal disaster recovery runbook that links each backup source to a tested restore procedure, RTO/RPO target, responsible operator, and validation checklist. For The Graph, validation should include PostgreSQL startup, graph-node startup, subgraph health, indexing head progression, query correctness checks, RPC connectivity, and monitoring alerts. Keep screenshots, command transcripts, backup identifiers, WAL ranges, and restore timings from each drill. Chainscore Labs can review these runbooks, run restore simulations, and identify gaps between documented backup coverage and the actual components required to resume indexer service safely.