How do we test disaster recovery for critical services on Google Cloud given the latest incidents?

To proactively spot latency spikes on Google Cloud before customers do, SREs should use a multi-layered approach, creating custom dashboards in Cloud Monitoring that combine platform-wide network intelligence with deep, service-specific metrics. Early warnings can come from either a degradation of the underlying network or from saturation in a specific application service.

• Layer 1: Network Intelligence Center

The Google Cloud Performance Dashboard provides a high-level view of network health. It's the first place to look for signs of a broader network issue, which can often precede application-level problems. 

• Layer 2: Application-layer observability

Custom dashboards in Cloud Monitoring should be configured to capture fine-grained metrics for your specific workloads. This provides early warnings of application-level saturation that can cause user-facing latency.

• Layer 3: Tracing and logging

For deeper analysis during an incident, integrate and analyze data from Cloud Trace and Cloud Logging.

For Google Cloud, protecting workloads from regional cable cuts relies on a combination of multi-regional load balancing and intelligent DNS routing policies. By distributing resources and leveraging Google's global network, workloads can automatically shift traffic away from an affected region with minimal disruption.

In light of recent, multi-regional connectivity disruptions, such as the July 2024 issue impacting Google Cloud Networking, runbooks for Google Cloud Platform (GCP) must shift from reactive, single-failure responses toward proactive detection and multi-layered, hybrid failover strategies. A passive approach assuming regional isolation is no longer sufficient.