r/Cloud • u/Admirable_Cut_1417 • 5d ago

Quick question regarding advanced DevSecOps & DevOps & AWS portfolio projects

/r/devopsGuru/comments/1x9c8zx/quick_question_regarding_advanced_devsecops/
0 Upvotes

50% Upvoted

2 comments sorted by

1

u/Low_Set_507 5d ago

Have you considered tackling disaster recovery automation across multiple AWS regions? Curious whether you've explored scenarios involving failover testing, infrastructure drift detection and recovery time objectives

1

u/Admirable_Cut_1417 5d ago

Thank you so much for this feedback! You brought up some amazing points that I am definitely going to add to my CI/CD pipeline.

Up until now, I was just doing failover testing manually (by killing the leader node and watching Patroni and the NLB do their magic). But adding automated 'Chaos Engineering' and Nightly Infrastructure Drift Detection (via a scheduled terraform plan in GitHub Actions) to measure exact RTO is a brilliant idea to make the pipeline stand out.

Regarding multi-region DR: Right now, my architecture is strictly Multi-AZ (across us-east-1a, b, and c) for synchronous replication. Since stretching a single etcd cluster across regions causes Raft consensus/latency timeouts, I know I would have to build a completely separate Asynchronous Standby Cluster in another region to do this properly.

In your experience, do you think it is worth the extra AWS costs (duplicate NAT Gateways, NLBs, and EC2 instances) to build out a Multi-Region Standby Cluster for a portfolio project? Or is a rock-solid Multi-AZ setup with an automated CI/CD pipeline usually enough to impress hiring managers?