top of page
Search

Jet Skis, SQL Servers, and the Joy of “High Availability”

  • Writer: Youssef Lteif
    Youssef Lteif
  • Sep 15, 2025
  • 2 min read

Picture this: it’s a sunny Saturday afternoon. I’m out on a jet ski in the middle of the Mediterranean, living my best life. Salt water, sunshine, no worries.


Then my phone starts lighting up like a Christmas tree.


  • Teams notifications

  • Emails

  • Alerts

  • Phone calls



All screaming the same thing:


“The SQL Server Always On Availability Group cluster is down.”


Oh, and did I mention? That cluster supported 50 different applications. Translation: everything was on fire.



Step 1: Stop the Bleeding



I ditched the jet ski, raced home, and logged in. What do I see?


  • Node A (the primary) had its C drive corrupted thanks to a Windows patch gone bad.

  • The failover didn’t work.

  • The cluster was basically a very expensive paperweight.



So first priority: get the apps talking to something.


I yanked down the AG, renamed Node B to take over the listener DNS name, and prayed to the networking gods for quick propagation.


It wasn’t instant, but eventually — poof — the apps came back online. Users stopped screaming. For now.



Step 2: The Long Weekend That Wasn’t



The “fun” part? Fixing Node A.


Here’s how I spent my weekend (and then some):


  • Rebuilding Node A from a clean image.

  • Re-syncing 12TB+ of databases.

  • Running on fumes because apparently sleep is optional when you’re a DBA.



By Wednesday afternoon — after 4½ days of this marathon — Node A was back in the cluster, databases synchronized, and the whole thing stitched together like nothing had ever happened.


Except, you know, my sanity.



Lessons Learned


  • “High Availability” doesn’t mean “no downtime.” It means “you’ll still work your butt off, just with more moving parts.”

  • Always On is great… until it isn’t. Especially when a patch decides to nuke your C drive.

  • Sometimes the only thing standing between 50 apps and chaos is your ability to think fast, rename a server, and buy time.





So yeah, while you were enjoying your weekend BBQ, I was living the dream: jet ski to disaster recovery sprint.


High availability? More like high blood pressure.

 
 
 

Comments

bottom of page