DNS Propagation Delays - Why Your Site Went Down When Nothing Changed

in #technology • last month

So your site just went offline. You check the server - it's fine. Code hasn't changed. Deploys look clean. What the hell happened?

There's a good chance it's DNS.

The thing nobody thinks about

DNS records have a TTL (time to live) value. That number tells every DNS resolver on the internet how long to cache your record before asking again. Set it to 3600 and resolvers will keep the old answer for up to an hour, even if you changed it 59 minutes ago.

Here's where it gets fun.

You moved to a new server last week. Updated the A record, saw your site load, called it done. But you left TTL at 86400 (24 hours). Some ISPs and corporate networks are even more aggressive - they'll cache beyond TTL if they feel like it. So when your old server finally goes offline...

Some users can still reach you. Others can't. And you're staring at your monitoring dashboard wondering why half the world thinks your site is dead.

TTL is a suggestion, not a rule

This trips up a lot of people. RFC says resolvers SHOULD respect TTL. In practice? Some resolvers (looking at you, certain ISPs in Asia and South America) cache way longer. Google's 8.8.8.8 is pretty good about it. Your office ISP from 2003? Not so much.

Before any DNS change, drop your TTL to 300 (5 minutes) at least 24-48 hours in advance. This is the part everyone skips. You can't lower TTL and change the record at the same time - the old high TTL is already cached everywhere.

The migration checklist nobody follows

  1. 48 hours before: Lower TTL to 300
  2. Wait the full old TTL period (if it was 86400, wait 24h)
  3. Make your DNS change
  4. Keep both old and new servers running simultaneously
  5. Monitor from multiple geographic locations
  6. After 48h with no issues, raise TTL back up

Step 4 is where most outages happen. People kill the old server the moment DNS is updated. But "updated" in DNS land means "started propagating" not "finished propagating."

Tools that actually help

dig +trace example.com shows you the full resolution chain. Run it from different machines if you can.

dig @8.8.8.8 example.com vs dig @1.1.1.1 example.com - compare what Google and Cloudflare see. If they disagree, propagation isn't done.

There's also whatsmydns.net for a quick visual check across global resolvers.

The sneaky one: CNAME chains

If you're pointing a CNAME to another CNAME to an A record, each hop has its own TTL. The lowest TTL in the chain wins for caching, but the highest one can block propagation of changes further up. Keep your CNAME chains short. One hop max if possible.

Real talk

Most DNS-related outages aren't DNS problems. They're planning problems. The system works exactly as designed - we just forget that "designed" means "eventually consistent across thousands of independent caching resolvers worldwide."

Drop your TTLs before changes. Keep old infrastructure alive during transitions. Monitor from outside your own network. That's basically it.

Not glamorous advice but it would've saved me a few 2 AM calls.

Sort:  

Bajé el TTL a 300 48 horas antes de cambiar el A record y esperé el TTL anterior de 86400; la diferencia se nota porque los usuarios dejaron de ver la página caída. Esto es genial para evitar el “half‑world down” que describís 🚀.

Keeping the old server running during the change is such an easy step to miss. I’d add one more thing to that checklist: save the A, NS and CNAME records before migrating, so you have a before-and-after comparison.

I work on whoisfreaks.com, which offers live and historical DNS lookups. The history can help recover older captured records if nobody saved a copy. It complements the resolver checks you’ve shown when you’re trying to piece together what changed.