DNS acting weird -- really weird

So, recently I began doing some development work Let's Encrypt

In doing so, I setup a new development machine and recycled my dev domain to point to it.

Since I use linode, I simply go to their DNS page and point that portion of the domain to the new ip address.

A while ago, I wrote a little python tool to help me find out when a series of domain name servers have been updated to the same ip address. Usually I run that code base before letting people know about a new domain. But since I was just doing the testing for myself, I settled for 8.8.8.8 having the most up to date address.

So, a simple script to tell me when the ip has changed to what I expect.

Let me know when googles dns has been updated

a="45.79.87.112"
while [ "$a" == `nslookup dev.jabbari.io |tail -n 3 | grep -i address| awk '{print $2}'` ]
do 
    sleep 1
done \
&& \ 
say done

So, I waited till my Macbook Air kindly notified me that it was done! I fire up my browser and... wait what? It's pointed to my old dev domain. There must be an issue with the browser cache. And NOPE, not a cache issue. It's really pointing to the old domain.

Seeing is believing

Let's count how often it's weird

while [ -ne ]
do a=`nslookup dev.jabbari.io | tail -n 2 | grep address -i | awk '{print $2}'`
if [ "$a" == "45.79.87.112" ]
    then
         echo "new" >> somefile
    else
        echo "old" >> somefile
    fi
done

Let's actually count

cat somefile | grep old | wc -l
13192

cat somefile | grep new | wc -l
1049

How can the same domain name server give a different response? When I login to another machine, my linodes in this case, and query 8.8.8.8 for dev.jabbari.io, I get the "new" ip address.

Every other DNS has it right

 python main.py dev.jabbari.io
Result for dev.jabbari.io
Country     DNS IP      Result
BD  203.112.202.195        45.79.87.112
BE     195.35.110.4        45.79.87.112
BG    213.169.55.10        45.79.87.112
BM     206.53.177.3        45.79.87.112
BO    200.58.161.25        45.79.87.112
JP  221.186.252.130        45.79.87.112
JM      200.9.115.2        45.79.87.112
JO      212.118.0.2        45.79.87.112
BR   200.221.11.101        45.79.87.112
BS      65.75.69.58        45.79.87.112
BY      217.21.48.1        45.79.87.112
RU  217.199.217.200        45.79.87.112
RO    217.156.106.1        45.79.87.112
GT    200.49.160.35        45.79.87.112
GR  195.251.123.232        45.79.87.112
...

What gives?

I have no idea. Could it be that my ISP is using some kind of cache and redirecting answers?

Of course, I took a break to eat dinner and by then, whomever was responding on behalf of 8.8.8.8 had updated its table.

Update:

I had some time to think about it and I think I know what was going on. 8.8.8.8 is not on DNS responding. It probably behind a node balancer and I was lucky enough to get connected to the one that had out dated information.

February 10 2016

---