Incident description

At 14:00 on Saturday the 7th of February geant.net domain was not being resolved by ROOT servers

Incident severity: CRITICAL

Data loss: NO

Affected Services 

All the systems trying to send email to user on dante.net and geant.net

• relay.geant.org and mail.geant.net, Dahsboard, repositories.geant.net, etc

Cause

Our DNS was routing requests to geant.net and dante.net to external ROOT servers. This configuration was in place for around 2 years, but suddenly the ROOT servers missed the records for our domains. 

Bind, running on the Consul servers, was setup to route as following: 

• domain service.ha.geant.net was forwarded to local consul on port 8600
• win.dante.org.uk and geant.local were forwarded to Windows servers geantdc01.geant.local and geantdc02.geant.local
• everything else was going to Internet ROOT servers

Resolution.

Bind was changed to forward as following:

• domain service.ha.geant.net remains the forward to local consul on port 8600
• geant.local remains the forward to geantdc01.geant.local and geantdc02.geant.local
• win.dante.org.uk is now forwarded to am-prd-dc01.win.dante.org.uk and am-prd-dc02.win.dante.org.uk
• geant.net, geant.org and dante.net are now forwarded to infoblox grid members 62.40.104.250, 62.40.116.122, 62.40.116.114
• everything else remains on Internet ROOT servers

Future Mitigation.

We are still investigating the issue.