After extensive research and testing, we learned that our DNS provider performed maintenance this morning too, at the same time as our internal maintenance. During this maintenance they shut-off a DNS server that had been decommissioned 4 weeks ago. In preparation of this, we had changed our DNS settings 4 weeks ago to indicate that this old server should not longer be used.
Unfortunately, it seems that certain servers from these ISP's are not respecting the 1 day validity (TTL / Time-to-live) on these DNS records, and were still contacting this old server, even though we indicated not to use those servers anymore 4 weeks ago.
We worked with our DNS provider and the decommissioned server has been restored for now. We are working on contacting the ISP's that are not following the internet standards for DNS TTL, in order to prevent this issue from returning at a later date.
Posted Sep 10, 2024 - 15:22 CEST
Monitoring
This morning we performed maintenance on our internal infrastructure, in preparation for a future migration. Shortly afterwards our support team started receiving a large amount of error reports related to DNS, and we performed a rollback at 10:38 this morning. Throughout the rest of the day we still received evidence that a small amount of visitors, with specific internet providers, were having problems resolving certain DNS records.
Posted Sep 10, 2024 - 12:47 CEST
This incident affected: Hypernode Platform (Hypernode Servers).