Problems with credit card payments in EPP

On April 22nd, one of our internal API servers was migrated to a new IPv6 only host. After monitoring and seeing no problems, the other API server was also migrated today on April 25th. Subsequently two more API servers where added in our second datacenter.

We got information about problems with credit card payments yesterday and today but it wasn't until today after all API servers had been migrated that we discovered the problem.

The endpoint at our payment provider for credit cards is IPv4 only so after the migration our API servers could no longer contact our payment provider. Customers using prepayments (majority of EPP users) where not affected, since those do not require contacting the payment provider.

The problem was fixed at 15:10 today and we will be looking at ways to add monitoring and documentation to prevent an issue like this occurring again.