Update: if you care about our email inboxes, we set the Honeybadger check interval back to a minute and removed the email notifications for our admins. We are currently working with Atlassian (Statuspage owner) to have https://status.rc24.xyz/ when the server goes down.
Posted Feb 19, 2021 - 18:21 CST
Resolved
The server has been running under a few hiccups today. We believe we have everything fixed.
While working on development of the Open Shop Channel, we noticed the server being unresponsive. After close investigation of the server analytics, we came to the theory of our mail server struggling because some IRC network servers kept trying to connect to our mail server as a proxy. While doing that, one of the admins restarted the server, while that might not be the best thing to do, it caused some more downtime.
It's hard to know for sure if it got fixed, but we edited the Apache server config to try to prevent it from accepting those sort of connections. And in the process, the configuration broke the Check Mii Out Channel giving HTTP 403 errors. We fixed that and it is back up and running now.
In other news, we are trying out a service called Honeybadger that constantly checks if the server is up. We set it to check every minute, but since we were spammed with emails today about the server going down and up, it now checks every 5 minutes.
We apologize for the downtime, it's hard to figure out what's going on when things like this happen, and to be honest, whenever something happens, there's a lot of commotion and drama between the admins trying to fix it. We are doing what we can to try to stop these issues for happening.
Thank you for using RiiConnect24 and reading this (long message)!
Posted Feb 19, 2021 - 18:12 CST
This incident affected: Wii Mail (Server) and Services (Check Mii Out Channel/Mii Contest Channel).