Server Outage Incident 05-08 December 2019

Summary

Our hosting providers servers went down for 72 hours, causing our website to go offline. We apologize for this inconvenience and we are taking steps to prevent this from happening again.

What happened?

On Thursday 05 December 2019 our hosting provider (GoDaddy) experienced technical issues impacting several of their servers that caused the MySQL databases to fail. Unfortunately, one of the servers affected by this issue was ours, which is why thesimscatalog.com and sims3createaworld.com were offline for almost 72 hours.

Our severs have alarms in place that monitor the website 24/7 and immediately alert our admins of any issues. Whenever something goes wrong, we work to resolve it as quickly as possible to reduce any inconvenience it may cause to our users. However, on this occasion the issue was on our hosting provider’s end, and there was nothing our admins could do but wait for GoDaddy support to fix it.

On 07 December, the issue with the physical server was resolved, but our admins quickly noticed the outage had damaged our databases beyond repair. Thankfully, since the previous site outage in July 2018, we had since implemented daily backup snapshots of the site so that it could be restored. A decision was made to restore the site to a backup taken on 04 December before the outage.

Due to the size of our websites (over 900 GB of data), backups can take a while to complete. This means that unfortunately some downloads were lost in the process. However, do not worry as we took manual backups and we will be restoring them back on to the website by end of 08 December.

How will you stop this from happening again?

Although in this instance the issue was with our hosting provider and outside of our control, we will be taking steps to mitigate the damage to our site if this happens again. Our admins have made a decision to move our servers to AWS, while still keeping our existing server as a backup. This will ensure that in the event of another outage, we will redirect traffic to our backup server.

Conclusion

We sincerely apologize to all our users for the inconvenience this incident caused. We pay thousands of dollars every year for fast servers to provide a better experience to our users, but on this occasion we were let down by our hosting providers. We will therefore be investing in having two servers to prevent this from happening again.