Scaling Issues and Server Delays
We have had some issues in the last 24 hours with the servers. We had some very high volume publishers sign up who are sending great volume. This caused some teething problems which are now solved. Some users may have found that the platform was slow or unresponsive during this time which I apologize for.
The main issue was the Apache setup which is used to proxy connections across to node.js
The standard settings are setup for high volume traffic with a max of 150 concurrent connections.
The settings have also changed since I last worked on tuning an Apache server. Whereas the old settings were limited by MaxConnections this is now set in /etc/apache2/mods-enabled/mpm_prefork.conf as the MaxRequestWorkers. To set this above 256 I believe you still have to set the ServerLimit to the same value.
Once this was updated it removed the bottleneck and we are now handling 500-1000 concurrent connections without significant load on the server.
If truth be told we weren’t as ready as we thought we were to start receiving high volume traffic. Testing should have been better to identify the bottlenecks in the system before we ran in to them. I would like to offer my apologies to all our early adopters and give reassurance that the issues should now be solved.