Server outage

Front Page Forums Development Server outage

This topic contains 10 replies, has 2 voices, and was last updated by  JSE Admin 1 year, 2 months ago.

Viewing 11 posts - 1 through 11 (of 11 total)
  • Author
    Posts
  • #529 Reply

    Peter

    Hey guys, you might already know and I know its quite early in the morning over there in the UK but the stats and maybe the mining servers are down.

    https://snag.gy/Qrz7M2.jpg

    Getting a 503 server error on a lot of the JS as well as not being able to log into the platform.

    Noticed it was down approximately 2.5 hours ago from posting this message

    #530 Reply

    Steve

    yep have been getting:
    <!DOCTYPE HTML PUBLIC “-//IETF//DTD HTML 2.0//EN”>
    <html><head>
    <title>503 Service Unavailable</title>
    </head><body>
    <h1>Service Unavailable</h1>
    <p>The server is temporarily unable to service your
    request due to maintenance downtime or capacity
    problems. Please try again later.</p>
    <hr>
    <address>Apache/
    on the JS for a couple of hours now – and login not possible

    #532 Reply

    Dave Mallett

    HI Guys,

    thanks for getting in touch. Small server issue this morning but all back up and running now.

    Thanks again.

    #533 Reply

    Peter

    Confirmed we’re all back up and running again.

    #535 Reply

    James

    Just to give some more details really, the Apache log files grew too big and caused it to crash. My own fault, I should have seen it coming. Seems like a lot of the scaling problems we have had are related to Apache so I’m hoping to move the web server across to nginx ASAP

    #537 Reply

    Gemino

    nginx also has scaling problems. Keep an eye on those log files on bother servers.

    #539 Reply

    James

    Hey Gemino, thanks for the heads up, maybe it is better the devil you know. I had chance to calm down a little bit and turned off the access log on Apache which should help a little bit. Tomorrow we are going to be moving the servers about and trying to distribute the load better. Still in two minds about whether to try and switch it across to Nginx or stick with Apache. Probably we are changing too much already so will stick with Apache for the time being.

    #547 Reply

    Gemino

    You must truly optimize any server to suit your needs, but for now all servers require heavy optimization to even handle any load of traffic for a long period of time. Especially heavy loads.

    Good luck with this project.

    #548 Reply

    Peter

    I suspect another AWS instance or two behind the load balancer will help.

    Appears that you’re down again or your rolling out the changes.

    #553 Reply

    JSE Admin
    Keymaster

    Yep we had an issue with the CPU this time. 6500 concurrent connections proving too much to handle. We are learning fast but not fast enough. Tomorrow we will be rolling out the update so the servers will be down around 1pm GMT. We should be able to separate the publisher demand from the platform then which will be a big help, or at least we can break multiple machines at the same time.

    #562 Reply

    JSE Admin
    Keymaster

    The servers still aren’t coping well so we are going to move the update forwards to 9am GMT 4am CST

Viewing 11 posts - 1 through 11 (of 11 total)
Reply To: Server outage
Your information: