503 Service unavailable

[Claude_T]

This morning, I get
"503
Service Unavailable
The server is temporarily busy, try again later!"
Is it just me?

[choralia]

No Claude, it's not just you. We're under a DDOS attack. Since July 18 we are on a new server, which apparently protects itself from DDOS attacks very well, because, using the administrator access, I can see that the server works fine and the attack just hits the "skin" of the server, not the contents. However, all traffic is temporarily suspended, and a "503" error is triggered for everybody is not an administrator.

Using the administrator access I've blacklisted 30+ offending IP addresses, and full IP ranges, too. However, this seems not yet sufficient to restore normal operations so far. I've opened a ticket to the hosting provider, let's see if they can do something more effective.

Max

[Claude_T]

Thanks for fighting, Max.
Could you privately e-mail me a way to access a server as an admin, if any?
Thanks in advance,
Claude.

[choralia]

Hi Claude, the support staff of the hosting provider found that, probably as a consequence of the DDOS attack, a service was not working correctly. They have restarted that service and everything seems OK now.

On the server that was in use until July 18 we had the possibility to restart services ourselves, which would have helped to more quickly solve this issue. I haven't found this feature on the new server, so I contacted support again and asked whether it is still available and I was unable to find it. I'm awaiting for their reply. I'll then prepare a brief guide addressing admin access and emergency procedures in the case similar issues occur again, and I'll provide it to you through a secure channel.

Max

[choralia]

We've just received another DDOS attack. I've blacklisted entire ranges of offending IP addresses, the situation seems OK now.

Max

[PeterGibson]

Looks as if things have gone downhill again. Another attack within 3 hours of the previous one?

Peter

[choralia]

It's the same DDOS attack indeed. I've just checked that all offending IP addresses are blacklisted, however I guess that the same side effect that occurred some days ago, i.e., a service not working correctly after the attack, is present again. Probably the server has to be rebooted. I've opened a ticket to the hosting provider, awaiting for a reply.

Max

[agarvin]

Hey all, I did notice static content is available and is being served:
https://www.cpdl.org/wiki/images/a/a3/0 ... -score.pdf

That probably bypasses MediaWiki or whatever is generating the 503.

This suggests to me that the error is arising from something in the MediaWiki application: thus a server reboot may be not be needed; rather, something in the application stack can probably be restarted to fix the issue.

May I ask the tech stack is being used here?

Allen Garvin

[Jonas15]

Only the php content gives the error, maybe you should restart the web server behind it.

[Ajhigg]

The wiki seems to be back up for me! Cheers to the administration for working on this!

[choralia]

The situation was precisely the same as it was few days ago: apparently, even if we resolve the DDOS attack by blacklisting the offending IP addresses, a specific service pertaining to the web server hangs, and it must be rebooted. Unfortunately, after the migration to this new server we cannot reboot services ourselves, we have to ask the technical support of the hosting provider, and their reaction has been rather slow this time.

I'll ask the technical support to provide a simple mean for us to reboot that service ourselves, so maybe next time we could more promptly fix the problem.

By the way, these forums are hosted on another server just to prevent that, if the main server is down, we have no way to communicate.

Max