The Way to Programming
The Way to Programming
One easy method I use to find out which site is being slammed (if that is the case), and I’ll assume your log files are in
/var/log/httpd
cd into your web log directly, and do an
ls -ltra
This will list everything in by modification time, and you can see which log is being written to continually.
(drop the -r if you would rather have the newest at top. I prefer bottom)
You can also do a
watch “ls -ltra”
to watch the list with a default 2 second update.
Then tail that puppy with a -f ollow to watch what and how the culprit his hitting it –
tail web-log-to-watch-access.log -f
Apache should be
2.2.16-6+squeeze12 and not older.
Do you use mod_jk at all??? It can be replaced by mod_proxy in apache 2.2
Do you think you have time to convert apache to apache-worker and mod_fcgid and php-cgi?
The log rotation did not seem to help. I have been slowly tracking things down and have managed to identify a couple of attacks that were occurring that were repeatedly hitting the same valid files on the server, files I would normally expect to see in the logs. It took a while but after sifting through a couple million rows of logs I was able to identify the attacks and stop them with a combination of .htaccess rules and turning off some features of the blogs. It’s tough to identify the good from the bad, when in your situation it is normal to have a couple hundred thousand requests from the same IP’s daily because they are coming from schools and businesses. So it’s normal to see 50,000 post requests to the login script etc.
The files I found that were specifically being targeted were
1. wp-login.php – someone was trying to brute force attack a user. I denied access to this page via .htaccess ruleon the blogs that we don’t allow users to register on, and added an environment check to allow our work IP Address to access it.
2. xmlrpc.php – I don’t know what they were trying to do with this but I deleted this file from the blogs
3. I also edited the wp-config.php and disabled wp-cron (I manually update the blogs and plugins, and we don’t schedule posts, so I think that should be fine.) I don’t know if this was a problem on my machine but I read about it being used as an exploit so figured I would disable it now.
I am still getting fake posts to the comments on the blogs on a regular basis, even though they are not getting through because we require moderation of all comments and have some anti-spam stuff in place, we are still getting too many, I am unsure how to stop this one since we do want people to comment on our posts… but that is a completely different question.
So even though wordpress was not allowing any of the attacks to actually do anything, no comments were getting through, people were still beating on the server in short waves causing the cpu spikes. Currently things are sitting at a 15 minute average of between .10 and .15 which is more along the lines of what I expect of the system under the expected current load, and I no longer see any apache threads reaching above 10% cpu or if they do, it’s only for a second and then they are back to 2-5% usually.
Attacks targed insecure wordpress. If you have no wordpress just disregard them. If you indeed use wordpress – rename those admin files to long random name and force over SSL.
If you have hard time containing CPU usage – evaluate if mod_fcgid + apache-worker is a viable option for you (examples in fcgid documentation talk about using it with php)
swap has nothing to do with the kind of media it’s on (hdd or sdd)
granted, being on an ssd would perform faster, it doesn’t dictate the requirement
with only 8GB, would recommend having something there; even 4gb is sufficient
if you don’t have any disk space available for a small partition, you can create a file and use that
Linux Add a Swap File – Howto
http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
as far as the high load goes, you should check those running scripts
either there is a problem with the script or someone is trying to do something malicious with the site causing that cpu activity for those cgi processes. i would also look at your web server (apache?) logs and look at the activity
Sign in to your account