Differences

This shows you the differences between two versions of the page.

--- apache_sitedown [2015/04/24 07:22] – created luke7858
+++ apache_sitedown [2024/05/23 07:26] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
+Note: I have written a python script to check a system and report back on any potential Out-Of-Memory issues:
+<sxh bash>
+curl -s https://raw.githubusercontent.com/LukeShirnia/out-of-memory-investigation.py/master/oom.py | python
+</sxh>
+\\
+\\
+== Step 1 ==
+did apache hit max clients?
+CentOS/RHL
+<sxh bash>
+grep -i maxc /var/log/httpd/error_log
+</sxh>
+Ubuntu/Debian
+<sxh bash>
+grep -i maxc /var/log/apache2/error_log
+</sxh>
+Note: Could use error.log rather than error_log
+== Step 2 ==
+Did the server run out of memory and kill processes that killed a vital service?
+CentOS/RHL
+<sxh bash>
+egrep -i 'killed process|invoked oom' /var/log/messages
+</sxh>
+Ubuntu/Debian
+<sxh bash>
+egrep -i 'killed process|invoked oom' /var/log/syslog
+</sxh>
+\\
+Summarise date and times the server ran out of memory (CentOS/RHEL):
+<sxh bash>
+zgrep -i 'killed process\|invoked oom' /var/log/messages* | awk '/Killed process/ {print $1, $2, $3}' | awk -F: '{print $2}' | uniq -c | sort -k2,3r
+</sxh>
+<sxh bash>
+zgrep -i 'killed process\|invoked oom' /var/log/messages* | awk '/Killed process/ {print $1, $2, $11}' | awk -F: '{print $2}' | uniq -c | sort -k2,3
+</sxh>
+Example output:
+<sxh bash>
+Apr 25 (driveclient)
+Apr 27 (driveclient)
+Apr 29 (driveclient)
+May 12 (driveclient)
+May 2 (driveclient)
+May 22 (driveclient)
+May 5 (driveclient)
+May 9 (driveclient)
+</sxh>
+\\
+Debian/Ubuntu
+<sxh bash>
+zgrep -i 'killed process\|invoked oom' /var/log/syslog* | awk '/Killed process/ {print $1, $2, $3}' | awk -F: '{print $2}' | uniq -c
+</sxh>
+\\
+\\
+==Step 3 ==
+Check to see if the website received traffic during the period of 'down time'
 Investigating a 'site down' issue.
+Best command to see check all log is:
+<sxh bash>
+cat /var/log/httpd/*access*log* | grep "27/Aug/2015:04:[3..5][0..9]" | awk {' print $1 '} | sort | uniq -c | sort -rn
+</sxh>
+<sxh bash>
+cat /var/log/httpd/*access*log* | grep "22/Aug/2015:04:[3-5]" | awk {' print $1 '} | sort | uniq -c | sort -n
+</sxh>
+Checking old log files (compresses) \\
+<sxh bash>
+zcat /var/log/httpd/*access*log*.gz | grep "22/Aug/2015:04:[3..5][0..9]" | awk {' print $1 '} | sort | uniq -c | sort -n
+</sxh>
+\\
 <sxh bash>
 grep 24/Apr/2015:02:[3-6][0-9]:* /var/log/httpd/access_log | awk '{print "IPaddress", $1,"Time",$4, $9}'
 </sxh>
 This will grep for:
 date 24/April/2015 02:30 - 02:60
@@ Line 12: / Line 83: @@
 IPaddress x.x.x.x Time 24/Apr/2015:02:36:11 200
 IPaddress x.x.x.x Time 24/Apr/2015:02:36:12 200
+</sxh>
+\\
+\\
+The following can summaries all of the IP addresses hitting the access logs during a certain date/time
+<sxh bash>
+grep "12/Jun/2015:12" access_log | awk {'print $1'} | sort | uniq -c | sort -n | tail -n 20
+</sxh>
+E.g:
+<sxh bash>
+x.x.x.1
+x.x.x.15
+x.x.x.158
+x.x.x.19
+x.x.x.201
+x.x.x.122
+</sxh>
+\\
+\\
+See what a specific IP address is doing (access logs) during a specific date/time
+<sxh bash>
+grep "12/Jun/2015:12" access_log | grep x.x.x.11 | tail -n 20
+</sxh>
+\\
+<sxh bash>
+x.x.x.11 - - [12/Jun/2015:12:43:05 +0500] "POST /newsite/stats/recording.php HTTP/1.1" 200 407 "-" "-"
+x.x.x.11 - - [12/Jun/2015:12:42:50 +0500] "POST /newsite/api2/index.php HTTP/1.1" 200 1238 "-" "Mozilla/5.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0 REQUESTED FROM: http://www.example.com/profolio/index.php?tabs=2&section=add_property REF: http://www.example.com/profolio/index.php?tabs=2&section=add_property API KEY:
+x.x.x.11 - - [12/Jun/2015:12:43:05 +0500] "POST /newsite/api/index.php HTTP/1.1" 200 3697 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5"
+x.x.x.11 - - [12/Jun/2015:12:43:06 +0500] "POST /newsite/api2/index.php HTTP/1.1" 200 766 "-" "Mozilla/5.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0 REQUESTED FROM: http://www.example.com/profolio/includes/add_property_single.php? REF: http://www.example.com/profolio
+</sxh>
+\\
+Following shows what IPs connected to the server during a time period, this can be used to see if the server was down at the time listed.
+<sxh bash>
+grep "16/Jun/2015:08:[0-2][0-9]:*" /var/log/httpd/access_log | awk '{print $1}' | sort | uniq -c | sort -nr
+</sxh>
+\\
+The following command shows all IP addresses accessing all domains on the server for a specific date/time:
+<sxh bash>
+cat /var/www/vhosts/*/logs/*log | grep "20/Jul/2015:19" | awk {' print $1 '} | sort | uniq -c | sort -n
 </sxh>