Soenke Ruempler | May 24, 2007 at 11:21 · Filed under Server-Administration, PHP, Development
While the last days I had an e-mail discussion with Theo Schlossnagle from OmniTI regarding the network messaging toolkit Spread that he mentions in his great book “Scalable Internet Architectures“.
The thread is about Spread and if it’s possible to build a durable message queing system. The main issue is that Spread cares about the reliable network delivery of messages but if one “listener” of a group goes down (host or daemon death) it’ll never get the message because (for spread) it’s not joined to the group anymore.
Theo posted our e-mail discussion in his Blog of the “Scalable Internet Architectures” book (read on!).
The result is that you either have to build your own queing system on top of spread that cares about the durability or use something “enterprise” like ApacheMQ what’d be maybe overkill.
Maybe you have other approaches for a durable network messaging system in PHP? I’d like to hear your voices!
Soenke Ruempler | May 14, 2007 at 20:43 · Filed under Server-Administration, PHP, Development
One big issue of the PHP error handling is that there’s no built-in way to catch fatal errors with an user-defined error handler. So I thought a little bit about it and maybe you have better approaches or solutions …
The short goal is to send the error via e-mail to the developer(s). As we are security-aware, we’re logging errors and do not display them to the world. (Hint: that should be your default on every production machine!)
With the error_log directive we have 3 possibilties: syslog, sapi and a common logfile. Sapi in my case is the mod_php Apache module that logs into the error_log by default so we can classify sapi and a normal logfile to be the same in this case.
Now I see two possibilities - file watching and syslogger.
First I looked into the syslogger if there is any possibility to send e-mails. This would be nice …
The original bsd syslogger used by the most Linux distributions can’t. And there’s a reason, yeah ;) The syslogger logs many events, even critical system ones (emergency etc). If the system is unstable and something really goes wrong the mailing subsystem shouldn’t be invoked. Yes that’s plausible to me. [1]
There are some people that built named pipes with the syslogger [2]. But that got too weird to me ;) Another issue is that the error_log-syslog behavior (facility, priority) of PHP seems to be not really documented. So I didn’t look deeper into syslog… did u make other / better experiences with syslog? I’d appreciate feedback because I didn’t got deeper into the syslog approach …
So lets get back. The more pragmatic approach is an external log file analyzier. The common suspects are:
Logwatch and Logcheck are not real-time, but we want real-time.
So there are Logsurfer and Swatch left. Because Swatch is available as Debian package (and I like standardized systems) I’ve chosen it. Swatch has a fine and clean but flexible configuration approach and is really easy to setup. The heart is this config-file:
watchfor /(PHP.*error:.*?)$/i
mail addresses=root
threshold=on
threshold track_by=$1,type=limit,count=1,seconds=10
This means in spoken words:
- Look for any line matching this regular expression (Fatal, Parse, Recoverable errors)
- Send this line to root
- Limit repeating lines to one per ten second (and use the $1 substitution of the matched line, this is the error message itself without the date)
Number 3 is the most interesting point because one can tune it and you don’t get spammed if the Fatal Error occures many times.
Now we start the daemon:
# swatch --daemon -c /root/php_fatal.conf --tail-file=/var/log/apache2/error.log
To test it raise a fatal error and wait for the mail:
From: root
To: root@server01.intern.northclick.de
Subject: Message from Swatch
[Tue May 15 13:24:12 2007] [error] [client 192.168.2.16] PHP Parse error: syntax error, unexpected T_CONSTANT_ENCAPSED_STRING, expecting ‘)’ in /home/soenke/tests/fatal.php on line 1
You want to put the swatch daemon into your rc.local or some other startup file so it’s started when the machine boots.
This technique seems the most clean to me.
What do you use? What other approaches do you have?
References:
[1] comp.os.linux.networking - How to send a syslog message by E-mail?
[2] http://www.softpanorama.org/Logs/Syslog/pipes_in_syslog.shtml
