First, we’d like to annouce that our staff has been reinforced by Boris Erdmann! We’re really proud of having another geek within our development team. Among other things he is the founder and maintainer of the german OpenID provider xlogon.
Update: The project is now called dropr!
Some time has elapsed since we wrote our draft for a message queue system written in and for PHP. Now it’s time to give you guys an update and working beta-code.
Boris and me have spent some nights to get a prototype running. First, we tried to work on tcp stream connections with php for the transport layer but that was buggy and slow. Our second approach was to use http uploads with curl (thanks to Jan Kneschke for the tip!). For the storage we decided to try a simple filesystem storage and it seems our predictions were right:
- With using the filesystem as client and server storage you won’t need to setup any database system
- The filesystem actually IS a (restricted) DBMS
- If you have files, you are able to get all the I/O work done by fast C-code. On the client side, the data is loaded via curl (we only pass the filenames to it) and gets uploaded to the server, where the php handles the uploads and makes tempfiles out of it => the server storage can use a simple rename() operation again to put it into it’s store
- Filesystem operations like moving and deleting are ATOMIC - a file is existent or it isn’t
- You can encode all metadata like priority in the filename - there’s enough space in it
- You can read huge dirs fast with “scandir()” function, it’s sorted by filename
- You’re still able to implement your own storages and transports, though, because everything is written within an abstract class model.
Some notes:
- With this setup we’re actually able to send 200-250 messages per second (benchmarked on a damn slow machine).
- we’ve set up a rc.d script for starting and stopping the client queue daemon
- the client queue daemon has an angel-process that cares about php breaking down
- the message queue daemon gets notified by the client via systemv inter-process-communication so there is no sleep() overhead within
- the system is also in production in a non-critical area of our software and we’re testing it for a few weeks.
You can find the project homepage at https://www.dropr.org/.
Please checkout the sources “svn co https://www.dropr.org/svn/trunk/” and have a look at it. We’re sorry that we have no real documentation at this stage of the project but it will follow soon hopefully.
Stay tuned, we’d like to get out a alpha/beta as PEAR (or maybe debian) package soon.
A tip: Get a free website with integrated content management at Jimdo.com - no programming knowledge required!
December 7th, 2007 at 00:21
You need a name ? what about :”Job Queue”
December 9th, 2007 at 16:42
Call it PHQ. “Job Queue” - unfortunately - requires Zend Platform ;)
December 9th, 2007 at 21:25
PHP Messaging System. Just simply call it to PMS. :D
December 11th, 2007 at 11:20
Good to see a PHP Message Queue, if it is well designed, maybe it could be integrated into the Zend Framework.
Instead of using rename() after upload, use move_uploaded_file().
And did you try a database storage first?
SQLite is a very small and fast file-based database without. the need to setup a server.
scandir() will not be so fast anymore when there are a few thousand small files in a directory, depending on the os and file system.
December 11th, 2007 at 14:15
Hi Rene,
first, thanks for reviewing the source code.
1) Zend Framwork: It’s unlikely that they will include some message queue because it’s a competor to their own $$-solution called Zend Job Queue.
2) We use rename() in the Server Storage because it’s not bound to the fileupload transport. You could also use another transport with filesystem storage. We’re checking the file(s) with is_uploaded_file() first so we guess it’s ok. We have to document the source code better, though ;)
3) We didn’t test sqlite yet. But because of the class structure it’s possible without problems in theory. In practice, you’ll maybe have some problems with locking of the sqlite database (daemon checks for messages and client wants to put some messages into the queue at the same time).
We had no real performance impact with 5000+ files in the spool directory on an ext3 filesystem. I guess with using XFS you shouldn’t run into problems.
Again, thanks for the feedback!
November 23rd, 2008 at 12:25
I saw a performance degrade when getting into 1000’s of messages in single dir…
I think it was related to file_exists() which is called to determine if filename is unique — i solved by generating a guid as part of filename, and skipped the file test.
April 6th, 2010 at 02:31
PHP Messaging System. Just simply call it to PMS.
June 26th, 2010 at 12:47
People deserve wealthy life time and home loans or just sba loan can make it better. Because people’s freedom depends on money state.