While the last days I had an e-mail discussion with Theo Schlossnagle from OmniTI regarding the network messaging toolkit Spread that he mentions in his great book “Scalable Internet Architectures“.
The thread is about Spread and if it’s possible to build a durable message queing system. The main issue is that Spread cares about the reliable network delivery of messages but if one “listener” of a group goes down (host or daemon death) it’ll never get the message because (for spread) it’s not joined to the group anymore.
Theo posted our e-mail discussion in his Blog of the “Scalable Internet Architectures” book (read on!).
The result is that you either have to build your own queing system on top of spread that cares about the durability or use something “enterprise” like ApacheMQ what’d be maybe overkill.
Maybe you have other approaches for a durable network messaging system in PHP? I’d like to hear your voices!
May 24th, 2007 at 15:45
This may not be exactly what you’re looking for, but for durability we need an asyncrhonous system that polls the different nodes for messages. There have to be two “message stores” one, which is local to the node and is just for “pick up”, and another which is the central hub and handles all the SQL.
Messages can be delivered into the queue in a number of ways, and there could be many points of entry, each point must store the message to disk, with a unique id in the filename, this could be something as simple as hostname-timestamp.mid.
Then a sweeper process comes along and processes the designated folders on the queue, and puts each message into the SQL table, and then deletes the message from the filesystem.
A separat message parser/sender script runs and sends messages from the SQL table, which can have it’s own transactions etc.
So, you get durability, and the ability to load balance when the need arises.
I could go into much longer detail, but this is a high level view of a system, that processes a few million messages/day already.
All done in PHP, MySQL. We started out with two servers to handle incoming messages that then needed to be sent out, and now have 3 front end machines that gather the messages locally, and then 2 DB servers across which the system is sending messages from. It’s designed to scale horizontally, so we can take specific queues and put them on dedicated machines, if a client chooses to pay for speed AND durability.
May 25th, 2007 at 15:24
Hi Vid,
thanks for your detailed answer.
I thought of a similar queing solution that you proposed in top of spread as message system. I like spread because I get all this messaging stuff deamonized without having a webserver involved.
June 5th, 2007 at 20:23
You could also do it with php sockets pretty easy, it all depends on the final goal
August 9th, 2007 at 15:23
Hi All,
since the pecl spread (1.1) module doesn’t works well (even if I patch & hack it), may I ask you does anybody has a working PHP Client (any kind… a script using sockets for an example) for Spread? I’m trying to create one but If you know a good working PHP Spread Client please drop a mail to me. Thank you!
August 9th, 2007 at 15:38
Hi dH,
an API using sockets is not recommended because you can’t use the underlying C api then. You had to implement the whole protocol, I guess that is hardcore ;)
The pecl module is simply not usable. We’re going to build our own que system based upon sqlite and xmlrpc. I’ll publish some more details soon.
October 15th, 2007 at 17:30
Hi,
I simply switched to ActiveMQ since there is support for the STOMP protocol and it’s pretty easy to use with any kind of script language. The scalability is not so easy like with Spread and there is other issues with ActiveMQ (5.0 is still Beta!) so if anybody has a good client OR a good description about the Spread protocol pls let me know! :)
January 5th, 2010 at 10:00
Very perfect topic about this post. The very good pre written essays and the chance to buy essay about this good topic is presented by essay writing service.
February 17th, 2010 at 18:46
How could you do the hot topic referring to this post I chew over? The buy dissertation service would employ professionals as you for the dissertation thesis doing. So you can have a good opportunity to join the team of masters.
April 7th, 2010 at 09:13
i think an open source application , VLC player can also do the some network communication. and since its open source checking it out will helo you lots :)
-Andy
April 7th, 2010 at 09:14
sorry.. my bad.. I just checked it out and VLC is no longer opensource. :(
-Andy
July 2nd, 2010 at 10:31
if you want to be rich and famous click on casino en ligne
July 2nd, 2010 at 18:51
One can learn just about this topic but I advice to choose the term paper writing service and buy custom essay papers some times that is possible to purchase pre written essay.
July 20th, 2010 at 13:26
thanks for this interesting topic!
July 28th, 2010 at 13:55
i wouldn’t use any of them!