So if you can’t wait for the weekend just watch one of our famous “Atomic Bomberman” matches held every evening in our office:

Thanks Rob! I’m gonna play a little bit around :)

Welcome Michael!

They’re escaping single quotes what is not neccessary and not allowed! This for example causes the Zend_Service_Delicious component of the ZF to fail and return an empty array if you try to get all bookmarks of a specific user whose bookmarks contain single quotes.

So please check the specification at json.org and fix your API!

Why? When Boris was writing the client angel script he somehow named it “dropr”. As we neither got better suggestions nor had any other idea we just decided for this name. Actually the name is a little bit fun because all those stupidR startupRs. But it’s nice and somehow our framework drops message into queues :)

If you’re using google to search for it, it’s already the first result.

So have a look at https://www.dropr.org/. We’re just writing a little installation manual about setting up the pre-release of our framework. We’ll keep you up-to-date on this blog, so stay tuned.

Some time has elapsed since we wrote our draft for a message queue system written in and for PHP. Now it’s time to give you guys an update and working beta-code.

Boris and me have spent some nights to get a prototype running. First, we tried to work on tcp stream connections with php for the transport layer but that was buggy and slow. Our second approach was to use http uploads with curl (thanks to Jan Kneschke for the tip!). For the storage we decided to try a simple filesystem storage and it seems our predictions were right:

• With using the filesystem as client and server storage you won’t need to setup any database system
• The filesystem actually IS a (restricted) DBMS
• If you have files, you are able to get all the I/O work done by fast C-code. On the client side, the data is loaded via curl (we only pass the filenames to it) and gets uploaded to the server, where the php handles the uploads and makes tempfiles out of it => the server storage can use a simple rename() operation again to put it into it’s store
• Filesystem operations like moving and deleting are ATOMIC - a file is existent or it isn’t
• You can encode all metadata like priority in the filename - there’s enough space in it
• You can read huge dirs fast with “scandir()” function, it’s sorted by filename
• You’re still able to implement your own storages and transports, though, because everything is written within an abstract class model.

Some notes:

• With this setup we’re actually able to send 200-250 messages per second (benchmarked on a damn slow machine).
• we’ve set up a rc.d script for starting and stopping the client queue daemon
• the client queue daemon has an angel-process that cares about php breaking down
• the message queue daemon gets notified by the client via systemv inter-process-communication so there is no sleep() overhead within
• the system is also in production in a non-critical area of our software and we’re testing it for a few weeks.

You can find the project homepage at https://www.dropr.org/.

Please checkout the sources “svn co https://www.dropr.org/svn/trunk/” and have a look at it. We’re sorry that we have no real documentation at this stage of the project but it will follow soon hopefully.

Stay tuned, we’d like to get out a alpha/beta as PEAR (or maybe debian) package soon.

A tip: Get a free website with integrated content management at Jimdo.com - no programming knowledge required!

Because the lack of implementations of a messaging system in PHP we at Jimdo are going to implement such a solution on our own. Some time ago I wrote another post and asked if somebody has experiences with such techniques or already designed such a beast.

You’ll find a Draft for a php-based messaging system below. We’d be glad if we get some comments from the readers. Because we’re heavily using open source we want to give something back to the community and make the message queue system open source. And, yes, if someone is planning something like this or already knows a solution, please let us know, too. We don’t wanna reinvent the wheel!

But now let’s come to the details ;-)

1. Problem
2. Existing Solutions
3. Implementation Draft

1. The problem

When you’re building an infrastructure that is distributed all over the internet, you’ll come to a point where you can’t rely on synchronous remote calls that - for example - synchronize data on 2 servers:

a) You don’t have any failover system that resends messages if something went wrong (network outages, software failures)
b) Messages are processed over time and you have no control if something goes overloaded by too many requests

Even if you don’t have to send messages all over the Internet there are enough points of failures where something can go wrong. You want a reliable and durable system that fails gracefully and ensures the delivery of messages even after temporary outages of any machine within the system.

2. Existing solutions

In the Java world there is a standard called Java Message Service (JMS). ActiveMQ from the Apache project implements such a solution.

In theory you can connect to a JMS service with PHP using a PECL Module called SAM (Simple Asynchronous Messaging for PHP). While it sounds easy to install and use, I ran into so many problems that I gave up. First, you need to register at IBM and download “XMS - The IBM Message Service API”. After you’ve downloaded the 60MB (!) package you can compile SAM. Add the extension in our php.ini and BAM: “PHP could not startup”. At this point I got really frustrated and gave up. I wanna have something more lightweight.

But in the PHP world I didn’t find anything similar.

3. Our approach to a PHP-based message queueing system

Vocabulary:

Client: The peer that invokes a message
Client-queue: The local client que that will run as a daemon and sends the messages to the servers
Server: The peer that receives messages

3.1 Features

3.1.1 Main Architecture

The system is peer-to-peer based with no central server (and therefore no single point of failure). Every (sending/client) node has its own local queue.

The local queue will have an “angel process” that restarts the daemon if it dies.

3.1.2 Asynchronous / Durability

Messaging is never direct. On the client side (the system that wants to send some message to another peer) there will be a local queue that sends out messages to its peers (called servers). There will be an API available for your applications to put messages into the queue, but the actual message transport will be handled by a separate process.

On the server-side there will be another process that writes the received messages into its queue. This queue will be queried by YOUR application using the provided API. See code examples below.

We will have to take care that operations are atomic. If the client gets a “true”, it can be sure that the data is saved to the local queue. And if the client que gets a “true” from the transport layer it can be sure the message is in the server’s queue.

As you may have noticed already, the solution never executes your code. It’s always queried by the application that makes use of the messaging service.

(HINT: Anyways we can implement a mode on the server side that calls your code. This would be an interface with only one method like “handleMessage(MQ_Message $message)” you’ll have to implement. If your method returns true, the message will be deleted from the server-side queue, otherwise it will be kept for later handling.)

3.1.3 Scalability

Because there is no single instance that could get overloaded scalability issues shouldn’t occur. The system scales somehow horizontal.

3.1.4 Content Independence

The message content is arbitrary. You could even send movies ;-)

3.1.5 Modularity

The underlying database is switchable. We will do our POC-implementation with PDO_sqlite but switching to other PDO-drivers, maybe even text-files, DBM etc. should be possible.

Also, the transport layer should be replaceable (XML-RPC, REST, SOAP, spread, <put here what you want>).

3.1.6 Easy to setup and use

The default configuration should work out of the box with an SQLite backend. See the examples.

3.1.7 “Job done” messages

If a message is completed and processed at the “server”, it can notify the original client that the operation was successful. We’ll have to think about possible solutions because this certainly implies a 2-way-communication.

One approach could be that the sender specifies if he want’s to get an “OK - message processed” from the peer. The client que will poll the server after submitting the message and checks if the message has been set to “PROCESSED” state.

3.1.8 Priority handling

Of course, you can actually set the priority of a message so you can be sure that more important messages will by favored by the que.

3.1.9 Multicasting

It’s possible to send a message to several servers with a single command. The queue will set the state to PROCESSED if all servers notified it that the message was processed.

3.2 Code Examples

3.2.1 Client-side

Starting the queueing daemon should look like this:

/opt/MQ/daemon start

FIXME: Configuration of the Daemon

Sending a message would look like this:

$queueDriver = MQ_QueueDriver::factory('PDO', 'dsn ....');
$transport = MQ_Transport::factory('REST');
$serverOne = new MQ_Peer('http://server1.com/path/to/server');
$serverTwo = new MQ_Peer('http://server2.com/path/to/server');
$queue = new MQ_Client($queueDriver, $transport);
$queue->send($serverOne, 'this is a test message');
$queue->multicast(array($serverOne, $serverTwo), 'this is a test multicast message');

// Send with feedback
$message = $queue->send($serverOne, 'this is a test message', true);

// Wait for the feedback
while ($message->getStatus() != MQ_PROCESSED) {
   sleep(1);
} 

echo "Now it's processed by the peer, yieehaaa!";

3.2.2 Server-side (get something out of the queue)

Because your main application is not involved into the messaging process the only thing it has to do is to query the queue:

$queueDriver = MQ_QueueDriver::factory('PDO', 'dsn ....');
$queue = new MQ_Server($queueDriver);

// Getting the messages with an infinite-loop iterator
foreach ($queue as $message) {
    echo "Sender was: " . $message->getSender();
    echo "Message is: " . $message->getMessage();

    // Mark the Message as processed, the client que checks if we were processed.
    // So the origin has the opportunity to check if we could process the message properly.
    $message->setProcessed(true);
}

3.3 Requirements

- PHP 5.2+
- certainly the Zend-Framework

4. The Name?

No name until we have a working prototype - haha.

5. Next steps

- evaluate feedback
- setup a trac, fill tickets
- begin to code

We appreciate your feedback, wishlist, improvements, criticism in the comments as the development will be started soon.

Sönke with assistance from Markus and Christian.

We at Jimdo are looking for reinforcements! You’ll be able to work in Hamburg, one of the most beautiful cities in Germany and work with a young team in an enthusiastic athmosphere. Honestly, I haven’t had this much fun for years before I started working here :-)

We’re looking for PHP experts, system administrators, web designers and Javascript gods (or a human vessel with equal powers).

What are you still waiting for? Go to the official job page, right now, don’t hesitate:
http://www.jimdo.com/jobs.php

Note: If you’re a resident of the European Union, you’re free to work in any EU country you like! You don’t speak German? Not a problem as long as you speak and understand English well enough. You’ll miss out on some of the jokes for a while, but you can still see me lose each and every single Bomberman tournament held in this office, which should be satisfactory enough. Oh, we also got excellent coffee! Still not interested? What’s wrong with you? :-)

Jimdo allows to define tags for your website. You can separate the tags with whitespace, but it’s also possible to use commas, like this:

tag1, tag2, tag3

Chinese users naturally are way more used to using UTF-8 characters than us westerners, and, lo and behold, UTF-8 has its own special comma character with integrated whitespace, that is quite frequently used by chinese users:

科学，思考，心情

As we’re using good ol’ regular expressions to split up the tag strings into single tags, one might think, “no problem, I’ll just add another character to the regex pattern”, like so:

$tags = preg_split("/[\s,;:，]+/", $input, null, PREG_SPLIT_NO_EMPTY);

And heureka, it works! Or does it? Nope. As UTF-8 works with multiple bytes per character and preg_split, like so many other current PHP functions, thinks of one byte = one character, you may encounter strange side-effects. Here’s an example using the above pattern on a random string with some German umlauts:


Splitting up 'Bääh Blöök Dübel', becomes:
Array
(
[0] => Bääh
[1] => Blöök
[2] => D�
[3] => bel
)


What to do? Simple: Add the unicode modifier, “u”, to the pattern:

$tags = preg_split(”/[\s,;:，]+/u”, $input, null, PREG_SPLIT_NO_EMPTY);

Now preg_split correctly recognizes multibyte characters and yields the expected results:


Splitting up 'Bääh Blöök Dübel', becomes:
Array
(
[0] => Bääh
[1] => Blöök
[2] => Dübel
)


Another lesson learned.