Message Queue vs. Web Services? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.

---

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed 9 years ago.

Improve this question

Under what conditions would one favor apps talking via a message queue instead of via web services (I just mean XML开发者_StackOverflow中文版 or JSON or YAML or whatever over HTTP here, not any particular type)?

I have to talk between two apps on a local network. One will be a web app and have to request commands on another app (running on different hardware). The requests are things like creating users, moving files around, and creating directories. Under what conditions would I prefer XML Web Services (or straight TCP or something) to using a Message queue?

The web app is Ruby on Rails, but I think the question is broader than that.

---

When you use a web service you have a client and a server:

1. If the server fails the client must take responsibility to handle the error.
2. When the server is working again the client is responsible of resending it.
3. If the server gives a response to the call and the client fails the operation is lost.
4. You don't have contention, that is: if million of clients call a web service on one server in a second, most probably your server will go down.
5. You can expect an immediate response from the server, but you can handle asynchronous calls too.

When you use a message queue like RabbitMQ, Beanstalkd, ActiveMQ, IBM MQ Series, Tuxedo you expect different and more fault tolerant results:

1. If the server fails, the queue persist the message (optionally, even if the machine shutdown).
2. When the server is working again, it receives the pending message.
3. If the server gives a response to the call and the client fails, if the client didn't acknowledge the response the message is persisted.
4. You have contention, you can decide how many requests are handled by the server (call it worker instead).
5. You don't expect an immediate synchronous response, but you can implement/simulate synchronous calls.

Message Queues has a lot more features but this is some rule of thumb to decide if you want to handle error conditions yourself or leave them to the message queue.

---

There's been a fair amount of recent research in considering how REST HTTP calls could replace the message queue concept.

If you introduce the concept of a process and a task as a resource, the need for middle messaging layer starts to evaporate.

Ex:

POST /task/name
    - Returns a 202 accepted status immediately
    - Returns a resource url for the created task: /task/name/X
    - Returns a resource url for the started process: /process/Y

GET /process/Y
    - Returns status of ongoing process

A task can have multiple steps for initialization, and a process can return status when polled or POST to a callback URL when complete.

This is dead simple, and becomes quite powerful when you realize that you can now subscribe to an rss/atom feed of all running processes and tasks without any middle layer. Any queuing system is going to require some sort of web front end anyway, and this concept has it built in without another layer of custom code.

Your resources exist until you delete them, which means you can view historical information long after the process and task complete.

You have built in service discovery, even for a task that has multiple steps, without any extra complicated protocols.

GET /task/name
    - returns form with required fields

POST (URL provided form's "action" attribute)

Your service discovery is an HTML form - a universal and human readable format.

The entire flow can be used programmatically or by a human, using universally accepted tools. It's a client driven, and therefore RESTful. Every tool created for the web can drive your business processes. You still have alternate message channels by POSTing asynchronously to a separate array of log servers.

After you consider it for a while, you sit back and start to realize that REST may just eliminate the need for a messaging queue and an ESB altogether.

http://www.infoq.com/presentations/BPM-with-REST

---

Message queues are ideal for requests which may take a long time to process. Requests are queued and can be processed offline without blocking the client. If the client needs to be notified of completion, you can provide a way for the client to periodically check the status of the request.

Message queues also allow you to scale better across time. It improves your ability to handle bursts of heavy activity, because the actual processing can be distributed across time.

Note that message queues and web services are orthogonal concepts, i.e. they are not mutually exclusive. E.g. you can have a XML based web service which acts as an interface to a message queue. I think the distinction your looking for is Message Queues versus Request/Response, the latter is when the request is processed synchronously.

---

Message queues are asynchronous and can retry a number of times if delivery fails. Use a message queue if the requester doesn't need to wait for a response.

The phrase "web services" make me think of synchronous calls to a distributed component over HTTP. Use web services if the requester needs a response back.

---

I think in general, you'd want a web service for a blocking task (this tasks needs to be completed before we execute more code), and a message queue for a non-blocking task (could take quite a while, but we don't need to wait for it).