Implementing At Least Once Delivery With RabbitMQ and Spring’s RabbitTemplate

28.07.2018 Robin De Silva Jayasinghe

Message Delivery Characteristics

First some theory about delivery semantics in messaging systems. When a system wants to communicate via a message broker the developer needs a clear understanding of the delivery semantics. At first one needs to know if and how often a message will be delivered to the broker (and potential consumers):

At most once – the message is delivered at most once but also not at all.
At least once – the message guaranteed to be delivered but can be delivered multiple times.
Exactly once – the message is guaranteed to be delivered exactly once.

The second point that is – at least at first sight – important for message consumers is the ordering of messages. Ordering in a messaging context means that messages arrive at the consumer in the same order as they have been sent by a given producer.

Even if some brokers claim to guarantee “exactly once in order delivery” it’s recommended to not take the order of incoming messages for granted. Because a) things can get messy in distributed systems and b) you might not want to rely on semantics of your current broker too much since you’ll end up with another broker in the future.

AMQP / RabbitMQ Send Semantics

In my current customer project we are using RabbitMQ as a message broker to implement an event-driven archticure. Since we are building our services and frontend apps with Spring Boot we can use the very convenient RabbitTemplate of Spring AMQP.

Note: AMQP is the messaging protocol. One implemantation is RabbitMQ. In this article I might use both terms interchangeably.

Once the correct connection paramters for your RabbitMQ paramters are provided to your applicaiton context your application will automatically connect to the RabbitMQ server. When your code needs to send messages via AMQP you just need to autowire an instance of RabbitTemplate and call the send method like this:

rabbitTemplate.send("your_exchange", "a_routing_key", yourMessageObject);

If the RabbitMQ server is available the call just returns and the message seems to be delivered. However, as you are leaving your JVM’s process you enter the realm of distributed systems. Virtually everything can go wrong now. 😉

So, comparing the above mentioned delivery semantics with the observations from our RabbitMQ usage in Spring Boot, RabbitMQ guarantees at least once delivery if not configured elsewise. Also, there is only one small cornercase where message ordering is guaranteed: https://www.rabbitmq.com/semantics.html#ordering

The Dirty Details

When looking at the send semantics of the RabbitTemplate in Spring Boot we can seperate into two topics:

Passing The Notwork

The first is about reaching the RabbitMQ server at all. So, the network connection can be shaky or not available. Or the host is reachable but the RabbitMQ service on the host is down. In that case the RabbitTemplate will throw an exception to the caller. From this point on the caller code can handle the failed message delivery conciously.

From an API interaction point of view the message can be considered as delivered (to the broker) if no exception is thrown by the send call. So this is quite comparable to an RestTemplate / HttpClient call. However, as you’ll see later things can still go wrong even if the call returned successfully.

Delivery Inside the Message Broker

When you use an AMQP message broker things are a bit different than when interacting via HTTP. This brings us to the second topic: Transferring the message to the broker is only the first part of a longer process. The broker tries to route messages to the queues that are bound to the exchange on which the message is sent.

As mentiond – the broker tries to route the messages. If no queue is bound the message is just rushing through the exchange and is lost in the end. Another case is that if the target exchange does not exist (for whatever reason) the broker returns an HTTP like error code: 404. On the client side the send() method call has already returned. The RabbitMQ template has no chance to notify it’s caller of the failed delivery. The only thing left is writing an error to the application’s log.

There are probably more cases that match this problem space. One can categorize them like “received message but could not deliver it to any queues”. The important point for all problems in this category is that once the RabbitTemplate.send() method returned the client code cannot be sure whether the message has really been delivered.

Choose Your Weapon

For most of your scenarios this behavior is perfectly OK. At least with many of our cases we emit event messages in a “fire and forget” style. The code is written without knowledge of and excpectation towards any consumers. Inconsistencies caused by not delivered messages need to be detected and handled seperatly.

There are however scenarios where you really want to be sure that a given message has a) reached the message broker and b) been routed – the above mentioned at least onće delivery.

In that case the AMQP protocol, RabbitMQ and Spring’s RabbitTemplate offer some (configuration) tools to help you.

Enable and Handle Publisher-Confirms Callbacks

The publisher confirms callback helps the application developer to have his code to be notified when the RabbitMQ server has received a message and has delivered it to the desired exchange. Technical details can be found here: https://www.rabbitmq.com/confirms.html

To enable the publisher confirms callbacks two things must be done:

The property spring.rabbitmq.publisher-confirms must be set to true. Please notice that this property has global effect. Even if there is more than one RabbitTemplate configured.
Since the RabbitMQ server cannot return the cofirmation synchonously some callback code has to be registered so it can be called once the confirm has arrived. The callback can be registered during the Creation of the RabbitTemplate. Instead of using the autoconfigured RabbitTemplate one needs to configure the RabbitTemplate manually and then provide a new Instance of RabbitTemplate.CofirmCallback to the RabbitTemplate::setConfirmCallback method. One of the arguments of the confirm method of that class is the boolean ‘ack’ if it’s ‘true’ the publisher has confirmed that the message has been received. If it’s false something went wrong.

template.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {

    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String cause) {
        if(ack) {
            applicationEventPublisher.publish(new PublishConfirmedEvent(correlationData.getId());
        } else {
            applicationEventPublisher.publish(new PublishNotConfirmedEvent(correlationData.getId(), cause);
        }
    }
});

Handle Return Callbacks

In contrast to the Publisher Confirms Callback the Return Callback is not activated by a configuration property but is directly set on the RabbitTemplate object with setMandatory(true). So this has only effect for the one instance of the RabbitTemplate. If you want to have different behavior to this respect you can configure several RabbitTemplates with different Qualifiers.

By activating the mandatory flag the sent messages indicate that the sender expects the message to be routed to a queue. If the message is not routed the RabbitMQ server needs to return the whole message to the sender with a reply code.

template.setMandatory(true);
template.setReturnCallback(new RabbitTemplate.ReturnCallback() {

    @Override
    public void returnedMessage(Message message, int replyCode, String replyText, String exchange,
        String routingKey) {

        if (replyCode == AMQP.NO_ROUTE) {
            applicationEventPublisher.publish(new NoRouteEvent(message.getMessageId(),
                replyText, replyCode, exchange, routingKey);
        } else if(...) {
            // more code for other cases goes here
        }
    }
});

Message delivery/handling inside the message broker can fail for several reasons. The constants in the com.rabbitmq.client.AMQP interface can give a hint to what can go wrong. Besides obvious codes like NOT_FOUND or ACCESS_REFUSED the NO_ROUTE code is special since other errors are mostly caused by configuration problems. NO_ROUTE is returned is when no queue (with a matching routing key) is bound to the target exchange and the mandatory flag is true. It is the message producer’s way to express it’s wish for ‘at least once’ semantics.

Also check the official docu: https://www.rabbitmq.com/reliability.html#producer

Identifying Message Deliveries in Global Callbacks

Both of the above mentioned techniques have one thing in common. They apply globally or at least to the scope to one RabbitTemplate instance. So it is important to set the messageId of the sent message with some value that can later be used to match the context from which the message has been sent. When you get a publisher confirms callback the initially set messageId is now the correllationData.Id and if you get a return callback you have the initial message at hand where you can directly access the messageId.

If you have different parts of your application communicating via RabbitMQ and want to be able to distinguish the messages from each other it helps to prefix the id with some meaning ful token like orders or notifications.

Handling Callbacks From the RabbitTemplate

It turned out to be a useful pattern to publish meaningful Application Events from the above mentioned callbacks and then listen to these in some other application code. This helps to avoid tangle between your different domains and the potential central RabbitMQ configuration.

Also, you can easily handle the events in an async manner if needed. Be aware that this callbacks will potentially arive very quickly and also do not rely on correct ordering. It helps to implement a state machine for handling async events without fixed order in a reliable manner.

also, to be more resilient against crashes (of the JVM) between delivery attempts you might want to persist your delivery attempts (in a database).