Managing Dead Letter Queues(DLQ) in RabbitMQ with Java Spring Boot

Introduction

Rabbit MQ is an open source message broker mainly used as a middle-ware(or broker) to communicate among the micro-services. Micro-service which produces or sends a message to the broker is termed as “Producer” whereas the service which receives the message is known as “Consumer”.

One can imagine the flow of communication as: Producer —> Message Broker —> Consumer.

Message Broker comprises two key parts:
1) Exchange
2) Queue

Messages by the producer are sent to the exchange first and then it is routed to the queue with the help of a routing key. This phenomenon is known as “Binding”. And, then the consumer consumes the message from the queue.

Sometimes, consumers fail to consume and acknowledge the message from the queue due to various factors, such as invalid format, processing failure or routing failure. In such cases, Dead Letter Queues (DLQ) are employed to capture these messages, allowing for analysis, troubleshooting, and potential reprocessing without disrupting the normal flow of messages.

For each queue, we can have a separate dead letter queue and exchange. It helps in maintaining system integrity, identifying issues and facilitating proper error handling and resolution in distributed systems:

1) Error Handling and Retry Mechanism: This ensures the system does not lose critical messages due to processing failure.

2) Visibility and Debugging: It allows the developer or admin to easily identify and inspect messages that failed to be processed.

3) Reprocessing and Recovery: Messages in the DQ can be manually or automatically reprocessed, allowing the system to recover from processing failure and data consistency.

4) Auditing and Compliance: Storing failed messages provides the records of all the messages that are accounted for, thus contributing to overall integrity and transparency of the system.

We can create Dead Letter Exchanges and Queues over the RabbitMQ UI Management and also through the code in Java with Spring Boot using “Spring for Rabbit MQ” dependency. Below is the Maven dependency code of RabbitMQ:

dependencies {
 implementation 'org.springframework.boot:spring-boot-starter-amqp'
 testImplementation 'org.springframework.amqp:spring-rabbit-test'
}  

Below is DLQ configuration implementation in java with spring boot:
1) Define a bean of Dead Letter Exchange(dlx)
2) Define a bean of Dead Letter Queue(dlq)
3) Bind the dlx and dlq
4) Associate the above dlx to the queue you are using .

a) Parameter variables of queue, exchange and routing keys: We define the instance variable and assign the values using “@Value” annotation by the help of key-value pair defined in application properties file.

b) Values to the above fields assigned in the application properties file in key-value format.

c) Configuration of dead letter exchange and queue in configuration file.

d) The above code creates the bean for the queue for which we are setting up the DLQ. With the help of “with Argument” method we can specify the exchange name and its routing key to establish the connection between the main Queue and dead letter exchange.

e) The code creates the dead letter queue and its bean.

f) Next, create the dead letter exchange and its bean.

g) Finally, bind the dead letter exchange and dead letter queue with the routing key.

And, remember to use the below library for different classes, interface and annotations of rabbitmq used in the config class:

import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

Failed messages will be routed to the dead letter queue. Additionally, one can also specify the expiration time while sending the message and set the TTL(time-to-live) while creating the bean of the queue, thus after that expiration time if the message is not consumed, it will be routed to the dead letter queue. You could use the help of below code to implement it.

1)While sending message:

rabbitTemplate.convertAndSend(MAIN_QUEUE, message, messagePostProcessor -> {
messagePostProcessor.getMessageProperties().setExpiration(“60000”); // TTL in milliseconds (e.g., 60 seconds)
return messagePostProcessor;
});

2) While creating bean of the queue:

@Bean
public Queue mainQueue() {
return QueueBuilder.durable(MAIN_QUEUE)
.ttl(60000) // TTL in milliseconds (e.g., 60 seconds)
.build();
}

At the consumer level, one needs to handle the failed processing messages. One of the ways can be as below:

I hope you enjoyed the article on Dead Letter Queue in Rabbit MQ.

Thank you! ?