search by tags

for the user

adventures into the land of the command line

rabbitmq network partitions

This happens when part of the cluster loses connectivity to the rest, which turns out, is pretty common, especially if you run your cluster in the cloud.

When a network partition occurs, a log message is written to the RabbitMQ node log:

=ERROR REPORT==== 15-Jul-2017::18:02:30 ===
Mnesia([email protected]): ** ERROR ** mnesia_event got
    {inconsistent_database, running_partitioned_network, [email protected]}

In a two node cluster, for example, both nodes will think they are each the master:

server0> $ rabbitmqctl cluster_status
[{nodes,
     [{disc,
          ['[email protected]',
           '[email protected]']}]},
 {running_nodes,['[email protected]']},
 {cluster_name,
     >},
 {partitions,
     [{'[email protected]',
          ['[email protected]']}]},
 {alarms,[{'[email protected]',[]}]}]

server1> $ rabbitmqctl cluster_status
[{nodes,
     [{disc,
          ['[email protected]',
           '[email protected]']}]},
 {running_nodes,['[email protected]']},
 {cluster_name,
     >},
 {partitions,
     [{'[email protected]',
          ['[email protected]']}]},
 {alarms,[{'[email protected]',[]}]}]

So how do we fix it?

In /etc/rabbitmq/rabbitmq.config in the “rabbit” vhost, you can set one of three options:

{cluster_partition_handling, pause_minority}
or
{cluster_partition_handling, autoheal}
or
{cluster_partition_handling, ignore}

Which is which and which one should I pick?

ignore

Your network really is reliable.
All your nodes are in a rack, connected with a switch, and that switch is also the route to the outside world.
You don't want to run any risk of any of your cluster shutting down if any other part of it fails (or you have a two node cluster).

pause_minority

Your network is maybe less reliable.
You have clustered across 3 AZs in EC2, and you assume that only one AZ will fail at once.
In that scenario you want the remaining two AZs to continue working and the nodes from the failed AZ to rejoin automatically and without fuss when the AZ comes back.

autoheal

Your network may not be reliable.
You are more concerned with continuity of service than with data integrity.
You may have a two node cluster.

The RabbitMQ tile uses the pause_minority option for handling cluster partitions by default. This ensures data integrity by pausing the partition of the cluster in the minority, and resumes it with the data from the majority partition. You must maintain more than two nodes. If there is a partition when you only have two nodes, both nodes immediately pause.

You can also choose the autoheal option in the RabbitMQ Policy tab. In this mode, if a partition occurs, RabbitMQ automatically decides on a winning partition, and restarts all nodes that are not in the winning partition. This option allows you to continue to receive connections to both parts of partitions.

So pause_minority when you have 3+ nodes and autoheal when you have just 2 nodes.

To simulate a node leaving the cluster due to network issues, you can turn of the network interface on one node:

$ ifdown eth0; sleep 3; ifup eth0;

Or tell iptables to drop packets fron one of the nodes:

10.10.0.2> $ iptables -A OUTPUT -d 10.10.0.1 -j DROP

Then to restore:

10.10.0.2> $ iptables -D OUTPUT -d 10.10.0.1 -j DROP