Act 1 – The Problem

Here be known our Complicating Incident.

Look at that fancy new server just installed in the rack with 8x HDR 200Gb InfiniBand HCAs on it. WOW. That is going to make your data scientist stakeholders very happy. Linux is installed and it’s just dying to consume input.

Need Input - Short Circuit

This particular node needs to connect to NFS and a parallel filesystem (PFS), as well as a head node for workload management. For simplicity’s sake, all HCA NICs are configured as

  • ib0 –
  • ib1 –
  • ib2 –
  • ib3 –
  • ib4 –
  • ib5 –
  • ib6 –
  • ib7 –

It’s ready to rock! Hold on to your horses, not yet. Executed from a peer node on

$ sudo ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.102 ms
64 bytes from icmp_seq=2 ttl=64 time=0.112 ms
64 bytes from icmp_seq=3 ttl=64 time=0.113 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 4120ms
rtt min/avg/max/mdev = 0.102/0.111/0.118/0.010 ms

That looks very good. And now we run into trouble:

$ sudo ping
PING ( 56(84) bytes of data.
From icmp_seq=1 Destination Host Unreachable
From icmp_seq=2 Destination Host Unreachable
From icmp_seq=3 Destination Host Unreachable
--- ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4101ms
pipe 4

Uh oh. ib_read_bw also fails:

$ ib_read_bw -a -R -F
Unexpected CM event bl blka 8
Unable to perform rdma_client function
Unable to init the socket connection

That’s  not too surprising. An ICMP failure is a fairly good indication that RDMA CM will also fail.

What is going on with this hot mess? “ip a” indicates that all of the IPoIB devices are up, connected and configured with an IP address. So, you check the network configs. They look OK. Then you probably don’t trust that and go through a process of checking link lights on the switch and changing out known-good cables for suspect ones. That doesn’t work. Then you complain about this issue on Slack while you’re pulling your hair out and your colleague hips you to “policy-based routing”.

Act 2 – Policy-Based Routing

What is policy-based routing? Policy-based routing is what is going to solve your problem at hand. The problem you are encountering is that the Linux default IP routing policy doesn’t understand how to route IP packets to multiple interfaces that are configured on the same IPv4 subnet.

It seems like Linux should understand this, but it doesn’t.

For reference, here is a document that describes this in detail:

TL;DR. To summarize: the node must be configured with routing tables, routes and rules per NIC that tell the IP stack which NIC to send packets through according to a received packet’s destination address.

Tables, Rules and Routes

For simplicity’s sake, this example will configure two NICs with policy-based routing. For all Linux:

# append these definitions to the existing config
100 t1
101 t2

This configuration adds human-readable names for routing tables indexed by a number. Granted, the names I used are fairly obtuse; your configuration may use whatever name seems fit. The names and numbers are arbitrary but must be orthogonal within rt_tables.

Next, configure the routes. This instruction is for RHEL ifconfig systems. See the Addendum for Netplan configuration. For each IPoIB interface (aka NIC), create a file in /etc/sysconfig/network-scripts/route-ib0 (or ib1, whatever. Use the NIC name).

/etc/sysconfig/network-scripts/route-ib0: dev ib0 scope link src table t1

And /etc/sysconfig/network-scripts/route-ib1: dev ib1 scope link src table t2

This creates a separate route for ib0 and ib1 according to a packet’s destination address.

Note that the route files are incompatible with NetworkManager, so ifcfg-ib0 etc. need an “NM_CONTROLLED=no” directive to make this work.

We are almost there. An additional rule file must be added per NIC:


table t1 from


table t2 from

This creates the routing tables per NIC IP address. The per-NIC tables, rules and routes are now configured. The last remaining step is to configure ARP.

ARP Configuration

ARP must be adjusted to require packet response upon the originating interface. Take the following template and write it to /etc/sysctl.d/50-multihoming-arp.conf:

#ib0 ARP config
net.ipv4.conf.ib0.rp_filter = 1
net.ipv4.conf.ib0.arp_filter = 1
net.ipv4.conf.ib0.arp_announce = 2
net.ipv4.conf.ib0.arp_ignore = 2
#ib1 ARP config
net.ipv4.conf.ib1.rp_filter = 1
net.ipv4.conf.ib1.arp_filter = 1
net.ipv4.conf.ib1.arp_announce = 2
net.ipv4.conf.ib1.arp_ignore = 2

Add directives for any additional interfaces.

Act 3 – To Test, or Not to Test

With your due diligence towards these instructions, rebooting the node will present the configured NICs  accessible via IP and RDMA CM.

Hopefully the node in question has a KVM interface in case it is remote and the IP config got totally whacked. Maybe that should have been stated earlier?

First-blush diagnosis is performed via “ip route show table main”. There should be an entry for each IPoIB device that was configured with an associated route and rules script.

$ ip route show table main
default via dev eno1 dev eno1 proto kernel scope link src dev ib0 proto kernel scope link src dev ib1 proto kernel scope link src dev eno1 scope link metric 1002 dev ib0 scope link metric 1004 dev ib1 scope link metric 1005

And individual tables:

$ ip route show table t1 dev ib0 scope link src
$ ip route show table t2 dev ib1 scope link src

If that looks reasonable, then test with ping from a remote node:

$ sudo ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.149 ms
64 bytes from icmp_seq=2 ttl=64 time=0.147 ms
64 bytes from icmp_seq=3 ttl=64 time=0.098 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2028ms
rtt min/avg/max/mdev = 0.098/0.131/0.149/0.025 ms

Test each of the node’s interfaces to verify connectivity.

Now test RDMA CM. Start a server on the node:

$ ib_read_bw -a -R -F

And launch the client on the remote peer:

$ ib_read_bw -a -R -F

ib_read_bw should output maximum bandwidth at ~10GBps for an EDR fabric. HDR performance will depend upon whether your nodes have the HCAs in PCIe 3 x16 or PCIe 4 x16 slots (PCIe 3 x16 is limited to ~126Gbps). Any connection failures would indicate that the policy-based routing is misconfigured. Or maybe there is a firewall running — turn it off if your server is in a secure environment.

Addendum: Netplan

A netplan configuration that approximates the previous configuration steps would resemble:

      dhcp4: true
      dhcp4: false
      addresses: []
       - to:
         table: 100
         scope: link
       - from:
         table: 100
         priority: 100
      optional: true
      dhcp4: false
      addresses: []
       - to:
         table: 101
         scope: link
       - from:
         table: 101
         priority: 101
      optional: true
  version: 2


Policy-based routing is key to driving traffic to multiple Linux NICs that cooperate in a single IPv4 subnet, be they Ethernet or InfiniBand IPoIB. The Linux routing tables must be configured to understand how to communicate over multiple NICs on the same subnet. This configuration is a detailed process and must be tested to ensure that connectivity exists between all cluster NICs.

Hope this helps. Avail me of your HPC travels at