Warning documentation for the implication of lack of rule on primary NIC

[tonyswu]

Recently we had an issue with our EC2 instance where some packets were intermittently dropped. The instance had two NICs with IPs on the same subnet, and the issue was traced (with the help of AWS support) to the lack of ip rule for primary NIC. Because the two NICs were on the same subnets under rare conditions the returning packets for NIC1 were sent out via NIC2 which were dropped.

While the behavior is documented (

amazon-ec2-net-utils/lib/lib.sh

Line 711 in b1a7565

# We only create rules for secondary interfaces so

), we would appreciate it if you could also document the implications of the missing rule in the scenario where the instance has one or more secondary NICs with IPs that share the same subnet as the primary NIC.

Thanks for your considerations.

[joeysk2012]

Can you provide reproduction steps or what the configuration for the NICs are?
Two Nics on the same subnet is a common use case.

amazon-ec2-net-utils/lib/lib.sh

Lines 410 to 411 in b1a7565

	local -i metric=$((metric_base+100*network_card+device_number))
	local -i tableid=$((rule_base+100*network_card+device_number))

The routing order should be based on device index + network card. Meaning, the primary NIC should always be lower priority than the secondary NIC.