多网卡多ip在同一个网段

发表于 | 分类于

问题：当某些特殊情况下，需要在同一个主机的多块网卡上配置同一个网段的地址，但linux系统默认系统配置是不支持这种配置的，会出现各种奇怪的问题，比如对于arp广播，多个网卡都会响应，对端交换机一般会记住返回最快的那个mac地址，但对于每次请求返回的快慢是不能保证一致的，所以会导致arp表不停变化；再比如对于icmp，默认返回只走第一条匹配的路由，即对于所有ip的请求都由一个网卡返回。

解决方案

1
2
3
4
5
6
7
8
9
10
11
1.只处理目的ip属于当前网卡的arp请求，其他arp请求直接丢弃
  sysctl -w net.ipv4.conf.all.arp_ignore=1
2.选择最优的arp源IP
  sysctl -w net.ipv4.conf.all.arp_announce=2 
3.只处理反向路由指向接收网卡的报文
  sysctl -w net.ipv4.conf.all.rp_filter=1
4.增加策略路由，假设eth0/1在同一个网段
  ip route add 0.0.0.0/0 dev eth0 table 1
  ip route add 0.0.0.0/0 dev eth1 table 2
  ip rule add from eth0-ip table 1
  ip rule add from eth1-ip table 2

参数解释

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
参见：https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
arp_ignore - INTEGER
	Define different modes for sending replies in response to
	received ARP requests that resolve local target IP addresses:
	0 - (default): reply for any local target IP address, configured
	on any interface
	1 - reply only if the target IP address is local address
	configured on the incoming interface
	2 - reply only if the target IP address is local address
	configured on the incoming interface and both with the
	sender's IP address are part from same subnet on this interface
	3 - do not reply for local addresses configured with scope host,
	only resolutions for global and link addresses are replied
	4-7 - reserved
	8 - do not reply for all local addresses

	The max value from conf/{all,interface}/arp_ignore is used
	when ARP request is received on the {interface}
arp_announce - INTEGER
	Define different restriction levels for announcing the local
	source IP address from IP packets in ARP requests sent on
	interface:
	0 - (default) Use any local address, configured on any interface
	1 - Try to avoid local addresses that are not in the target's
	subnet for this interface. This mode is useful when target
	hosts reachable via this interface require the source IP
	address in ARP requests to be part of their logical network
	configured on the receiving interface. When we generate the
	request we will check all our subnets that include the
	target IP and will preserve the source address if it is from
	such subnet. If there is no such subnet we select source
	address according to the rules for level 2.
	2 - Always use the best local address for this target.
	In this mode we ignore the source address in the IP packet
	and try to select local address that we prefer for talks with
	the target host. Such local address is selected by looking
	for primary IP addresses on all our subnets on the outgoing
	interface that include the target IP address. If no suitable
	local address is found we select the first local address
	we have on the outgoing interface or on all other interfaces,
	with the hope we will receive reply for our request and
	even sometimes no matter the source IP address we announce.

	The max value from conf/{all,interface}/arp_announce is used.

	Increasing the restriction level gives more chance for
	receiving answer from the resolved target while decreasing
	the level announces more valid sender's information.
rp_filter - INTEGER
	0 - No source validation.
	1 - Strict mode as defined in RFC3704 Strict Reverse Path
	    Each incoming packet is tested against the FIB and if the interface
	    is not the best reverse path the packet check will fail.
	    By default failed packets are discarded.
	2 - Loose mode as defined in RFC3704 Loose Reverse Path
	    Each incoming packet's source address is also tested against the FIB
	    and if the source address is not reachable via any interface
	    the packet check will fail.

	Current recommended practice in RFC3704 is to enable strict mode
	to prevent IP spoofing from DDos attacks. If using asymmetric routing
	or other complicated routing, then loose mode is recommended.

	The max value from conf/{all,interface}/rp_filter is used
	when doing source validation on the {interface}.

	Default value is 0. Note that some distributions enable it
	in startup scripts.

原理解释

linux收到arp请求后的处理流程如下：

1
2
3
4
5
6
代码参考：https://github.com/torvalds/linux/blob/master/net/ipv4/arp.c
arp_rcv-->arp_process-->ip_route_input_noref-->ip_route_input_slow-->
fib_validate_source(rp_filter参数生效)-->arp_ignore(arp_ignore参数生效)-->
arp_filter(arp_filter参数生效)
1.由代码可知，arp模块的处理逻辑是与ip路由紧密相关的。
2.icmp及tcp/udp的处理都需要经过ip路由的入向和出向匹配的，此处就需要rp_filter参数(入向)及策略路由(出向)的支持。

参考文档

  1. serverfault高分答案
  2. 推荐阅读：TCP/IP协议栈在Linux内核中的运行时序分析