Calico 網路通訊原理揭秘

2019 年 10 月 3 日
筆記

Calico ??????????????????????? OpenStack ?? Iaas ??????????? VM????????? IP ??????????????????????????????????????????????? BGP ??????????????????????????????????

?????Calico ?????????? veth pair????????????????????????????????????????????????????

1. Calico ??????

?????????????????? Calico ???????????? k8s ??????????????????? A????? A ? IP ???

$ ip a  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00      inet 127.0.0.1/8 scope host lo         valid_lft forever preferred_lft forever  3: eth0@if771: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue state UP      link/ether 66:fb:34:db:c9:b4 brd ff:ff:ff:ff:ff:ff      inet 172.17.8.2/32 scope global eth0         valid_lft forever preferred_lft forever

???????? /32 ??????????? A ???????????

????? A ??????

$ ip route  default via 169.254.1.1 dev eth0  169.254.1.1 dev eth0 scope link

??????????????? 169.254.1.1 ???????????????????????? IP ??????????

??????????????????????????????????????????????????? ARP ????? MAC ?????????????????? MAC ????? MAC????? IP ??????????????????????????? IP ???????????????? MAC ?????? ARP ????

?????????????????????? ip neigh ????????? ARP ???

$ ip neigh  169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee REACHABLE

?? MAC ????? Calico ???????????? ARP?????????????

???????????????????? ARP ??????????????? 169.254.1.1 ?? IP ??????? IP ?????????? MAC
????????????????????????????? IP ???????????? calicba2f87f6bb?MAC ????????? ee:ee:ee:ee:ee:ee???????????????????????? Calico ????????

????????????? Calico ???????? ARP ????? ARP ? ARP ????????? ARP ???????????????? ARP ???????? MAC ?????????????? ARP?Proxy ARP???????

??????????? ARP ????? 8.8.8.8 ? MAC ??????????????????????????? 8.8.8.8 ??????????????????????? MAC ??? PC?????????????? MAC ????? MAC254?

???????Calico ?????????? ARP ????「?????」???????????

????????????????

$ ip addr  ...  771: calicba2f87f6bb@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default      link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 14      inet6 fe80::ecee:eeff:feee:eeee/64 scope link         valid_lft forever preferred_lft forever  ...    $ ip route  ...  172.17.8.2 dev calicba2f87f6bb scope link  ...

???????? ARP?

$ cat /proc/sys/net/ipv4/conf/calicba2f87f6bb/proxy_arp  1

??????????? tcpdump ???????

$ tcpdump -i calicba2f87f6bb -e -nn  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode  listening on calicba2f87f6bb, link-type EN10MB (Ethernet), capture size 262144 bytes      14:27:13.565539 ee:ee:ee:ee:ee:ee > 0a:58:ac:1c:ce:12, ethertype IPv4 (0x0800), length 4191: 10.96.0.1.443 > 172.17.8.2.36180: Flags [P.], seq 403862039:403866164, ack 2023703985, win 990, options [nop,nop,TS val 331780572 ecr 603755526], length 4125  14:27:13.565613 0a:58:ac:1c:ce:12 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 172.17.8.2.36180 > 10.96.0.1.443: Flags [.], ack 4125, win 2465, options [nop,nop,TS val 603758497 ecr 331780572], length 0

???

Calico ?????????? workload ??????????????? 169.254.1.1????????? calixxx ???????????????????????????
?????????? ARP ????? ARP ????? ARP ?????????????????????? ARP ???????

2. ????

????????? Calico ???????????????????????????

?? Host0 ????????

$ ip link add veth0 type veth peer name eth0  $ ip netns add ns0  $ ip link set eth0 netns ns0  $ ip netns exec ns0 ip a add 10.20.1.2/24 dev eth0  $ ip netns exec ns0 ip link set eth0 up  $ ip netns exec ns0 ip route add 169.254.1.1 dev eth0 scope link  $ ip netns exec ns0 ip route add default via 169.254.1.1 dev eth0  $ ip link set veth0 up  $ ip route add 10.20.1.2 dev veth0 scope link  $ ip route add 10.20.1.3 via 192.168.1.16 dev ens192  $ echo 1 > /proc/sys/net/ipv4/conf/veth0/proxy_arp

? Host1 ????????

$ ip link add veth0 type veth peer name eth0  $ ip netns add ns1  $ ip link set eth0 netns ns1  $ ip netns exec ns1 ip a add 10.20.1.3/24 dev eth0  $ ip netns exec ns1 ip link set eth0 up  $ ip netns exec ns1 ip route add 169.254.1.1 dev eth0 scope link  $ ip netns exec ns1 ip route add default via 169.254.1.1 dev eth0  $ ip link set veth0 up  $ ip route add 10.20.1.3 dev veth0 scope link  $ ip route add 10.20.1.2 via 192.168.1.32 dev ens192  $ echo 1 > /proc/sys/net/ipv4/conf/veth0/proxy_arp

????????

# Host0  $ ip netns exec ns1 ping 10.20.1.3  PING 10.20.1.3 (10.20.1.3) 56(84) bytes of data.  64 bytes from 10.20.1.3: icmp_seq=1 ttl=62 time=0.303 ms  64 bytes from 10.20.1.3: icmp_seq=2 ttl=62 time=0.334 ms

?????

??????????

ns0 ??????????????????? IP ?? 169.254.1.1??? ARP ???
Host0 ? veth ??? ARP ???????????? ARP ???????? MAC ????? ns0?
ns0 ??????? ns1 ? IP ????
????? 169.254.1.1 ??????Host ???????????????? 10.20.1.3 via 192.168.1.16 dev ens192 ????? Host1?????? BGP??????? proto ??? BIRD?
? Host1 ?? 10.20.1.3 ?????????????? 10.20.1.3 dev veth0 scope link??????????? veth0 ?????? ns1?
????

????????????????? Calico ?????????????????? ns ????????????? veth ??? ARP ??? ns ??????????????????????????????????????????????????????????????