19 KiB
19 KiB
IS-IS Issues FRR vs Junipers, but in One Case Only!
TL;DR:
i have three similar frr/vm systems, kvm on ganeti: ashburn, dallas, and seattle. all peering with junipers among others. ashburn is the only one showing this symptom.
and i have a small cloud of eight frrs running is-is with each other in a libvirt world.
all are a classic configs, bgp between loopbacks discovered through is-is. all mtus 1500 and lsp-mtus of 1497, the default.
replace is-is with statics and it all works. so removed statics and turned off frr leaving the two junipers. they are happy with each other is-is and bgp.
so it is most likely triggered or caused by the frr.
same frr is-is config works in seattle with junos on an m7i, arcos on a fat ufi box, and xe on a 1001. same frr config works in dallas with junos on an mx240 and xr on an asr9001
how to debug?
A Map
+--------------------------+
| |
| r1.iad |
| mx80 |
| |
| lo 192.168.254.11 |
| |
+--------------+-----------+
+--------------------------+ 198.180.150.121|
| | |
| r0.iad | |
| mx240 |198.180.150.120 | LAN
| +--------------------+---------------
| lo 192.168.254.10 | |
| | |
+--------------------------+ |
198.180.150.122|
+-------------+------------+
| |
| r2.iad |
| FRR VM |
| |
| lo 192.168.254.12 |
| |
+--------------------------+
Hostname: r0.iad
Model: mx240
Junos: 20.4R3.8
Hostname: r1.iad
Model: mx80
Junos: 20.4R3.8
Hostname: r2.iad
Debian 12 in a Ganeti VM
FRR 10.1
Without and With FRR IS-IS Running on r2.iad
from r0 ping r1's loopback with FRR not running on r2
--- 192.168.254.11 ping statistics ---
231 packets transmitted, 231 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.814/7.990/568.651/38.563 ms
from r1 ping r0's loopback with FRR not running on r2
--- 192.168.254.10 ping statistics ---
265 packets transmitted, 265 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.654/2.937/101.835/7.161 ms
Start FRR on r2.iad
from r0 ping r1's loopback
--- 192.168.254.11 ping statistics ---
103 packets transmitted, 39 packets received, 62% packet loss
round-trip min/avg/max/stddev = 0.954/16.457/81.277/22.696 ms
and from r1 ping r0's loopback
--- 192.168.254.10 ping statistics ---
113 packets transmitted, 62 packets received, 45% packet loss
round-trip min/avg/max/stddev = 0.827/13.004/68.070/19.908 ms
remember those pings would have come from the pinger's external address/interface, not even their loopback.
pinging the directly facing lan interfaces all work, of course
Config Snippets
r0.iad (r1.iad is Very Much the Same)
isis {
lsp-lifetime 3600;
level 1 disable;
level 2 wide-metrics-only;
interface lo0.0 level 2 metric 1 passive;
interface ge-2/0/0.0 passive;
interface ge-2/0/1.0 passive;
interface ge-2/0/9.0 level 2 metric 2 hold-time 300;
}
...
ge-2/0/9 {
speed 1g;
link-mode full-duplex;
unit 0 {
family inet {
no-redirects;
address 198.180.150.120/25 {
primary;
preferred;
}
}
family iso;
family inet6 {
address 2001:418:8006::120/64 {
primary;
}
}
}
}
lo0 {
description "main loopback";
unit 0 {
family inet {
address 127.0.0.1/32;
address 192.168.254.10/32 {
primary;
}
filter {
input loopback-in;
}
}
family inet6 {
address fc00::10/128;
}
family iso {
address 47.0001.1981.8015.0253.00;
}
}
}
r2.iad config snippets
`/etc/network/interfaces`
auto lo
iface lo inet loopback
auto lo:0
iface lo:0 inet static
address 192.168.254.12/32
iface lo:0 inet6 static
address fc00::12/128
`/etc/frr/frr.conf`
interface ens13
ip router isis p1
ipv6 router isis p1
isis circuit-type level-2-only
vrrp 6
vrrp 6 priority 66
vrrp 6 shutdown
vrrp 6 ip 198.180.150.126
vrrp 6 ipv6 2001:418:8006::126
interface ens14
ip router isis p1
ipv6 router isis p1
isis circuit-type level-2-only
exit
interface lo
ip router isis p1
ipv6 router isis p1
isis circuit-type level-2-only
isis passive
exit
...
router isis p1
net 47.0001.1921.6825.4012.00
lsp-timers gen-interval 30 refresh-interval 900 max-lifetime 3600
exit
hmmmm. this is the only frr instance with vrrp. but removing the vrrp part of the config did not help.
Maybe There is a Physical Issue?
No Errors on r0's LAN Interface
r0 ge-2/0/9 is the LAN over which we have the errors
randy@r0.iad> show interfaces ge-2/0/9
Physical interface: ge-2/0/9, Enabled, Physical link is Up
Interface index: 160, SNMP ifIndex: 513
Description: GigE to LAN sw0.iad.rg.net GigabitEthernet1/0/25
Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online, Speed-negotiation: Disabled,
Auto-MDIX: Enabled
Pad to minimum frame size: Disabled
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 8 supported, 8 maximum usable queues
Current address: 00:26:88:2b:dd:33, Hardware address: 00:26:88:2b:dd:33
Last flapped : 2024-07-07 20:05:00 UTC (6w4d 23:58 ago)
Input rate : 31856 bps (27 pps)
Output rate : 1401520 bps (393 pps)
Active alarms : None
Active defects : None
PCS statistics Seconds
Bit errors 0
Errored blocks 0
Ethernet FEC statistics Errors
FEC Corrected Errors 0
FEC Uncorrected Errors 0
FEC Corrected Errors Rate 0
FEC Uncorrected Errors Rate 0
Interface transmit statistics: Disabled
Comparing r0's Optics
randy@r0.iad> show chassis pic fpc-slot 2 pic-slot 0
FPC slot 2, PIC slot 0 information:
Type 10x 1GE(LAN) Builtin
State Online
Uptime 47 days, 32 minutes, 15 seconds
PIC port information:
Fiber Xcvr vendor Wave- Xcvr JNPR
Port Cable type type Xcvr vendor part number length Firmware Rev
0 GIGE 1000SX MM FINISAR CORP. FTLF8519P2BNL-J1 850 nm 0.0 REV 01
9 GIGE 1000SX MM FINISAR CORP. FTLF8519P2BNL-J1 850 nm 0.0 REV 02
r0 ge-2/0/9 is the LAN over which we have the errors
randy@r0.iad> show interfaces diagnostics optics ge-2/0/9
Physical interface: ge-2/0/9
Laser bias current : 7.282 mA
Laser output power : 0.3570 mW / -4.47 dBm
Module temperature : 18 degrees C / 65 degrees F
Module voltage : 3.2670 V
Laser receiver power : 0.2827 mW / -5.49 dBm
r0 ge-2/0/0 is the NTT fiber
randy@r0.iad> show interfaces diagnostics optics ge-2/0/0
Physical interface: ge-2/0/0
Laser bias current : 6.720 mA
Laser output power : 0.3580 mW / -4.46 dBm
Module temperature : 27 degrees C / 81 degrees F
Module voltage : 3.2720 V
Laser receiver power : 0.0992 mW / -10.03 dBm
The Switch Port Into Which r0 Is Plugged Shows No Errors
sw0.iad#show interfaces GigabitEthernet1/0/25
GigabitEthernet1/0/25 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0018.7328.ea99 (bia 0018.7328.ea99)
Description: r0.iad.rg.net ge-2/0/9
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:23, output 00:00:00, output hang never
Last clearing of "show interface" counters 1y45w
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2836
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 5221000 bits/sec, 1003 packets/sec
30 second output rate 525000 bits/sec, 166 packets/sec
517441244634 packets input, 311995567259457 bytes, 0 no buffer
Received 1162654089 broadcasts (58381912 multicasts)
0 runts, 0 giants, 0 throttles
86 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 58381912 multicast, 0 pause input
0 input packets with dribble condition detected
238702388495 packets output, 174422778944407 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
as r1 and r2 had the same issues with r0, i did not bother to look at the ethers on the ganeti cluster holding the frr vm.
fwiw, i tried ether config changes on r9.iad
randy@r0.iad# set interfaces ge-2/0/9 gigether-options no-flow-control
randy@r0.iad# set interfaces ge-2/0/9 gigether-options no-auto-negotiation
but negligible improvement. from r2 to r0's loopback
--- 192.168.254.10 ping statistics ---
278 packets transmitted, 198 received, +74 errors, 28.777% packet loss, time 279199ms
rtt min/avg/max/mdev = 0.568/7.602/61.807/12.391 ms
so is it is-is?
Down an IS-IS Rabbit Hole
it kinda looks like is-is breakage. so, let's go down the is-is rabbit hole.
it is as if r0 and r1 learn poorly but teach well.
It is Probably Not Configuration
ashburn is pretty much the same frr and juniper is-is configs as are rock solid in seattle r1.sea (m7i) to r3.sea (frr in vm), and dallas r1.dfw (mx240) to r3.dfw (frr in vm).
e.g. dallas ping from r1 (mx240) to r3 (frr) loopback.
--- 148.28.7.4 ping statistics ---
881 packets transmitted, 881 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.259/0.565/28.096/1.405 ms
So What Does IS-IS Think?
r0.iad, the mx240, thinks
randy@r0.iad> show isis database detail
IS-IS level 1 link-state database:
IS-IS level 2 link-state database:
r2.iad.rg.net.00-00 Sequence: 0x9c1, Checksum: 0x89fb, Lifetime: 1128 secs
IP prefix: 192.168.42.0/24 Metric: 10 Internal Up
IP prefix: 192.168.254.12/32 Metric: 10 Internal Up
IP prefix: 198.180.150.0/25 Metric: 10 Internal Up
V6 prefix: 2001:418:8006::/64 Metric: 10 Internal Up
V6 prefix: fc00::12/128 Metric: 10 Internal Up
r2.iad.rg.net.02-00 Sequence: 0x943, Checksum: 0xc76f, Lifetime: 1184 secs
IS neighbor: r2.iad.rg.net.00 Metric: 0
IS neighbor: r0.iad.00 Metric: 0
IS neighbor: r1.iad.00 Metric: 0
r0.iad.00-00 Sequence: 0xbd6a, Checksum: 0x29bc, Lifetime: 3557 secs
IS neighbor: r2.iad.rg.net.02 Metric: 2
IP prefix: 129.250.10.156/30 Metric: 10 Internal Up
IP prefix: 192.168.254.10/32 Metric: 1 Internal Up
IP prefix: 198.180.150.0/25 Metric: 2 Internal Up
V6 prefix: 2001:418:8000:5000::18/126 Metric: 10 Internal Up
V6 prefix: 2001:418:8006::/64 Metric: 2 Internal Up
V6 prefix: fc00::10/128 Metric: 1 Internal Up
r1.iad.00-00 Sequence: 0x2175, Checksum: 0xd8c7, Lifetime: 3589 secs
IS neighbor: r2.iad.rg.net.02 Metric: 2
IP prefix: 38.88.248.236/31 Metric: 10 Internal Up
IP prefix: 192.168.254.11/32 Metric: 1 Internal Up
IP prefix: 198.180.150.0/25 Metric: 2 Internal Up
V6 prefix: 2001:418:8006::/64 Metric: 2 Internal Up
V6 prefix: 2001:550:2:1::22/127 Metric: 10 Internal Up
V6 prefix: fc00::11/128 Metric: 1 Internal Up
r1.iad.02-00 Sequence: 0x376a, Checksum: 0, Lifetime: 0 secs
and r2.iad, the frr, thinks
r2.iad# show isis database detail
Area p1:
IS-IS Level-1 link-state database:
LSP ID PduLen SeqNumber Chksum Holdtime ATT/P/OL
r2.iad.rg.net.00-00 * 137 0x000008eb 0x4014 1184 1/0/0
Protocols Supported: IPv4, IPv6
Area Address: 47.0001
Hostname: r2.iad.rg.net
TE Router ID: 192.168.254.12
Router Capability: 192.168.254.12 , D:0, S:0
IPv4 Interface Address: 192.168.254.12
Extended IP Reachability: 198.180.150.0/25 (Metric: 10)
Extended IP Reachability: 192.168.42.0/24 (Metric: 10)
Extended IP Reachability: 192.168.254.12/32 (Metric: 10)
IPv6 Reachability: 2001:418:8006::/64 (Metric: 10)
IPv6 Reachability: fc00::12/128 (Metric: 10)
1 LSPs
IS-IS Level-2 link-state database:
LSP ID PduLen SeqNumber Chksum Holdtime ATT/P/OL
r2.iad.rg.net.00-00 * 150 0x000009c6 0x88b0 1178 0/0/0
Protocols Supported: IPv4, IPv6
Area Address: 47.0001
Hostname: r2.iad.rg.net
TE Router ID: 192.168.254.12
Router Capability: 192.168.254.12 , D:0, S:0
Extended Reachability: 1921.6825.4012.02 (Metric: 10)
IPv4 Interface Address: 192.168.254.12
Extended IP Reachability: 198.180.150.0/25 (Metric: 10)
Extended IP Reachability: 192.168.42.0/24 (Metric: 10)
Extended IP Reachability: 192.168.254.12/32 (Metric: 10)
IPv6 Reachability: 2001:418:8006::/64 (Metric: 10)
IPv6 Reachability: fc00::12/128 (Metric: 10)
r2.iad.rg.net.02-00 * 27 0x00000947 0xb4db (1185) 0/0/0
r0.iad.00-00 240 0x0000bd71 0x1bc3 3591 0/0/0
Protocols Supported: IPv4, IPv6
Area Address: 47.0001
Hostname: r0.iad
TE Router ID: 192.168.254.10
IPv6 TE Router ID: fc00::10
Router Capability: 192.168.254.10 , D:0, S:0
Extended Reachability: 1921.6825.4012.02 (Metric: 2)
Link Local ID: 141074
Link Remote ID: 0
Local Interface IP Address(es): 198.180.150.120
Local Interface IPv6 Address(es): 2001:418:8006::120
IPv4 Interface Address: 192.168.254.10
Extended IP Reachability: 192.168.254.10/32 (Metric: 1)
Extended IP Reachability: 129.250.10.156/30 (Metric: 10)
Extended IP Reachability: 198.180.150.0/25 (Metric: 2)
IPv6 Reachability: fc00::10/128 (Metric: 1)
IPv6 Reachability: 2001:418:8000:5000::18/126 (Metric: 10)
IPv6 Reachability: 2001:418:8006::/64 (Metric: 2)
r1.iad.00-00 240 0x00002184 0xbad6 3594 0/0/0
Protocols Supported: IPv4, IPv6
Area Address: 47.0001
Hostname: r1.iad
TE Router ID: 192.168.254.11
IPv6 TE Router ID: fc00::11
Router Capability: 192.168.254.11 , D:0, S:0
Extended Reachability: 1921.6825.4012.02 (Metric: 2)
Link Local ID: 333
Link Remote ID: 0
Local Interface IP Address(es): 198.180.150.121
Local Interface IPv6 Address(es): 2001:418:8006::121
IPv4 Interface Address: 192.168.254.11
Extended IP Reachability: 192.168.254.11/32 (Metric: 1)
Extended IP Reachability: 198.180.150.0/25 (Metric: 2)
Extended IP Reachability: 38.88.248.236/31 (Metric: 10)
IPv6 Reachability: fc00::11/128 (Metric: 1)
IPv6 Reachability: 2001:418:8006::/64 (Metric: 2)
IPv6 Reachability: 2001:550:2:1::22/127 (Metric: 10)
r1.iad.02-00 27 0x00003789 0x0000 (56) 0/0/0
5 LSPs
And It Is Not MTU
randy@r0.iad> ping size 1472 do-not-fragment sw0.iad
PING sw0.iad.rg.net (198.180.150.112): 1472 data bytes
1480 bytes from 198.180.150.112: icmp_seq=0 ttl=255 time=2.277 ms
^C
--- sw0.iad.rg.net ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.277/2.277/2.277/0.000 ms
randy@r0.iad> ping size 1473 do-not-fragment sw0.iad
PING sw0.iad.rg.net (198.180.150.112): 1473 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
^C
--- sw0.iad.rg.net ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
same to the vm running frr
randy@r0.iad> ping size 1472 do-not-fragment vm0.iad
PING vm0.iad.rg.net (198.180.150.8): 1472 data bytes
1480 bytes from 198.180.150.8: icmp_seq=0 ttl=64 time=1.624 ms
1480 bytes from 198.180.150.8: icmp_seq=1 ttl=64 time=2.827 ms
^C
randy@r0.iad> ping size 1473 do-not-fragment vm0.iad
PING vm0.iad.rg.net (198.180.150.8): 1473 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
2024.08.26