2 BGP Session dropping randomly same time

michmoor

@nbhatti Do both BGP sessions drop at the same time? If so this strongly indicates a problem within your domain.
Also, start checking the links connected for any errors/drops. Do you have a link thats bouncing?
Check system health of pfsense - CPU spikes or memory utilization that occurs during the outage?

Is the topology: pfsense----Layer 2 cloud---ISP
If so, is spanning-tree involved? Anything in the switch logs to indicate STP failure or flapping?

NBhatti

@michmoor This is TNSR not pfSense, but yes the topology is more or less the same. TNSR --> Mikrotik Switch --> Cisco Switch --> ISP.

Mikrotik running SwitchOS and has very limited logging capabilities but none of them are showing any single Frame drops, Mac errors or anything like that. The link is direct 10G SFP+ DAC Cable from 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) network card. RSTP/STP is also disabled on Mikrotik and cisco both.

Both session should not go down/reset same time. Initially I thought this is hardware, so I changed, but the new h/w is doing the same thing. I enabled option debug neighbor-events and

route dynamic manager
    debug events
    log syslog
exit

but nothing should in syslog or FRR logs. I feel like this is a local issue but can't seem to figure out what to look for. Maybe BGP or connectivity debug may help in this case? BGP says session timeout which means it's not able to talk to the peer, but both doing so the same time? That l can not figure out, yet.

michmoor

@nbhatti Are the switches connected to each other with a single link or multiple links?

NBhatti

@michmoor Mikrotik is connected to Cisco via single Multimode SFP cable.

3750-01#show interfaces Gi1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0024.1406.9183 (bia 0024.1406.9183)
  Description: TO_MIKROTIK_CCS_P3
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 53/255, rxload 18/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseBX10-U SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:01, output hang never
  Last clearing of "show interface" counters 2w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 42197
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 71404000 bits/sec, 21307 packets/sec
  5 minute output rate 211588000 bits/sec, 30550 packets/sec
     35882878680 packets input, 17777933845598 bytes, 0 no buffer
     Received 318434790 broadcasts (148384674 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 148384674 multicast, 0 pause input
     0 input packets with dribble condition detected
     52252747898 packets output, 47370079035222 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

It's 1G connectivity, but not showing more than few hundred Mbps.

michmoor

@nbhatti said in 2 BGP Session dropping randomly same time:

Total output drops: 42197

I see drops. Granted your counters havent been cleared so it would be a good idea to clear them
Another option is to connect the ISP handoff directly to yoru TNSR. If its stable then we know the problem is between the Cisco and Mikrotick

NBhatti

@michmoor yes there are drops but no CRC no input or anything Layer 2. Connecting ISP to the system is a good idea. I would have to arrange SFP card for that since ISP is on Fiber. Thanks for the idea. Let's see if anyone from Netgate can come up as well to see if some debug can be turned on to see if DPDK is having some fun here maybe :)

Thanks.

mleighton

One suggestion is to run a packet capture outside of TNSR on TCP/179 to figure out if any connectivity problems can be seen there. That is most likely a quicker step than obtaining an SFP card to test without the switches in the path.

Derelict

@michmoor said in 2 BGP Session dropping randomly same time:

I see drops.

A "drop" is counted any time a packet is sent to the drop node for any reason, even perfectly normal ones. This could be as simple as being blocked by an ACL, having no route to host, etc.

If you are working in the <= 10Gbit/sec space and are using reasonably-current physical hardware and Intel NICs, you are probably not experiencing an inability of tnsr to forward whatever traffic you are asking it to forward.

michmoor

@derelict The drops are from the Cisco side from the picture provided. More specifically output drops which are unrelated to any ACL, no route or even Intel NICs.

A Former User

If its stable then we know the problem is between
the Cisco and Mikrotick

If a MikroTik router is in "game" and there is also the version
7.4 installed, it could be coming from that devices, the BGP problems I mean. On some devices, I don´t know them exactly all, there will be only the eth ports able to use together with BGP and not the SFP(+) ports, but if
you will use them instead of that "warning" you will be running in trouble with BGP.

NBhatti

@dobby_ Mikrotik is in the path but it’s running SwitchOS 2.3. That’s completely different than RouterOS. Unfortunately SwitchOS Has very limited logging capabilities can’t even see syslog. It does however show interface stats and other stuff on the GUI only but during all those BGP drops there was not a single packet loss, CRC or anything like that on the switch.
It happens very randomly something twice in a day and at times once a week. Can’t really establish any pattern. Funky thing is that both sessions drop at the same time.
I even disables one season to see if anything related but even though single session also drops. I am going to put the ISP cables directly in the NIC and see how it behaves.

Thanks.

NBhatti

@derelict said in 2 BGP Session dropping randomly same time:

@michmoor said in 2 BGP Session dropping randomly same time:

I see drops.

If you are working in the <= 10Gbit/sec space and are using reasonably-current physical hardware and Intel NICs, you are probably not experiencing an inability of tnsr to forward whatever traffic you are asking it to forward.

Even though i am using less then 10Gbit/sec but that should (worst case) cause queues or outbound packet drops? Why the BGP session drop?

Derelict

@nbhatti As has been said, we don't know. For what you are describing to happen there would need to be practically zero traffic passing, to both BGP peers, at the same time, for long enough to trigger the hold timer expiration. It doesn't sound like that is the case from what you have stated. Some occasional packet loss will not cause two TCP sessions to stop passing traffic at the same time and not recover.

I would packet capture the BGP sessions to the peers (TCP port 179) and try to capture the event. Then load it up into wireshark and see what happened to the session(s).

This would best be done at from place in the topography like a switch mirror port mirroring the traffic of the port connected to the tnsr node.