Bridged pfsense stop to pass traffic

slagr

Ah, OK. Easily done.

Is there some reason you're running old versions of pfSense?

I'm conservative about any upgrade, and as I cannot test upgrading process, I don't want to risk.
All I can say i,s that I got the same issues with 2.0-RC3, 2.0-R, 2.0.1
So, I'm not sure upgrading to 2.0.2,2.0.3,2.1 would help.
Didn't find any related info about any improvements for us on redmine. But I might be wrong.
@stephenw10:

Since you're also running Intel NICs you could try the additional tuning parameter suggested:
hw.em.num_queues=1
I'm running Intel NICs and have never needed any tuning though. :-\

When it stops passing traffic you still have access to the box?

Steve

Intel NIC is using for pfsync interface.
Unfortunately broadcom NICs are integrated, and I cannot replace them with Intel.
I cannot log in from outside (WAN), but I can log in from inside the net (LAN).

stephenw10

I agree, it should be working fine under older versions. I just wondered in case you were running something custom.

Broadcom NICs are normally quite reliable (second only to Intel). Has the setup been working and just recently started to be unreliable?

Please define what you mean by 'stops all traffic'. You say you can log in from the LAN and you previously said you can SSH in via the pfsync interface. Can you normally connect from the WAN then?

Steve

slagr

@stephenw10:

I agree, it should be working fine under older versions. I just wondered in case you were running something custom.

Broadcom NICs are normally quite reliable (second only to Intel). Has the setup been working and just recently started to be unreliable?

Please define what you mean by 'stops all traffic'.

Incorrect wording. By 'stop all traffic' I meant, that I could not access anything behind the pfsense. We have a few /24 networks behind it, no NAT involved.

@stephenw10:

You say you can log in from the LAN and you previously said you can SSH in via the pfsync interface. Can you normally connect from the WAN then?
Steve

That's the point. I can connect via LAN,pfsync interfaces, but not via WAN.
Another thing I forgot to mention (sorry!), is that I can see WAN interface "Errors In" counter increasing slowly. All other interfaces error counters are 0.

stephenw10

So the errors appear on WAN whilst it's still operating normally?

You didn't say if this behavior has just recently started but I assume it has.

Do the boxes share a common upstream switch? Could that be failing?

Steve

slagr

@stephenw10:

So the errors appear on WAN whilst it's still operating normally?

That's correct, errors appears while box is still operational.
@stephenw10:

You didn't say if this behavior has just recently started but I assume it has.

Do the boxes share a common upstream switch? Could that be failing?

Steve

Yes, they share a common switch. Don't seem to have issues with it so far tho.
Do you think that could be a (baystack) switch (port) issue ? When switching ports and enabling a spare pfsense, the second one is starting to work just fine.
We have a ~ 1/6 mbps, sometimes up to 10 outgoing, but very rare.

slagr

@slagr:

Yes, they share a common switch. Don't seem to have issues with it so far tho.
Do you think that could be a (baystack) switch (port) issue ? When switching ports and enabling a spare pfsense, the second one is starting to work just fine.
We have a ~ 1/6 mbps, sometimes up to 10 outgoing, but very rare.

Could that happen because of:

"Jun 3 12:42:15 fw apinger: ALARM: WANGW(x.x.x.1) *** WANGWdown ***" ?

My apinger.conf has :


alarm down "WANGWdown" {
    time 120s
}

target "x.x.x.1" {
    description "WANGW"
    srcip "x.x.x.253"
    interval 10s
    alarms override "WANGWloss","WANGWdelay","WANGWdown";
    rrd file "/var/db/rrd/WANGW-quality.rrd"
}

Thanks.

stephenw10

Errm…. I don't understand what you're asking. :-\

Steve

slagr

@slagr:

Yes, they share a common switch. Don't seem to have issues with it so far tho.
Do you think that could be a (baystack) switch (port) issue ? When switching ports and enabling a spare pfsense, the second one is starting to work just fine.
We have a ~ 1/6 mbps, sometimes up to 10 outgoing, but very rare.

Well, the problem was (I think), that both em0 and bg0 shared the same IRQ.
We went ahead and removed em0 (as we were unable to assign a dedicated IRQ to em0 in bios).
Now waiting for the "outage".
OTOH, we got carp broken, as em0 has been used as a dedicated carp NIC.
So, as I'd like to get carp back for the time being, my question would be if I won't break anything (many thousands miles away), by assigning to both LAN interfaces (3) IPs from the same network we use behind the LAN iface, and protect carp on firewall. We used to have such configuration: bridge : WAN (bge0) - management IP, LAN (bge1) - no IP, OPT1 (em0) - carp. em0 went down from that picture.
Thanks.

stephenw10

Having a shared IRQ should not prevent the NICs from working. Having disabled msix for all pci devices it's likely to have more of an effect (I would have thought) but even so it shouldn't stop all traffic.
I am unsure of your network configuration from your description and I have only experimental experience with a CARP setup so I can't really tell you what would happen. Since you will be thousands of miles away getting it wrong would be very bad so I would have to advise waiting for another opinion. :-
In the mean time giving us a network diagram would help greatly.

Steve

slagr

@stephenw10:

Having a shared IRQ should not prevent the NICs from working. Having disabled msix for all pci devices it's likely to have more of an effect (I would have thought) but even so it shouldn't stop all traffic.
I am unsure of your network configuration from your description and I have only experimental experience with a CARP setup so I can't really tell you what would happen. Since you will be thousands of miles away getting it wrong would be very bad so I would have to advise waiting for another opinion. :-
In the mean time giving us a network diagram would help greatly.

Steve

Thanks, Steve.
You're right, that having a shared IRQ should not lead to such results. Reading/googling further, I think next step is to exclude CARP interface from the bridge. As I stated, that is an old, inherited setup, and all 3 NICs are members of the bridge.