Update to 2.5.2 from 2.4.5-p1 - no traffic from LAN to WAN anymore

tohil

Hi

I've upgraded the most complex pfsense box (the last of a few) from 2.4.5-p1 to 2.5.2. Direct update and fresh install and config import caused, that there was no package forwarding from internal interfaces to wan.

Ping from pfsense box to 8.8.8.8 works from WAN interface, but not when i select any internal interface.
I had to downgrade to 2.4.5-p1 to get all working again....

This box is running these additional packages:

avahi
haproxy
openvpn-client-export
-pfBlockerNG

Hardware is APU2 Board.

I have 1 WAN interface, and LAG on ibg1 and igb2. 6 VLANs on LAGG0

Traffic Shaper just on WAN interface for upload/outgoing traffic. because of limited support of VLAN/LAG Interfaces, and not limiting inter-LAN Traffic

Limiters on 2 VLANs

2 OpenVPN Servers

5 IPsec Tunnnels, with around 30 Phase2 at all

unbound dns resolver us lan interface as source, because there are some domains resolved over ipsec tunnels.

Config runs very smooth on 2.4.5

Anybody with some hint, what could cause this issue? It seems like an internal routing issue... I can see traffic in firewall log, but the traffic is not been routed....

WAN is DHCP with addtional DHCP Config file.. I see correct default gw in routing table... and as already said, i can reach internet from the box itself.

tohil

Cloud this be the issue?
https://redmine.pfsense.org/issues/9643

stephenw10

Unlikely since that not affect pings from pfSense with LAN set as the source. That never hits the LAN side firewall rules and hence the limiters.

No default route or an incorrect default route would behave like that though.
Traffic using the WAN as source IP would still work since there is a route-to rule applied to that. Other traffic uses the system routing table so a bad default route there would prevent it.

Check Diag > Routes.

Check the state table when trying to ping.

Steve

tohil

@stephenw10

Hi Steve

Thanks for your input and explanation on routing behavior from LAN or WAN interface.
I will setup a virtual VM with that config, to compare the routing table.

Also I ordered a spare device and can perform some tests on that device with the same hw config and interface names

regards

Angelo

tohil

@stephenw10

Hi Steve

I've setup an virtual pfsense and imported the config. After running some tests i found the issue....
-> NAT/Outgoing -> I have set Hybrid outbound NAT with some additional manuel rules... but there are not automatic rules...

Even if I change to "automatic outbound nat" setting. When i create a manual NAT Rule for internal network, I can ping again from this particular subnet.

System/Advanced/Firewall & NAT
NAT Reflection mode for port forwards: disabled
Enable NAT Reflection for 1:1 NAT: disabled
Enable automatic outbound NAT for Reflection: disabled

stephenw10

Automatic rules are on interfaces that gateways defined (WANs) from internal subnets on interfaces that don't have gateways defined (LANs).
If you're not seeing any rules created it pretty much has to be because your WAN interface does not have a gateway set on it.
That has not changed since 2.4.5 though.

Steve

tohil

@stephenw10 wan is DHCP with Gateway... internal interface have no Gateway.

Settings are working on 2.4.5, but not on 2.5.2 without changes.

I can share you more details via DM or Remote Session...

tohil

@stephenw10

I've drilled it down a littlebit more... missing automatic outbound NAT rules is caused by enabling "Configuration Override" on DHCP WAN Interface... I need the following option dhclient option 60 for my VDSL ISP:

interface "{interface}" {
  send dhcp-class-identifier "100008,0001,,pfsense 2.4";
}

Is there an other way to set this values in GUI directly? maybe you can reproduce this on your lab?
this option is not uncommon here in switzerland for users, which are not using the providers standard router....

stephenw10

Ah, that's interesting. And that behaviour changed between 2.4.5 and 2.5? You had that custom option there in 2.4.5p1 and auto outbound NAT rules were created?

Nice catch! That could have taken an age to find.

You should be able to set that in the 'Send Options' that are made available when you check 'Advanced Configuration'. Without having to use configuration override entirely.

Steve

stephenw10

Yeah, if you set it there you get, for example:

interface "vtnet0" {

	supersede interface-mtu 0;
# DHCP Protocol Timing Values

# DHCP Protocol Options
	send dhcp-class-identifier "100008,0001,,pfsense 2.4";

	script "/usr/local/sbin/pfSense-dhclient-script";
}

But automatic outbound NAT rules are still created.

Steve

tohil

@stephenw10 Hi Steve,

yes this setting worked since years until now :-)

Can you tell me where the mentioned interface config file is located at?
I will compare the file if I add the addition options by config file and by gui setting.

can you tel me at which gui field this option 60 has to be added?

Send options
Request options (Option 55)
Require options
Option Modifiers

thanks for your reply, I will test it as soon as possible.

regards Angelo

stephenw10

It's set in 'Send Options'.
The file is /var/etc/dhclient_wan.conf, assuming it's your WAN interface.

Steve

tohil

@stephenw10
Seems to working...not tested on prod system yet

but adding

dhcp-class-identifier "100008,0001,,pfsense 2.4"

to send options genereates the following config file, which matches my old config method. And Outbound NAT Rules are created!

interface "em0" {

        supersede interface-mtu 0;
# DHCP Protocol Timing Values

# DHCP Protocol Options
        send dhcp-class-identifier "100008,0001,,pfsense 2.4";

        script "/usr/local/sbin/pfSense-dhclient-script";
}

I'm going to test this on prod asap! thanks Steve!

tohil

@stephenw10

Changed WAN Interface Config on 2.4.5-p1, created a backup and performed a fresh installation with 2.5.2.

working now :-)

thanks for your support

tohil

I'm using RAM drives, /var is showing up multiple times on dashboard System Status. Is this known?

noplan

@tohil

ram drive has something to do with no traffic from LAN 2 WAN after update ?
brNP

tohil

@noplan No... just mentioned it... its off-topic... if you are mod, you can delete :-)
issue was just the dhcp settings on WAN.

noplan

@tohil

no no mod just wondering .. .. still got here some 2.4.5xxx poxes to do
even a p1 version yesterday, but was not able to run in the same problem like you ... ;)

brNP

stephenw10

In 2.5.2? Are you using ZFS?

That seems familiar though I'm not hitting it here...

Edit: Yup, this: https://redmine.pfsense.org/issues/12144

It's ugly but just cosmetic.

Steve

tohil

@stephenw10

Yes, ZFS after reinstalled 2.5.2. Bug seems to be known and would be fixed someday... as you said, its just cosmetic :-)