pfSense 2.5.0.a.20201127.0650 NAT Issues

dragoangel

Hi, has somebody same issues: https://redmine.pfsense.org/issues/11188 ?

dragoangel

I have also same issues with NPt IPv6, not only with IPv4 NAT. On 2.4.5 and before it was working fine for both WAN01 and WAN02.
For me it looks like reply-to autorule is broken with NAT\NPt or something near there on 2.5.0.a.20201127.0650.
I rechecked all system_advanced_firewall.php and I not have enabled:

Static route filtering
Disable reply-to on WAN rules
Disable Negate rule on policy routing rules

I have enabled:

NAT Reflection mode in Pure NAT
Enable NAT Reflection for 1:1 NAT
Enable automatic outbound NAT for Reflection

Tried to disable\change NAT reflection on specific NAT Rules, tried enable\disabled one by one system_advanced_firewall.php mentioned at top, but nothing helps.
I see on firewall tab that I have connection by allow rule, but from client it not reachable. If for example I simply bind service to needed IP on pfSense without NAT it start to work correctly for IPv4. If I remove NPt it also start work correctly.

More symptoms:
If my WAN01 is default TIER1 I can only connect to WAN01 IP4 and IPv6 from outside, but not WAN02 IPs. If I swap priority beetween WAN02 and WAN01 - I can only connect to WAN02 now. The most strange part that if I mark TIER1 GW as DOWN then I can access both WAN01 and WAN02 IPs from outside! For NPt - if I move WAN02IPV6 to TIER1 and WAN01IPV6 to TIER2 I also has access to both IPs from outside. I also can access from LAN any WAN IPs not related what is going outside.

dragoangel

Nobody?

kiokoman

@dragoangel
nope, work fine for me

rdr on vmx0 inet proto { tcp udp } from ! $DNS to ! $DNS port 53 -> 172.17.0.100
rdr on vmx0 inet proto { tcp udp } from ! $NTP to ! $NTP port 123 -> 192.168.10.200
rdr on vmx0 inet6 proto { tcp udp } from ! $DNSv6 to ! $DNSv6 port 53 -> 2001:470:b682:xxxx:xxxx:xxxx:xxxx:fffe
rdr on vmx0.30 inet proto { tcp udp } from any to ! $DNS port 53 -> 172.17.0.100
rdr on vmx0.30 inet proto { tcp udp } from any to ! $NTP port 123 -> 192.168.10.200

check your /tmp/rules.debug
but i don't have nat reflection and that stuff enabled
only Enable automatic outbound NAT for Reflection

dragoangel

@kiokoman but your rules looking like outbound ones. For me not working rules when they nated to 2 wans on wan where trafic doesn't tier1. I have same rules for DNS and they work, so issue I have not in outbound NAT itself. Will check rules conf now.

dragoangel

@kiokoman my npt rules looks like:

binat on $WAN01HE inet6 from 2001:470:1111::/48 to any -> 2001:470:2222::/48
binat on $WAN01HE inet6 from any to 2001:470:2222::/48 -> 2001:470:1111::/48

But with this my 2001:470:2222::/48 not accessible till my 2001:470:1111::/48 working and TIER1.

stephenw10

So I assume you have the default gateway set to a gateway group?

Do you see the traffic hit the target from the port forwards?

But replies go out the default gateway as though reply-to tagging is not being applied?

Steve

dragoangel

@kiokoman sample of one of my NAT rules:

rdr on lagg0.3081 inet proto { tcp udp } from $admin to (self) port 30001 -> 192.168.2.15 port 3389
rdr on lagg0.3082 inet proto { tcp udp } from $admin to (self) port 30001 -> 192.168.2.15 port 3389
# Reflection redirect
rdr on { lagg0.3091 gif0 gif1 lagg1 lagg1.11 lagg1.12 lagg1.13 enc0 openvpn } inet proto { tcp udp } from $admin to (self) port 30001 -> 192.168.2.15 port 3389

With this on TIER1 public IP I have access, while on TIER2 - doesn't. If I swap TIER beetween WANs - access also swaps. On pfSense 2.4 and far ago it was works on both WANs as it must. It started right after update to 2.5.0.

dragoangel

@stephenw10 I have WANGW group for IPv4 and WANHEGW group for IPv6. First ISP set to TIER1 and second to TIER2.
On firewall tab of WAN2 I see indication of traffic reaching allow rules and connected states, e.g.: 1/64kb - 1 connected client, 64kb pass, but I not get anything in reply from pfsense - not even 1 bit. If I swap WAN2 to TIER1 and WAN1 to TIER2 then I loss connection on WAN1 and get working on WAN2.
I doesn't know how exactly I need troubleshoot or debug "reply-to tagging applied or not", but from my tests it only one logical conclusion I can make - reply-to not working. If you can explain me how to debug it - please. Or maybe you can try reproduce same setup.

stephenw10

We need to see the firewall rules added to pass that traffic really. Both what's in the ruleset and the resulting rule table.

I assume lagg0.3081 and lagg0.3082 are your two WANs there?

Run a packet capture for the traffic on the 192.168.2.0/24 interface. So you see the incoming requests? Do you see replies?
Run a pcap on the tier1 WAN whilst testing on the other WAN. Do you see replies leaving via the wrong WAN?

Steve

dragoangel

@stephenw10 yes lagg0.3081 and lagg0.3082 are my two WANs.

To note: real IPs and ports is changed here to privacy.

About pcap - send by email.

In short - not see traffic on TIER1 while trying connect to TIER2 NATed port, but see strange TCP re-transmission. Note: for this rules NAT reflection disabled to simplify all scheme, in any case changing reflection on rule - do not change anything.

stephenw10

Replied on PM. But new need a pcap on the tier1 WAN while connecting via the tier2 WANB to verify replies are being sent that way rather than dropped for some reason.

You gateway groups are not being populated in the ruleset. The only reason that should ever happen normally is if you have not set Skip rules when gateway is down and all the gateways were down. Which clearly isn't the case here.
You are not policy routing anything via those so it won't affect anything directly but could indicate the gateways have some odd setting.

Steve

dragoangel

@stephenw10 said in pfSense 2.5.0.a.20201127.0650 NAT Issues:

You gateway groups are not being populated in the ruleset. The only reason that should ever happen normally is if you have not set Skip rules when gateway is down and all the gateways were down. Which clearly isn't the case here.

Yes, this isn't the case. I not see any bit of traffic when dump WAN TIER1 while trying connect to WAN TIER2. Can it be due promiscuous mode isn't enabled? I doesn't think so.

dragoangel

@stephenw10 send pfSense status report to your email

dragoangel

Hi, @stephenw10 I done full reinstall from scratch to 2.4.5_p1 on ssd and updated to 2.5.0.a.20201127.0650 and restored from backup - still same issue with:

GWWANGROUP = "  "
GWWANGROUP6 = "  "

I also found in logs:

Jan 5 00:16:21 	php-fpm 	97323 	/rc.filter_configure_sync: An error occurred while trying to find the interface got `MyMainIPv6GWIP`. The rule has not been added.
Jan 5 00:16:21 	php-fpm 	97323 	/rc.filter_configure_sync: An error occurred while trying to find the interface got `MyMainIPv4GWIP`. The rule has not been added.

Maybe this root case why I have this?

Also want to note: when I restore from backup - if I used console\terminal it always "merges" in strange way my xg7100u switch configs and break everything, due to this reinstall takes for me crazy long and was successful only on second time. It will be cool if pfsense on terminal also ask about preserving switch conf or not.

dragoangel

@stephenw10 can you please help with this issue? It still in place. Also I doesn't receive any updates on my development 2.5 pfsense even that comes on 2.4.5_p1 stable (on another xg7100u).

saeed

Hi,
after upgrading to 2.5.1 my port forwards only works for active wan. is it related to this bug?
any solution?

dragoangel

@saeed you need update to latest version and it will fix nat, but not NPt for ipv6.

saeed

@dragoangel said in pfSense 2.5.0.a.20201127.0650 NAT Issues:

you need update to latest version and it will fix nat, but not NPt for ipv6.

it's a production server and already updated to 2.5.1
you mean update to latest development snapshot?

saeed

@dragoangel
https://redmine.pfsense.org/issues/11805