Static IP WAN block, devices not connecting

tcw · Jan 11, 2023, 5:50 PM

Hello, I think I'm configuring something incorrectly in pfSense.

I switched service providers to AT&T and ordered a /29 IP address block. They gave me the following addresses:

Subnet mask: 255.255.255.248
Network base address: 70.x.x.8
Router: 70.x.x.14
Broadcast: 70.x.x.15
Usable Range: 70.x.x.9 - 70.x.x.13

I set the WAN interface to Static IPv4 and entered the following:

IPv4 address: 70.x.x.13/29
IPv4 upstream gateway: 70.x.x.14

This is the only gateway and is selected as default. DNS servers seem to be set up properly. I can ping, nslookup, and traceroute fine (IPs and hostnames) from the firewall diagnostics and shell, and I can check for pfSense updates, so the router is connected. But nothing behind it is.

I can't connect to the gateway interface at 192.168.1.254 (it's a Nokia BGW320) from my browser. I opened the firewall allow rule for any/any between my LAN Net (192.168.10.1/24) and the "LocalAddr" alias that contains 192.168.0.0/16 and the others.

The network switch and BGW320 are connected to two different physical interfaces on the router. I am using VLANs but I did not have the WAN interface configured for the previous cable modem/ISP so I made no changes there (other than changing DHCP to Static IP).

I must have misunderstood how to configure static IP on the router. Maybe LAN IP vs. WAN IP?

What am I doing wrong? Thanks.

viragomann · Jan 11, 2023, 6:34 PM

@tcw said in Static IP WAN block, devices not connecting:

The network switch and BGW320 are connected to two different physical interfaces on the router. I am using VLANs but I did not have the WAN interface configured for the previous cable modem/ISP so I made no changes there (other than changing DHCP to Static IP).

You did not make any changes in the internal subnets, but they cannot communicate anymore after replacing the WAN IP and gateway?

tcw · Jan 11, 2023, 6:40 PM

@viragomann said in Static IP WAN block, devices not connecting:

You did not make any changes in the internal subnets, but they cannot communicate anymore after replacing the WAN IP and gateway?

Correct, I can communicate within and across subnets/VLANs but not with the BGW320 or beyond. None of my subnets use the 192.168.1.x space the BGW320 uses as default, so I'm not thinking it would be an IP conflict.

photomankc · Jan 11, 2023, 6:41 PM

@tcw To be completely clear, the hosts you can ping from the firewall are internet host IPs right?

Is there is a NAT rule to NAT your internal hosts to the firewall's WAN IP address in place?

tcw · Jan 11, 2023, 6:47 PM

@photomankc said in Static IP WAN block, devices not connecting:

@tcw To be completely clear, the hosts you can ping from the firewall are internet host IPs right?

Is there is a NAT rule to NAT your internal hosts to the firewall's WAN IP address in place?

Correct, all of the pfSense diagnostics work for internet host IPs. Issuing "ping cisco.com" from shell gives me a host IP of 72.163.4.185 and ~ 35ms pings with no packet loss.

No NAT rules between internal hosts and the WAN IP, maybe that is what I am missing with static WAN IP assignment? Would I then need a rule for each VLAN interface, and how would that look?

viragomann · Jan 11, 2023, 6:51 PM

@tcw said in Static IP WAN block, devices not connecting:

I can't connect to the gateway interface at 192.168.1.254 (it's a Nokia BGW320) from my browser.

So this is your internet gateway, connected to the WAN interface?

photomankc · Jan 11, 2023, 6:52 PM

@tcw You need a NAT rule that NATs all your internal private IPs that will land at this firewall to the external IP of the firewall. Look at Firewall -> NAT -> Outbound. There is usually an automatic rule there to handle it. If not you might need to change the "Mode" to "Hybrid" and add a manual rule there. Source is all your internal subnets, destination is "any" and ports are "any". The NAT address is your "WAN address".

That should handle the NAT. It's odd though because if you just swapped providers I would expect there would have been NAT in place already.

tcw · Jan 11, 2023, 6:55 PM

@photomankc said in Static IP WAN block, devices not connecting:

@tcw You need a NAT rule that NATs all your internal private IPs that will land at this firewall to the external IP of the firewall. Look at Firewall -> NAT -> Outbound. There is usually an automatic rule there to handle it. If not you might need to change the "Mode" to "Hybrid" and add a manual rule there. Source is all your internal subnets, destination is "any" and ports are "any". The NAT address is your "WAN address".

That should handle the NAT. It's odd though because if you just swapped providers I would expect there would have been NAT in place already.

My apologies, you are correct and I was looking at the wrong tab. Outbound NAT is set for hybrid, with auto WAN and ISAKMP rules, and a manual static port rule for a Nintendo Switch.

photomankc · Jan 11, 2023, 6:57 PM

@viragomann AT&T network box will have that address, but that IP would normally not be the default gateway for the WAN side. Just management IP of the AT&T network box. I'm on their fiber service and that's how mine works anyway. My DFGW is a public IP, but I can still get to the 192.168.1.254 management page IF the firewall is NATing the traffic going to it.

photomankc · Jan 11, 2023, 7:09 PM

@tcw Okay, so we're good on WAN side, the firewall's WAN IP is getting around the world. So the issue is through the device or somehow on the LAN side.

You might try going under Diagnostics -> States -> Reset State table. That should force NAT to clear and start over. After that check the States output and see if you are getting connections built.

tcw · Jan 11, 2023, 7:21 PM

@photomankc said in Static IP WAN block, devices not connecting:

@tcw Okay, so we're good on WAN side, the firewall's WAN IP is getting around the world. So the issue is through the device or somehow on the LAN side.

You might try going under Diagnostics -> States -> Reset State table. That should force NAT to clear and start over. After that check the States output and see if you are getting connections built.

Thanks to everyone in this thread... I'm still troubleshooting. I see connections established between the pfSense public IP I assigned (70.x.x.13) and 1.1.1.1:853, but nothing from any of the internal devices. I'm seeing some "NO_TRAFFIC:SINGLE" and "CLOSED:SYN_SENT" states on the LAN with no packets/bytes, so it looks like things are trying and failing.

viragomann · Jan 11, 2023, 7:23 PM

@tcw
Maybe we can get a step further if you answer my question...

tcw · Jan 11, 2023, 7:27 PM

@viragomann said in Static IP WAN block, devices not connecting:

@tcw said in Static IP WAN block, devices not connecting:

I can't connect to the gateway interface at 192.168.1.254 (it's a Nokia BGW320) from my browser.

So this is your internet gateway, connected to the WAN interface?

Sorry if you were referring to this question, I did lose it in the shuffle! Yes, this is my internet gateway. It has a private IP address of 192.168.1.254 and is connected to the WAN interface of pfSense. In the pfSense GUI I have the WAN interface configured for Static IPv4, with the 70.x.x.13 address and 70.x.x.14 gateway.

stephenw10 · Jan 11, 2023, 8:07 PM

It's almost certainly because there is no outbound NAT happening. And that is probably because the WAN interface doesn't have the gateway set on it directly.
In the outbound NAT rules page do you see the auto generated rules for 192.168.10.0/24 on WAN?

Steve

tcw · Jan 11, 2023, 8:13 PM

@stephenw10 said in Static IP WAN block, devices not connecting:

It's almost certainly because there is no outbound NAT happening. And that is probably because the WAN interface doesn't have the gateway set on it directly.
In the outbound NAT rules page do you see the auto generated rules for 192.168.10.0/24 on WAN?

Steve

Thanks, Steve, yes I do.

Could you explain what you mean by "doesn't have the gateway set on it directly"? I have the upstream IPv4 gateway set for 70.x.x.14 in the WAN interface on pfSense, and the gateway itself has its "Public Gateway Address" set for 70.x.x.14 in its "Public Subnet" section of the Subnets & DHCP GUI.

I'm back at the point of power cycling. I appreciate everyone's help.

tcw · Jan 11, 2023, 8:22 PM

Just to be thorough:

The pfSense router can communicate with the world
The LAN devices can communicate among themselves (and across VLANs) and with the router
A device connected directly to the gateway's built-in switch can communicate with both the world and the gateway GUI
No LAN device (connected to the router's LAN port through a switch) can communicate with the world or the gateway GUI

viragomann · Jan 11, 2023, 8:24 PM

@tcw
In this case you have to assign an IP to pfSense WAN in the 192.168.1.0/24 subnet.
Firewall > Virtual IPs. Add an IP of type "IP alias" to WAN, maybe 192.168.1.2, set the correct mask.

Then add an outbound NAT rule to WAN to the top of the rule set for the source of all your internal subnet (e.g. 192.168.0.0/16), destination = network 192.168.1.254/32, translation = the virtual IP you've added before.

photomankc · Jan 11, 2023, 9:07 PM

@tcw So I might use the PCap feature here to see whats going out the WAN interface. If you ping (internal) ---> (4.2.2.2), when that comes out the WAN interface what is the source address then? If it's not 70.x.x.13 you have a NAT issue. What is the return traffic look like if there is any.

As an aside:
I have AT&T fiber and other than mine being DHCP with pass-through it's just like what you have. I can access my AT&T network box without any virtual IP or extra NAT setup. I'd start with why you can't get to internet hosts first and tackle the GUI on their gear after that is sorted.

Here's the result when I ping from and internal client to that address:
15:03:43.879742 IP (tos 0x0, ttl 62, id 7358, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->e8b8)!)
104.X.X.253 > 4.2.2.2: ICMP echo request, id 17514, seq 22, length 64
15:03:43.891511 IP (tos 0x0, ttl 55, id 14527, offset 0, flags [none], proto ICMP (1), length 84)
4.2.2.2 > 104.x.x253: ICMP echo reply, id 17514, seq 22, length 64

Here is the same to 192.168.1.254:
15:06:12.813810 IP (tos 0x0, ttl 62, id 37941, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 0 (->b49e)!)
104.x.x.253 > 192.168.1.254: ICMP echo request, id 34505, seq 18, length 64
15:06:12.814471 IP (tos 0x0, ttl 64, id 59990, offset 0, flags [none], proto ICMP (1), length 84)
192.168.1.254 > 104.x.x.253: ICMP echo reply, id 34505, seq 18, length 64

It "just works" as long as my other internet bound NAT is working.

tcw · Jan 11, 2023, 9:18 PM

Solved. The LAN interfaces' IPv6 configuration was still set to Track Interface (instead of Disabled). I disabled DHCP6 on the WAN interface before I started, but I didn't go back to the LAN interfaces to disable stateless DHCP and IPv6. It was not enough to disable IPv6 on the WAN side even though there was no WAN IPv6 interface to track.

I did find a flaky cable in the process, and I learned a lot about outbound NAT. Thanks all for walking me through everything!

viragomann · Jan 11, 2023, 9:22 PM

@tcw said in Static IP WAN block, devices not connecting:

The LAN interfaces' IPv6 configuration was still set to Track Interface (instead of Disabled).

Strange that this matters even when connecting to an IPv4.