Ubiquiti USG-3P to PFSense

chumunga · Nov 19, 2020, 7:54 AM

I will do my best to not get pedantic as I have been getting mixed results.
3 Attempts, 3 different Results but as stated ill keep my request simple.

First I am a new PFSense user.
I'd like....

USG to be the edge router
PFsense - DHCP and all firewall

I tried to configure as such and i configure as such

Modem <=> USG-3p <=> PFSense <=> Switch

USG
- WAN : (DHCP from ISP)
- LAN : 192.168.1.1 DHCP Server Disabled

PFsense
- WAN : 192.168.1.2
- IPv4 Upstream Gateway for WAN : 192.168.1.1 as well as "none"
- LAN : 192.168.2.1 DHCP Server Enabled

But so far my attempts has been strange...

1st Attempt, I can ping out from PFSense Console, and reach USG Config WebGUI from PC but couldn't get internet from PC

2nd Attempt, Performed Reset same Config, Couldnt ping out or reach webgui

I admit i did once get Internet on PC to work but while attempting to get Unifi Controller Software to work, I couldn't "adopt" anything (adopt is Ubiquiti's term for the controller software to take control of other Ubiquiti Unfi devices and allows for webgui management/configuration)

Suggestion is much appreciated, Thank you.

stephenw10 · Nov 19, 2020, 5:29 PM

No default route in pfSense would present like that. It can reach the USG because it's in the WAN subnet which it does have a route to.

Check Diag > Routes.

Make sure there is a gateway set on WAN and its 192.168.1.1.
Make sure that is set as the default gateway in Sys > Routing > Gateways.

Otherwise you have outbound NAT set as auto (the default)?
You have the default pass all rule on LAN?

Steve

chumunga · Nov 25, 2020, 8:50 AM

Hi Steve,

I configured a static route on the PFSense to my ISP network, and I had to reboot my modem, as comcast doesn't like me changing my router.

Now to figure out unifi controller which is another challenge :)

TY

chumunga · Nov 25, 2020, 9:01 AM

Belay that, the static route didn't do anything it was rebooting the modem. I disabled the static route to see what would happen, and nothing :D

In case anyone out there is attempting a similar configuration here is what I did.

USG
-WAN DHCP
--LAN default with DHCP Off, 192.168.1.1
-DNS, Comcast DNS

PFSense
-WAN - Static 192.168.1.2
--UPStream Gateway 192.168.1.1
-LAN - 192.168.2.1
-DHCP Server Enable

stephenw10 · Nov 25, 2020, 1:18 PM

Indeed you should not need any static routes in pfSense. It just has to have a default route via the upstream router. If it only has one gateway set that should be the default.

Steve

MacUsers · Feb 19, 2023, 5:10 PM

@chumunga said in Ubiquiti USG-3P to PFSense:

USG
-WAN DHCP
--LAN default with DHCP Off, 192.168.1.1
-DNS, Comcast DNS

PFSense
-WAN - Static 192.168.1.2
--UPStream Gateway 192.168.1.1
-LAN - 192.168.2.1
-DHCP Server Enable

I know, this is an old post but my sitution is exactly like this and trying to figure out a couple of other stuff.

I pluged in the USG in my existing pfSense network. Following the above configuration (I only changed the USG LAN from the default 192.168.1.0/24 to 192.168.10.0/29), I got pfSense working just like before.

The first issue I have though: VPN connection stopped working. I believe that because of double natting? Does anyone know how can I fix that?

The second issue: My CloudKey is connected to the physical port on pfSense and having an IP of 10.0.20.5, which USG cannot reach to for obvious reason. I added a static route on USG like this:
set protocols static route 10.0.20.0/28 next-hop 192.168.10.2 distance 1

where 10.0.20.0/28 is the subnet where CKG2 resides and 192.168.10.2 is WAN IP of pfSense, conencted to the LAN1 of USG. netstat shows the routes:

ubnt@ubnt:/config$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 pppoe0
10.0.20.0       192.168.10.2    255.255.255.240 UG        0 0          0 eth1
192.168.50.0    192.168.10.2    255.255.255.240 UG        0 0          0 eth1
xx.xxx.xx.xx    0.0.0.0         255.255.255.255 UH        0 0          0 pppoe0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
192.168.10.0    0.0.0.0         255.255.255.248 U         0 0          0 eth1

As you can see, I also did another static route for my VPN subnet (192.168.50.0/28) but neither working. CKG2 can ping USG but not the other way around.

I'm pretty sure I didn't understand something in here but any help would be highly appreciated.

-S

stephenw10 · Feb 19, 2023, 7:27 PM

Other than the cloud key is everything else working?

By adding static routes to all the subnets behind pfSense like that you probably want to disable outbound NAT on pfSense and allow the USG to outbound NAT those subnets directly.

A VPN client connecting out should still work fine. External clients connecting to a VPN server on pfSense would break unless the USG is forwarding that.

Steve

NightlyShark · Feb 19, 2023, 8:50 PM

@macusers See my last reply here.

NightlyShark · Feb 19, 2023, 8:52 PM

@macusers In your case the PPPoE (passed-through the CPE) has to be handled by the USG.

MacUsers · Feb 19, 2023, 9:05 PM

thanks @stephenw10!!
Could you explain a bit more? Things like accessing Internet etc. from any subnets/VPCs are working fine. Anything from inside, whatever I checked so far, is working, apart from connecting to CKG2. Looks like anything frpm outside, not working.

NightlyShark · Feb 19, 2023, 8:59 PM

@macusers Please be aware, your screenshot suggest you have the USG behind a NAT already.

MacUsers · Feb 19, 2023, 9:06 PM

@nightlyshark said in Ubiquiti USG-3P to PFSense:

@macusers In your case the PPPoE (passed-through the CPE) has to be handled by the USG.

Thanks @NightlyShark!
Just trying to understand - isn't USG already handing the PPPoE for me - otherwise I wouldn't get IP from ISP (or even the Internet) in the first place, right?

Or, I completely misunderstood?

MacUsers · Feb 19, 2023, 9:04 PM

@nightlyshark said in Ubiquiti USG-3P to PFSense:

@macusers Please be aware, your screenshot suggest you have the USG behind a NAT already.

If I disable NAT on USG, I don't get Internet from inside any of the LAN subnets.

NightlyShark · Feb 19, 2023, 9:13 PM

@macusers Hi!
That means you have the USG in a private network (192.168.XXX.XXX is not routable through the internet) and are behind a NAT. Check out the post I pointed to before, it explains most of it, although, in your case, you need to configure USG to use PPPoE in WAN, not PfSense.
https://download.discomp.cz/Ubiquiti/Navody/UniFi_Controller_V4_UG.pdf
Page 58

stephenw10 · Feb 19, 2023, 9:19 PM

That is the USG. Looks like the PPPoE is just not connected at that point. Or hasn't yet passed a gateway. Or, since it's a PtP protocol, it doesn't need one.

NightlyShark · Feb 19, 2023, 9:22 PM

@macusers You need to be able to PPPoE from your USG to your ISP (Internet Provider) through your CPE (customer premises equipment, the cheep modem, router or both in one your ISP gives you), in order to have a true public IPv4. The ability to connect your own router (USG) through your CPE modem hinges on either have your CPE not block layer 2 direct PPPoE connections from other devices through it's modem (layer 2) stream (what is called PPPoE passthrough and is usually only enabled by calling your ISPs support and specifically request it) or have the CPE modem not connect to your ISP at all (bridge mode, again, consult your ISP for details). You then configure USG with PPPoE.

PPPoE is a Layer 2 protocol, meaning its packets are not routable, eg are not sent to an IP from USG, but rather, the USG must be able to connect with your ISP on a "switch" (Data Link Layer) level, to put it simply, with MAC addresses. That is why the modem needs to allow PPPoE, because it bypasses it entirely. I hope I did not confuse the issue, some things may be lost in translation (english is my second language, not my native)

NightlyShark · Feb 19, 2023, 9:26 PM

@stephenw10 PtP protocols are Layer 3 (they have IP packets). PPPoE is Layer 2 (the sender and the recipient are identified at an electronic digital network circuit switch (what we call a switch) level, so they need special treatment to pass from any LAN ports on the CPE to the ISP, what we call PPPoE passthrough. Without PPPoE pass-through (it's in the word) the packets are treated as invalid by the switch or bridge on the CPE, because the ISP's MAC (or any MAC that responds to PPPoE frames) address doesn't exist in that network, so they are dropped.

MacUsers · Feb 19, 2023, 9:42 PM

@nightlyshark said in Ubiquiti USG-3P to PFSense:

@macusers You need to be able to PPPoE from your USG to your ISP (Internet Provider) through your CPE (customer premises equipment, the cheep modem, router or both in one your ISP gives you), in order to have a true public IPv4. The ability to connect your own router (USG) through your CPE modem hinges on either have your CPE not block layer 2 direct PPPoE connections from other devices through it's modem (layer 2) stream (what is called PPPoE passthrough and is usually only enabled by calling your ISPs support and specifically request it) or have the CPE modem not connect to your ISP at all (bridge mode, again, consult your ISP for details). You then configure USG with PPPoE.

PPPoE is a Layer 2 protocol, meaning its packets are not routable, eg are not sent to an IP from USG, but rather, the USG must be able to connect with your ISP on a "switch" (Data Link Layer) level, to put it simply, with MAC addresses. That is why the modem needs to allow PPPoE, because it bypasses it entirely. I hope I did not confuse the issue, some things may be lost in translation (english is my second language, not my native)

I'm very confused here - perhaps the lack of my understandimg but very confused. I do get the real Public IPv4 from ISP, which is assigned to me as Static.

This is tghe WAN setup on USG:

And this WAN in pfSense:

What am I missing?
My another issue is: as USG cannot connect the CK (hence cannot adopt), cannot do much in terms of configuration on USG apart from those settings only.

NightlyShark · Feb 19, 2023, 9:54 PM

@macusers To clarify:
You want to have a topology of:
---INTERNET---
---ISP PPPoE endpoint---
---ISP CPE Modem with PPPoE passthrough enabled---
---USG PPPoE configuration of WAN interface---
---USG NAT---
---PfSense FW---
---Clients---

or

---INTERNET---
---ISP PPPoE endpoint---
---ISP CPE Modem with PPPoE passthrough enabled---
---PfSense PPPoE configuration of WAN interface---
---PfSense NAT---
---USG and other Clients---
?

NightlyShark · Feb 19, 2023, 9:54 PM

@macusers For example, my config is:
---INTERNET---
---ISP PPPoE endpoint---
---ISP CPE Modem with PPPoE passthrough enabled---
---PfSense:

---PfSense NAT---
---Clients---

Ubiquiti USG-3P to PFSense

@macusers For example, my config is: ---INTERNET--- ---ISP PPPoE endpoint--- ---ISP CPE Modem with PPPoE passthrough enabled--- ---PfSense:

@macusers For example, my config is:
---INTERNET---
---ISP PPPoE endpoint---
---ISP CPE Modem with PPPoE passthrough enabled---
---PfSense: