how to separate two WAN connetions into home internet and work internet virtually

Djkáťo

Hello~
I have a Netgate 1100, and just had a second ISP install a second internet into our home.

I'd like some guidance on what the best way is to achieve my goal:
I intend one to be for home use, and I'd like the other to be for fallback and server access. Home network is through vDSL (WAN), and server network is through fiberoptic(OPT1).

From some research I did, I found that the way to do this would be VLAN. Looking at pfsense documentation and just browsing around the settings, I don't quite understand how it works. All interfaces already have a VLAN (I think by default?) so do I need to make new ones according to https://docs.netgate.com/pfsense/en/latest/vlan/configuration.html#web-interface-vlan-configuration?

Or do I just tell my switches to map the server port to VLAN 4092 (OPT interface) and that's it? How do I forbid it access to the rest of the network then? By changing WAN to VLAN 1?

Here's a diagram of my network:
home network.drawio.png

Thanks for any tips, vlans are a bit confusing. Or if there's a better, non-vlan way you'd do it lemme know!

Patch

@Djkáťo I guess you need the following interfaces
LAN x3

Home
Office
Server

WAN x2

DSL
Fibre optic

As Netgate 1100 has an internal programmable switch with 3 exposed NIC you will need an external programmable switch to achieve this.
So at least one of the internal NIC will need to pass multiple tagged VLAN to the external programmable switch.

Using LAN labled NIC on the 1100 to connect all 3 LAN VLANs is reasonable. The optimal solution probable depends on the maximum aggregate through put each NIC will need to handle.

To see how to program the internal Netgate 1100 switch see https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/configuring-the-switch-ports.html

To see how to program OPT as and additional WAN See https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/opt-wan.html

Djkáťo

@Patch I've been rereading your message over and over but I'm still not sure I understand what you mean.
So you say ideally I'd have 3 LAN ports, but since I don't, I should program the LAN interface to pass tagged VLAN to the switch that goes after it, which will split which cable is which VLAN? If I got that right, then yeah that's what I was thinking initially I need to do, just unsure how to set that up.

The switch right after the Netgate is not webmanaged, but the other ones are. Is it reasonable to assume that the VLAN tags will propagate to the next switches that are webmanaged? So after I somehow tag VLANs, in the switch that my server is connected to, I set the servers port to the same VLAN ID as the one I set for the fibreoptic, and the rest leave at default? I'd like the default network (VLAN 1 I think) to stay as HOME.

If I understood, then the missing info for me is how I should do that tagging of VLANS in pfsense you mentioned. Have any rough steps/guides that could help me? The links you gave I read, but the "configuring switch ports" turns OPT into another LAN I think. I don't see from that guide how to split OPT and WAN into two VLANs and have that work. OPT I already figured out how to turn into another WAN.
thank you

Djkáťo

Okay I got the vlans up and running, all that's left is how do I force all vlan traffic through OPT only?

Djkáťo

I've been googling and playing around for hours in pfsense but I just can't find a way to:

force my vlan interface to only route traffic trough OPTs internet, and not WAN
redirect all traffic that hits my public IP in OPT interface to my vlan PUBLIC_OPT1 interface -> server local IP

The way I set up my vlan is:

create a public vlan in interface->switches, where id 0 and 2 are tagged.
I created a VLAN in interface -> assignments -> VLAN, and added it as an interface in Interfaces -> Interface Assignments, and marked all as active.
Then I could set all my switches to use tagged vlan on all ports, and enable VLAN in unraid and force docker to only use the vlan subnet.

Any tips on how to do my last steps?

stephenw10

You need to use policy based routing to send traffic from the internal VLAN subnet via the OPT gateway:
https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html

You would need to use either port forwards or 1:1 NAT to send external traffic to the internal server IP:
https://docs.netgate.com/pfsense/en/latest/nat/index.html

Patch

@Djkáťo said in how to separate two WAN connetions into home internet and work internet virtually:

The switch right after the Netgate is not webmanaged, but the other ones are. Is it reasonable to assume that the VLAN tags will propagate to the next switches that are webmanaged?

No.
You are suggesting learning how to set up VLAN in your network and reverse engineer undocumented switch behaviour at the same time. That will waste a lot of your time.

IMO
Move a managed switch to connect directly to your Netgate 1100 to ensure all tagged traffic is running only on Ethernet cable and managed switches.

Djkáťo

Thanks @stephenw10 , that's the settings I've been toying with for hours now, but I can't get it to work.
When I navigate to my public ip, nothing is loading. I'd expect at least unraid webpage to show up.

I redid them with guidance of the docs you send again. PUBLIC_OPT1 interface has my VLAN. PUBLIC_OPT1 uses 10.0.1.0/24, everything else 10.0.0.0/24. gateway OPTGW is the gateway OPT1 uses, which wires to a fiberoptic router.

What am I doing wrong? I allowed all sources and destinations to get it working at all, don't know if setting destination on OPT to public ip / OPT address is why it isn't working.

Djkáťo

I realized and had done that, switched some switches around and the whole path to the server is now managed switches, set it up and the server works with the VLANs now :) thank you

@Patch said in how to separate two WAN connetions into home internet and work internet virtually:

Move a managed switch to connect directly to your Netgate

stephenw10

@Djkáťo said in how to separate two WAN connetions into home internet and work internet virtually:

When I navigate to my public ip, nothing is loading.

How are you testing that? It won't work from anything behind pfSense unless you have enabled NAT reflection. You need to test from an external IP, so using a cell phone connection for example.

Djkáťo

@stephenw10 is that true even when I'm connecting from a "different network"? I use WAN on my pc, and the server is on OPT. Regardless, I just tested on my phone data and it still just times out. It should work, cuz when I try to reach for example {public_ip}:2283 it doesn't work, but 10.0.1.10:2283 works (I made a virtual eth connection for that vlan on my pc).

stephenw10

Yes, it will fail from any internal subnet.

Look at the state counters on the firewall rules you created. That shows if that are actually matching traffic.

Check the actual states in Diag > States when you are connecting from some external site. You should see the incoming state on OPT with NAT and an outgoing state on PUBLIC_OPT1.

Djkáťo

@stephenw10 said in how to separate two WAN connetions into home internet and work internet virtually:

You should see the incoming state on OPT with NAT and an outgoing state on PUBLIC_OPT1.

On my phone through data I tried connecting either {public IP}:{port}, or {domain}:{port}

Soo I'm not sure what I see, but I see some traffic, but it's CLOSED. I assume this is because the server is not responding? From what I read online unraid doesn't have any firewall, and accepts anything coming in. The ports are to real services, and when I copy paste the destination IP to those services I see them, just not publicly.
I guess I must be close. Any idea what this could be?

stephenw10

OK two things there.

The incoming state is shown on all interfaces and that's a problem for a non-default WAN because it need 'reply-to' tags in order to go back out the OPT.
Do you have an interface group or floating rules passing traffic there?

The closed state from the target really does look like it's just rejecting that traffic. A lot of devices will reject any traffic from outside their own subnet unless set to allow it specifically.

Djkáťo

@stephenw10 I don't have any floating rules or interface groups configured. I looked through all firewall settings, and don't see anything that would enforce something like that? Here's my entire config(stripped of certs, public IPs, passwords), maybe you'll find the offender: config-pfSense.lan-20250513135023.xml

I'll look again into how to allow unraid to accept those connections, but I already looked before and didn't find any settings regarding firewall or public access before.

Thank you for your time

stephenw10

What pfSense version is that running?

Djkáťo

@stephenw10 pfsense+ 24.11

Djkáťo

I'm kind of out of my depth here, I'd appreciate any tips on how to solve this issue :)
Anyone got any ideas?

stephenw10

Is it just the inbound traffic that's failing or are you also seeing outbound not using the correct WAN?

Try to ping the target server from pfSense itself. It should respond.

Then try again but set the ping source IP to some other interface. Does it still respond?

Djkáťo

@stephenw10

PUBLIC_OPT1(servers VLAN) > server works
OPT > server doesn't work
WAN > server doesn't work
Public IP > server doesn't work

Though I'm not sure if I'm using the wrong address? The interfaces' ips are in different subnets, so I don't know if they should be able to ping.

My server has access to the internet, ping 1.1.1.1 from the server works. I don't really know which WAN it's using though. I assume WAN and not OPT, because the only way I got the vlan to work is by keeping default vlan untagged and add vlan 10 as tagged. How could I force only vlan 10 to be what the server has access to?