"WRT-54GL"

My gawd.. That hardware is what 10 years old.. How could you be still using G for your wifi??  You have to be faster just using your phone cell connection.  G would be like dialup..

Dude time to update… really!!!

I can hear it now as people come over to the place to use your wifi
Hey buddy whats your wifi password, oh shit G - no thanks I think I will just use my data plan, I want the info like this week ;)

Surely, this will be the best way. So you have both WANs on pfSense and the box can manage a failover.

And also a new version of pfSense will be recommended.  :)
My first version was 1.3, but only for play around.

what is the mask on this 10.90-91 network? (private network) and the lan network.. Its quite possible they overlap if your using say default /8 of a 10 network.

Also what are the rules on your lan interface of pfsense if your dong any sort of policy routing out a specific gateway you have to have rules that allow the traffic to the other opt1 network before sending out a gateway, etc.

Also what is the other vm - is it running any sort of software firewall, windows for example out of the box will block icmp from other than its local network.

Only 3 interfaces.

WAN - PPPoE - Ip Address 123.321.123.320 [Routed subnet of 123.321.123.321/28 is here]
LAN - 10.1.1.1/24
LAN2 - 123.321.123.322/28 [DHCP runs here serving the remainder of the /28]

Any IP in LAN2 can connect to the main WAN IP [port forwarding to the LAN works]

Any machine in the LAN cannot connect to any of the /28 IPs.

As for what I have done, the above is the interface config, I have not setup any virtual IPs.
I have created a rule on LAN2 to allow all traffic to pass.

As I write this I have answered my own question. I created a rule on LAN to allow all traffic from LAN destined for LAN2 Subnet to the default gateway and that seems to have solved it.
My outbound LAN rules are not allow all so that must be where the block was.

Asked and answered. Thanks for all your help so far.

I proposed this type of setup myself a couple times on this forum with the same non-existant replies. Every so often I experiment with setup and have yet to find a workable solution to do this without NAT/Firewall enabled.

I am confused why VLAN is being used if these networks are indeed separated and use different LAN interfaces. Do they share the same switch?

A diagram would help more than anymore confusing comments.

any 20-100 dollar router does/can do NAT (network address translation).
there are, as far as i know, no consumer routers available, that DONT do nat out of the box.

i don't have brand to recommend

Well I'm here to reply itself to this quest. After some trial & error I get all to work as wanted.
Be sure to follow this setup , if anyone wants to reproduce:

pfSense box
WAN interface ( IPv4 DHCP lease from other ISP)
LAN interface (192.168.0.0/24) and add two more VLANS and one PPPoE connection as follow:
add VLAN 22 to LAN interface (192.168.50.0/24) Private users
add VLAN 33 to LAN interface (192.168.0.0/24) Guests
add PPPoE to LAN interface (for get my public ADSL IP)

LAN port <> ethernet cable <> managed switch "port 1"

5 Port Managed switch (my TL-SG105E)

port 1 Tagged/Trunk ; VLAN 22 (Private); VLAN 33 (Guests) to pfsense LAN interface
port 2 Tagged/Trunk ; VLAN 22 (Private); VLAN 33 (Guests) to Unifi AP LR LAN interface
port 3 Tagged/Trunk ; VLAN 33 (Guests) to other tp-link AP (multissid VLAN enabled)
port 4 Tagged/Trunk ; VLAN 33 (Guests) to other tp-link AP (multissid VLAN enabled)
port 5 Disabled

VLAN 22 Not member port: 3,4,5
VLAN 33 Not member port: 5

VLAN Untagged/Access port not needed because all my devices connected , supports the VLAN Tagging over own ethernet port.

managed switch "port 2" <> ethernet cable <> Unifi AP LR ethernet port

Unifi AP LR box
Managed with own software in default subnet.
Setup with multi SSID as follow;

-VLAN 1 (default 192.168.0.0/24 subnet) and SSID "AWAN" for PPPoE and managing option. With WPA2. Can talk only with AP client associated for this purpose. (follow next)

-VLAN22 (Private users 192.168.50.0/24 subnet)  With WPA2. Can talk with private clients of course when DHCP server is running into pfs box grab it own lease.

-VLAN 33 (Guests 192.168.60.0/24 subnet) Open. Can talk with Guests trought captive portal and share only internet access, not allow to see everything on the internal network.

UNIFI AP LR<> wifi ssid AWAN wpa2 <> TL-WA701ND

TL-WA701ND client mode
This is associated with wpa2 key over Unifi AP base station ssid AWAN, when connect I see only my default subnet on VLAN1 of course.
Managed IP set to 192.168.0.x/24

TL-WA701ND<> ethernet cable <>DSL modem/router (ISP provider)
This device needing special setup to work with PPPoE directly from pfs box.
-DISABLE DHCP SERVER
-Set own static IP to 192.168.0.2 for managing purpose.
-The internal modem interface (DSL line) is "Bridged" with own ethernet port. This allow pfs box to "dial" trought PPPoE
Also not any PPPoE "user" connection is generated by this modem/router itself, Only allow "service" connection eventually maked by provider for diagnostic its device.

Finish! well now is possible to make this special setup, amazing! will put with one AP in multi ssid to "route" any kind of "traffic your need" for internal network segment and plus PPP's connection. Without need to connect ISP modem/router directly on dedicated pfs box interface. Well done. Hope this be useful for others, fell free to ask for more details if your need. Goodbye folks.

Multiple NICs in the same wire on the WAN side and with IP addresses from the same range would be quite problematic, use virtual IPs instead.

If possible ask your ISP to offer those 5 IPs (I take the subnet is a /29?) as a routed subnet with a transit network on the WAN side of pfSense, that would be the optimal solution.

Maybe the outbound NAT isn't working properly.

I don't know if pfSense adds the outbound NAT rule correctly for vLANs or maybe your outbound NAT is set for manual rule generation. Check the rules in Firewall > NAT > Outbound.

Thanks.

Yes.  Purchased a support ticket and was told to purchase a cell modem that used ethernet rather than usb.  I was trying to migrate from a peplink multi-wan router.  I guess I could use it as a front end to the pfSense box if I want to continue down this path.

Thank you.

Thanks heaps.

I have two good DNS servers for each gateway in System > General.

I decided not to do load-balance due to the issues with change of IP address causing issues with individual sessions, even with sticky connections enabled. I instead set up two gateway groups, one with each gateway as Tier 1 and the other Tier 2. I then use one of the groups in half of my LANs and the other in the remaining LANs.

The firewall rules ended up still being very tidy. I simply put an allow RFC1918 rule with default routing above the last rule which specifies the gateway group. That way, no matter how many LANs and/or VPN connections I add in the future, they will work properly.

Any re-thought about adding this? while sending the fail signal if multiple points are down could be useful, we really want it to for Historical Quality reporting, Packet Loss, Latency… really like the new easy to read Std. deviation in 2.3.. nice

We always have multiple points of reference, in EU ISP network, Our data center, Our ISP and all possible interconnects between. Having this historical info in EU end points would be very helpful in a lot of ways.

anyway the first step in getting multiple fail confirm, is to have multiple monitoring!!!!