Are the network settings and the gateway set correct on the destination host?

Check the local firewall and default gateway on that target computer then.

A Feature-Request for PfSense:

I jut configured Openstack to deploy an mtu using DHCP Option 26, but PFsense seems to ignore this one consequently.

dhcpdump -i tapb8659f7c-df
  TIME: 2017-06-29 13:22:00.059
    IP: 10.40.50.3 (fa:16:3e:2c:e0:61) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
    OP: 1 (BOOTPREQUEST)
HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
  XID: 11ac2ce2
  SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: fa:16:3e:2c:e0:61:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION:  53 (  1) DHCP message type        3 (DHCPREQUEST)
OPTION:  50 (  4) Request IP address        10.40.50.3
OPTION:  61 (  7) Client-identifier        01:fa:16:3e:2c:e0:61
OPTION:  12 (  9) Host name                xxxxxxxxx
OPTION:  55 (  9) Parameter Request List      1 (Subnet mask)
                                            28 (Broadcast address)
                                              2 (Time offset)
                                            121 (Classless Static Route)
                                              3 (Routers)
                                            15 (Domainname)
                                              6 (DNS server)
                                            12 (Host name)
                                            119 (Domain Search)

–-------------------------------------------------------------------------

TIME: 2017-06-29 13:22:00.059
    IP: 10.40.50.2 (fa:16:3e:63:19:c0) > 10.40.50.3 (fa:16:3e:2c:e0:61)
    OP: 2 (BOOTPREPLY)
HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
  XID: 11ac2ce2
  SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.40.50.3
SIADDR: 10.40.50.2
GIADDR: 0.0.0.0
CHADDR: fa:16:3e:2c:e0:61:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION:  53 (  1) DHCP message type        5 (DHCPACK)
OPTION:  54 (  4) Server identifier        10.40.50.2
OPTION:  51 (  4) IP address leasetime      86400 (24h)
OPTION:  58 (  4) T1                        43200 (12h)
OPTION:  59 (  4) T2                        75600 (21h)
OPTION:  1 (  4) Subnet mask              255.255.255.0
OPTION:  28 (  4) Broadcast address        10.40.50.255
OPTION:  15 ( 14) Domainname                openstacklocal
OPTION:  12 ( 15) Host name                host-10-40-50-3
OPTION:  3 (  4) Routers                  10.40.50.3
OPTION: 121 ( 14) Classless Static Route    20a9fea9fe0a2832  .....(2
                                            02000a283203    ...(2.
OPTION:  6 (  8) DNS server                xxxxxxxxxxxxx
OPTION:  26 (  2) Interface MTU            1450
–-------------------------------------------------------------------------

This shouldn't be ignored because it'll result in fragmented packets / incorrect checksums since OS itself adds ~50 Bytes to a VXLAN-paket anyway.

In 1:1 you can set the NAT for the whole subnet if you enter 172.16.0.0 at "External subnet IP" and at "Internal IP" select network and 172.16.100.0/24

It doesn't matter if this also includes IPs assigned to computers in group A, since you haven't add an IP alias for these addresses to WAN.

@Valley:

When I check my public IP address my dynamic Xfinity is shown - how can I setup up my Comcast static IP address as the default address?

One way is to bridge a third interface to your WAN port and put the static IP's on devices on that interface.

Other is to use VIP's and then port forwarding or 1:1 NAT to the intended devices.

https://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses

Not wanting to hijack the OP's thread but, yeah, it took quite a lot of web searching to find that solution - not really knowing what to look for.

Then, I had to find a workaround for the fact that the table is cleared on a firewall rule change.

Doktornotor pointed me to the shellcmd package and its "afterfilterchangeshellcmd" option.  I use that to call a tiny PHP script that checks whether the table is empty.  If it is, the connection to the sending host is restarted and the table is rebuilt.

Lots of fun :)

I think both agree on bad design.

@dotdash:

It appears you are not getting any routes from the peer router.
The fact that the pinger showed the remote router down seems to indicate the transit layer to 100.100.100.100 failed.
Is that IP provider-assigned, or did you obfuscate the real IP? That block is not a traditional private space, but is supposed to be used by service providers for their internal networks.

I've obfuscate the real ip.

The only way to do that is to get Multilink PPP from your ISP.

Maybe with pfsense it is hard but it is easy to setup using a layer 3 switch.  All you have to do is point the local traffic to the layer 3 switch.  It knows where everything is and will route or switch to the device.  Nothing hard.  It is a good way to bring a layer 3 switch into the fold without disrupting normal operations.

Check the system log for hints.

Just set a static route for 192.168.45.0/24 on the Exchange pointing to pfSense:

Replace "<gateway>" with the LAN IP of pfSense within 192.168.44.0/24.</gateway>

If it is critical to get this working I suggest you buy some support hours.  Contact support to discuss first.  For done things which were very complicated this is what I have done.

It is been more than a year but I eventually gave up on this.  I think using openvpn might be easier but not sure.  I wish you luck with this!

Not sure what this document refers to but it is innacurate.

You must configure 3G/LTE setting on interface if using as a wan connection or at least multi wan.  Directions say not to make any changes to the interface but you have to in order to make it work.

https://doc.pfsense.org/index.php/Configuring_3G_modems

I will reply to myself as I've found the problem so anyone with the same hardware can use this as a solution.

The problem was that the VLAN3 on the WDR3600 was incorrectly set to the interface eth1.3 and on the WDR-3600 the switch is on the eth0 interface, so the VLAN3 had to be set to eth0.3 and voila! everything works!!!

The only thing that should be added to this setup are the firewall rules, that I have set as the following screenshot shows.

VLAN3_fw_rules.png_thumb
VLAN3_fw_rules.png

@kpa:

No, that's not policy routing at all. What you need is a normal static route on pfSense with the WAN address of the inner router as the target for the traffic that going to the LAN of the inner firewall. Static routes are set at System->Routing->Static Routes.

Actually thats exactly what I try to admit :)

@kpa:

Additionally I hope you're using a transit network between pfSense and inner firewall with no hosts on it? Otherwise you have a broken network setup with asymmetric routing.

Yeah, a nice firewall transit network :)

My understanding of policy based routing, comes from Barracuda and Juniper. There it works on the routing and not on firewall level. So both ways are possible, in- and outbound. Posted a screen as an example.

barra_pbr.PNG
barra_pbr.PNG_thumb
barra_pbr2.PNG
barra_pbr2.PNG_thumb