Moved tunnel to pfsense, can't reach IP's exposed to the world

[jdcynical]

I am at a loss.

Old setup on a smoothwall VM (m0n0wall fork):

Interfaces: LAN, DMZ, WAN. WAN and DMZ in the same subnet

• IP6 tunnel setup on the WAN interface as a tunnel
• IP6 exposed addresses setup in their own tagged VLAN/subnet (call it outside) with the DMZ interface as the parent.
• WAN firewall rules set up to allow traffic into outside subnet, outside rules set to allow traffic out.
• VM stack is two hosts with LAN/DMZ interfaces, smoothwall VM runs on one of the two hosts as needed

New setup:

• pfsense 2.6.0-RELEASE, dedicated hardware (new install and HW, migrated from an ancient pfsense 1.2.x install. Cringe away, it was solid ). Three physical interfaces, LAN, DMZ, WAN4.
• GIF tunnel (WAN6) established to HE.net over WAN4, machine has been given a full reboot.
• Outside VLAN subnet using DMZ interface as parent.
• Outside subnet assigned an IP in the subnet.
• IP's on the previous setup have been changed.
• Inbound rules for the outside subnet setup in WAN6, outbound rules setup in outside.

What I see:

• I can ping the outside interface IP in the subnet from inside the LAN.

• I can not ping the outside interface IP in the subnet from inside the DMZ using the vlan interface.

• I can not ping the outside machine IP's in the subnet from inside the DMZ using the vlan interface (Destination unreachable: Address unreachable)

• I can not ping the assigned IP in the subnet from inside the DMZ using the default (DMZ) interface.

• I can ping the outside IP's from the DMZ (not surprising).

• I can not ping the outside IP's from the LAN (Destination unreachable: Address unreachable).

• Outside testing sites (DNS, SMTP, ping) can not reach the outside addresses.

• Outside testing (ping) can ping the tunnel local ip6 address and the interface IP.

• Switch has been verified that the outside vlan tag is present on the ports the DMZ interfaces use, all interfaces have the correct subnet mask assigned.

Example rules on WAN6:
Source of any, allow ICMP echoreq and rep in to outside net
Source of any, allow TCP connections to port 1234 to testmachine

And on the outside interface:
Source of any, allow ICMP echoreq and rep out
Source of testmachine, allow TCP connections over port 1234 to any

I'm not sure what I am missing.

The new pfsense install is obviously different with the tunnel being it's own interface, but as I understand it, the rules are set up in the right places and are similar to what was present and working on the smoothwall setup, and I do not see anything showing up in logging for the default block rule that would explain it. I almost feel that I'm missing something small and/or obvious, hiding in plain sight as it were.

And, of course, outbound on the LAN and DMZ interfaces works a treat.

Ideas?

[jdcynical]

And solved, kind of?

More troubleshooting with tcpdump, noticed that the neighbor solicitation packets from the outside subnet were not making it back to the parent interface on the pfsense machine. Turns out that Suricata was preventing them from going where they needed to go, never mind that no drops were showing up in the logs for this.

Since being able to use Suricata is one of the ideas behind this move, looks like it's time to dive deeper into the Suricata setup.

[Bob.Dig]

Hard to read, something is not working but also your setup is rather complicated with a tunnel and now you want to go deep in suricata, which will not make things easier.
Better start fresh with a simple setup and asks question about things you don't yet fully understand because you come from a different product.

[jdcynical]

That's the thing. I've had suricata running on the system with just the three physical interfaces for a while now with no problems, but once I got around to moving the tunnel so I could make use of suricata, things started getting weird.

Right now, I'm suspecting that it's going to come down more to a combo of pfsense/suricata not liking the use of tagged vlans with my particular configuration (I did see netmap_ring_reinit igb3, which happens to be the parent for the vlan at one point, causing traffic to stop flowing until the system was restarted, with a full reset to POST at the worst case).

This was partially me derping any not expecting inline mode to inspect the tagged packets, which I should have, and partially it apparently blocking the neighbor solicitation packets, which was totally unexpected, and resolved by disabling it on the parent interface.

I'm not totally adverse to moving the parent interfaces around, or moving the tagged vlan to something port based on the switch, which may also lead to cleaning up some, ahem, 'technical debt' in the layout of the local network space.

Regardless, I reserve the right to change my opinion as I look into this more. At this time, things are stable and the public facing server IP's are receiving traffic as expected.