Strange behavior with ICMP

SM7I

Hi everyone !

I have a strange issue with my pfSense 2.2.6. I have the following setup:

WAN (fxp0), LAN1 (ste0), LAN2 (ste1)

WAN uses default gateway from ISP and that´s also the exitway for LAN1 with hide NAT.

LAN2 uses an external router to access an alternate way to the Internet. The rule for LAN2 is configured with this external router. LAN2 is also configured with static public IP via the alternate way to Internet (I´m the ISP there and routes a chunk of a C class network to the pfSense via the external router)

LAN2 uses pure L3 and no NAT at all.

LAN2 workstation (also with public static IP) can ping and surf the Internet as expected via the external router.

LAN1 works also as expected, but uses the default gateway via WAN.

Everything fine this far.

The strange thing is this: When I traceroute or ping from a totally external computer on the Internet to my workstation on LAN2 it gets all the way to the pfSense, and all the way to the LAN2 workstation (checked using netstat).

If I look at the packetcapture (tcpdump) on the pfSense at the same time I can see that ICMP request is coming in from the totally external computer on the Internet correctly on the LAN2 interface and is passed on to LAN2 workstation. The reply, however, is not using LAN2 external router to reply, but uses WAN interface instead.

Normally in my world a reply goes out the same way a request came in….

This causes assymetric routing and that´s really unwanted from my point of view.

When I do the opposite, ie. tracerouting from LAN2 workstation to the totally external computer on the Internet the trafficflow is totally correct and uses the external router.

I´ve tried anything I could think of to correct this behaviour, but I´m stuck. I cannot for my life get this to work even though I´ve worked with firewalls for 20 years.....

Anyone care to explain why I get this behaviour and a possible solution ?

Regards from Johan

johnpoz

"LAN2 uses an external router to access an alternate way to the Internet."

If it has a way to the internet then its not really a lan interface to pfsense but a wan interface..

Also if if your going to put workstations on a transit network, then your going to have to put host routing on them or yes your going to have asynchronous routing issues.

You would think someone working with firewalls for 20 years would understand basic routing principles ;) See attached… Any time you connect 2 routers with a network that becomes a transit network. If host on this network gateway is A, and A has to send traffic to B to get to somewhere else.. When that traffic comes back in B - the dest IP on the transit network is locally attached and it has no need to send traffic back to A.. So you have asynchronous issue.

If you want to use hosts on a transit, then you would need to have routes on the host to use B when need to get to network B is attached to that doesn't go back to A..

The normal solution is to not put hosts on a transit network..

transitnetworkproblem.png_thumb

SM7I

Hi !

Thanks for answering.

OK, I feel like I have to clarify a few things for you to understand the setup a bit more :)

First of all, the LAN2 workstation is not situated on the transit network, but rather on a network "behind" the firewall so to speak…

OK, let me try to explain:

The pfSense sure is connected to a transit network in this way: a TUN interface is created and established contact with the router on another site. Local (pfsense) side TUN of the transit has IP 172.16.20.x and the router TUN has 172.16.20.y

We agree that this is a transit, yes. Furthermore, the remote router and pfSense runs RIP over the transit to exchange specific information (remote router injects OLSR routes (per se the kernel routes) into RIP and sends them to pfSense, whereas pfSense RIP back it´s LAN2 network for the remote router to inject into OLSR.

LAN2 workstations or whatever is placed on the network will -never- use the pfSense WAN interface as egress, only the remote router via transit network and this works fine as long as this network is the initiator. When acting as responder pfSense points the traffic towards WAN interface for some strange reason.

Hope this clarifies. We are far past basic routing principles in this case....Still stating my 20 yrs tho :)

johnpoz

How about you draw it!!! So we are all clear

"not situated on the transit network, but rather on a network "behind""

If it is a on a network between 2 routers then it is on a transit network..

Oh so you do agree "We agree that this is a transit"

Traffic to any box on a transit is going to cause asynchronous unless that host has routes to tell it with router to use to get to that network.. You should never have hosts on a transit network!! Its really that simple.

SM7I

Hi !

OK, still not being clear enough apparantly :) Maybe because english is not my primary language, but I´ll try again.

Now I´ve drawn a nice Visiodraft schematics on the essential parts of the setup.

Please look at this. The workstation is not situated on the transit (maybe we have different opinions on what is a transit. Over here the TUN is clearly a transit). As you clearly can see, the workstation is behind the firewall.

The transit is purely between the pfSense itself and the TUN router (routing applied by RIP).

All else is just as I described before regarding traffic behavior, rules and routes.

The traffic from WS LAN2 is routed perfectly the correct way as initiator, ie over the TUN, but as responder pfSense chooses to send it out on WAN. Meaning that a computer on the public internet sends it requests via the alternate Internet, through the TUN router, via transit, through pfSense and then it reaches the WS LAN2, but WS LAN2 response is going out the WAN IF.

If I ping the computer on the Internet from WS LAN2 I get response.

Ritning1.jpg_thumb

johnpoz

So you have a routed /29 to this network behind pfsense..

Why would traffic come in on you dynamic wan destined for your /29??? That should never happen. As to psfense send traffic that comes in your static wan interface via the transit tunnel to psfense.. Why would it ever send traffic out the wan.. Unless you have gateway setup on the rules that the /29.. your rules on your /29 should be set to always send traffic back out the tun gateway.. Since that is the only place that traffic for this routed /29 should come in.

SM7I

Hi !

I just got it working actually….accidentially I would say...

I was trying to figure out why all this happened so I temporarily removed the 0.0.0.0/0 route (default) via the normal DSL gateway.

I instead added default route via tun0

Ofcourse this led to traffic flowing as expected following the default route over tun0, even the replies...

Now the really strange this happens :) I wanted to reset the default gateway by up/downing the WAN fxp0 interface and regained the IP for default gateway, BUT, and here´s the strange thing....everything still works as expected.....strange I thought and checked the routingtable (I was expecting traffic to fail in reply as I´ve written above)

What I found bugs me alot...

Destination Gateway Flags Netif Expire
default 10.42.43.254 UGS tun0

The default route is OK via 10.42.43.254 as this is the DSL router, but look at the Netif for this.....

I really have no answer to what really is happening other than I am happy everything now works as I wanted....In theory this should not work at all...

Can you explain this ?