Source routing problem

crtek · Dec 10, 2008, 3:26 PM

hi,
like I red in this post http://forum.pfsense.org/index.php/topic,9736.msg55103.html#msg55103
incoming WAN connections should leave pfsense from the interface that came in.
But my box is routing traffic according to the routing table even if it should not.

so this is my network:

WAN–ips x.x.x.162-190/27
gw x.x.x.161

WAN2 --ip x.x.6.177/16
gw x.x.0.1

LAN --192.168.0.0/24

the problem is when there is a connection to my ips from the WAN2 range of ips,
the traffic comes in from WAN, reaches my servers on LAN, and then leaves on WAN2.
(have checked with tcpdump on all 3 interfaces), but all other connections (not from WAN2 range)
works well.

have I misconfigured something?

crtek · Dec 11, 2008, 8:50 AM

if someone would like to take a look to my dumps

em0 is the WAN interface with ip y.y.81.190/27 that recieves the syn package from a host
which is on the same subnet as WAN2 with ip x.x.6.177/16

tcpdump -ni em0 | grep x.x.110.216

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
07:20:23.825672 IP y.y.110.216.16232 > x.x.81.190.3050: S 2983556618:2983556618(0) win 65535 <mss 1460,nop,nop,sackok="">07:20:26.698932 IP y.y.110.216.16232 > x.x.81.190.3050: S 2983556618:2983556618(0) win 65535 <mss 1460,nop,nop,sackok="">07:20:32.715480 IP y.y.110.216.16232 > x.x.81.190.3050: S 2983556618:2983556618(0) win 65535 <mss 1460,nop,nop,sackok="">07:20:50.671860 IP y.y.110.216.16233 > x.x.81.190.3050: S 3571933983:3571933983(0) win 65535 <mss 1460,nop,nop,sackok="">07:20:53.605189 IP y.y.110.216.16233 > x.x.81.190.3050: S 3571933983:3571933983(0) win 65535 <mss 1460,nop,nop,sackok="">07:20:59.620756 IP y.y.110.216.16233 > x.x.81.190.3050: S 3571933983:3571933983(0) win 65535 <mss 1460,nop,nop,sackok="">07:21:24.149863 IP y.y.110.216.16234 > x.x.81.190.3050: S 1033442227:1033442227(0) win 65535 <mss 1460,nop,nop,sackok="">07:21:27.183256 IP y.y.110.216.16234 > x.x.81.190.3050: S 1033442227:1033442227(0) win 65535 <mss 1460,nop,nop,sackok="">07:21:33.199196 IP y.y.110.216.16234 > x.x.81.190.3050: S 1033442227:1033442227(0) win 65535 <mss 1460,nop,nop,sackok="">^C30564 packets captured
30665 packets received by filter
0 packets dropped by kernel

then em3 LAN interface forwards the package to the server 192.168.0.4 which then sends the ack to x.x.81.190
and it shoud live pfsense from em0 but it leaves on em1 which is the WAN2 interface

tcpdump -ni em1 | grep x.x.110.216

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
07:20:23.825963 IP 192.168.0.4.3050 > x.x.110.216.16232: S 3570009020:3570009020(0) ack 2983556619 win 16384 <mss 1460,nop,nop,sackok="">07:20:27.482623 IP 192.168.0.4.3050 > x.x.110.216.16232: S 3570009020:3570009020(0) ack 2983556619 win 16384 <mss 1460,nop,nop,sackok="">07:20:36.232464 IP 192.168.0.4.3050 > x.x.110.216.16232: S 3570009020:3570009020(0) ack 2983556619 win 16384 <mss 1460,nop,nop,sackok="">07:20:50.672153 IP 192.168.0.4.3050 > x.x.110.216.16233: S 958877085:958877085(0) ack 3571933984 win 16384 <mss 1460,nop,nop,sackok="">07:20:54.279243 IP 192.168.0.4.3050 > x.x.110.216.16233: S 958877085:958877085(0) ack 3571933984 win 16384 <mss 1460,nop,nop,sackok="">07:21:02.935453 IP 192.168.0.4.3050 > x.x.110.216.16233: S 958877085:958877085(0) ack 3571933984 win 16384 <mss 1460,nop,nop,sackok="">07:21:24.150192 IP 192.168.0.4.3050 > x.x.110.216.16234: S 452266070:452266070(0) ack 1033442228 win 16384 <mss 1460,nop,nop,sackok="">07:21:27.747754 IP 192.168.0.4.3050 > x.x.110.216.16234: S 452266070:452266070(0) ack 1033442228 win 16384 <mss 1460,nop,nop,sackok="">07:21:36.060205 IP 192.168.0.4.3050 > x.x.110.216.16234: S 452266070:452266070(0) ack 1033442228 win 16384 <mss 1460,nop,nop,sackok="">^C209 packets captured
210 packets received by filter
0 packets dropped by kernel

why are those packets leaving on the wan2 interface?</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss>

crtek · Dec 19, 2008, 10:25 AM

finaly I found a solution

so pfsense does not create correct pf rules on wan when there is another
wan connection
basicaly it does create those rules (from /tmp/rules.debug)

User-defined aliases follow

User-defined rules follow

pass in quick on $wan from any to any keep state label "USER_RULE"
pass in quick on $OPT1 reply-to (le2 y.y.y.y) from any to any keep state label "USER_RULE"

but the rule should be like this

User-defined rules follow

pass in quick on $wan reply-to (le0 x.x.x.x) from any to any keep state label "USER_RULE"
pass in quick on $OPT1 reply-to (le2 y.y.y.y) from any to any keep state label "USER_RULE"

so I had to make a change in the file /etc/inc/filter.inc
on the line 1581 from this
if(($rule['gateway'] == "") and ($ri != "") and ($rg != "") and (stristr($rule['interface'],"opt") == true)) {
to this
if(($rule['gateway'] == "") and ($ri != "") and ($rg != "")) {

so this way rules are created correctly