Bug in pf on freebsd 7
-
I had a problem with creation of repl-to rules on wan in 1.2,
you can see my post here http://forum.pfsense.org/index.php/topic,13024.0.html.Then I tried to upgrade to 1.2.1rc4
and I can see that those rules on wan are created correctly in this releaseUser-defined rules follow
pass in quick on $wan reply-to (le0 x.x.x.x) from any to any keep state label "USER_RULE"
pass in quick on $OPT1 reply-to (le2 y.y.y.y) from any to any keep state label "USER_RULE"but my pfsense did not reply back to source addresses like it should so i did some testing
I created a testing environment with 5 pcs
pc 1 ip 20.50.10.106/16 gw 20.0.0.101
pc2 ip1 10.50.0.1/16 gw 10.0.0.2
ip2 20.50.10.105/16pfsense wan ip 10.50.0.2/16 gw 10.0.0.1
lan ip 192.168.233.2
opt1 20.50.0.1/16 gw 20.0.0.3pc3 ip 20.50.0.3/16 gw 20.0.0.1
pc4 ip 192.168.233.1/24 gw 192.168.233.2
on pc4 i run firebird server just to have some service runningI loaded my custom pf.conf
that looks like this (only 4 rules)rdr on le0 proto tcp from any to 10.50.0.2 port { 3050 } -> 192.168.233.1
rdr on le2 proto tcp from any to 20.50.0.2 port { 3050 } -> 192.168.233.1pass in quick on le0 reply-to ( le0 10.0.0.1 )from any to any keep state
pass in quick on le2 reply-to ( le2 20.0.0.1 ) from any to any keep stateMy test went like this
1. pfsense machine running pfsense 1.2
-loaded rules with this command pfctl -F all -f /root/pf.conf
-from pc1 tried to acces my service on pc4 and there was no problems
pc1 issued command:
#telnet 10.50.0.2 3050
Trying 10.50.0.2…
Connected to 10.50.0.2pfsense issued command:# tcpdump -ni le0 | grep 3050
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
10:54:09.671074 IP 20.50.0.1.2657 > 10.50.0.2.3050: P 1712515217:1712515222(5) ack 804699406 win 16384 <nop,nop,timestamp 0="" 2445543632="">10:54:09.671276 IP 10.50.0.2.3050 > 20.50.0.1.2657: F 1:1(0) ack 5 win 65530 <nop,nop,timestamp 1176208="" 2445543632="">10:54:09.671490 IP 20.50.0.1.2657 > 10.50.0.2.3050: . ack 2 win 16384 <nop,nop,timestamp 1176208="" 2445543632="">10:54:09.671941 IP 20.50.0.1.2657 > 10.50.0.2.3050: F 5:5(0) ack 2 win 16384 <nop,nop,timestamp 1176208="" 2445543632="">10:54:09.672150 IP 10.50.0.2.3050 > 20.50.0.1.2657: . ack 6 win 65530 <nop,nop,timestamp 1176208="" 2445543632="">10:54:11.382724 IP 20.50.0.1.7338 > 10.50.0.2.3050: S 2478544458:2478544458(0) win 16384 <mss 0="" 3331861149="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">10:54:11.382864 IP 10.50.0.2.3050 > 20.50.0.1.7338: S 2788614409:2788614409(0) ack 2478544459 win 65535 <mss 0="" 1460,nop,wscale="" 0,nop,nop,timestamp="" 0,nop,nop,sackok="">10:54:11.383167 IP 20.50.0.1.7338 > 10.50.0.2.3050: . ack 1 win 16384 <nop,nop,timestamp 0="" 3331861149="">10:54:14.057866 IP 20.50.0.1.7338 > 10.50.0.2.3050: P 1:6(5) ack 1 win 16384 <nop,nop,timestamp 0="" 3331861154="">10:54:14.058088 IP 10.50.0.2.3050 > 20.50.0.1.7338: F 1:1(0) ack 6 win 65530 <nop,nop,timestamp 1176251="" 3331861154="">10:54:14.058325 IP 20.50.0.1.7338 > 10.50.0.2.3050: . ack 2 win 16384 <nop,nop,timestamp 1176251="" 3331861154="">10:54:14.058559 IP 20.50.0.1.7338 > 10.50.0.2.3050: F 6:6(0) ack 2 win 16384 <nop,nop,timestamp 1176251="" 3331861154="">10:54:14.058631 IP 10.50.0.2.3050 > 20.50.0.1.7338: . ack 7 win 65530 <nop,nop,timestamp 1176251="" 3331861154=""># netstat -nrf inet
Routing tablesInternet:
Destination Gateway Flags Refs Use Netif Expire
default 10.50.0.1 UGS 0 45 le0
10.50/16 link#1 UC 0 0 le0
10.50.0.1 00:0c:29:32:bc:0c UHLW 2 70 le0 399
20.50/16 link#3 UC 0 0 le2
20.50.0.3 00:0c:29:6c:39:c3 UHLW 1 70 le2 1143
127.0.0.1 127.0.0.1 UH 0 28 lo0
192.168.233 link#2 UC 0 0 le1
192.168.233.1 00:50:56:c0:00:01 UHLW 1 104 le1 9018801 packets received by filter
1839 packets dropped by kernel
All well than2. pfsense machine running pfsense 1.2.1rc4
-loaded rules with this command pfctl -F all -f /root/pf.conf
-from pc1 tried to acces my service on pc4 and there was no problems
pc1 issued command:
#telnet 10.50.0.2 3050
Trying 10.50.0.2…on pfsense i then dumped traffic on all interfaces
WAN interface
pfSense:~# tcpdump -ni le0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
12:12:39.799344 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957288991="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:42.908623 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289003="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:49.101611 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289027="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:55.073943 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467175="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:58.183807 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467187="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:04.420355 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467211="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:17.725867 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467259="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:27.849623 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539422="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:30.906400 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539434="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:37.122792 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539458="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:39.448390 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433760="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:42.544310 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433772="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:48.789239 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433796="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:56.349084 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316660="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:59.470146 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316672="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:05.642091 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316696="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:18.934221 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316744="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">LAN interface
pfSense:~# tcpdump -ni le0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
12:12:39.799344 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957288991="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:42.908623 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289003="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:49.101611 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289027="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:55.073943 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467175="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:58.183807 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467187="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:04.420355 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467211="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:17.725867 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467259="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:27.849623 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539422="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:30.906400 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539434="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:37.122792 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539458="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:39.448390 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433760="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:42.544310 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433772="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:48.789239 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433796="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:56.349084 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316660="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:59.470146 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316672="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:05.642091 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316696="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:18.934221 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316744="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">OPT1 interface
pfSense:~# tcpdump -ni le2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le2, link-type EN10MB (Ethernet), capture size 96 bytes
12:12:55.074689 arp who-has 20.50.0.1 tell 20.50.0.2
12:12:56.606935 arp who-has 20.50.0.1 tell 20.50.0.2
12:12:58.184179 arp who-has 20.50.0.1 tell 20.50.0.2
12:12:59.792644 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:04.420392 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:17.725956 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:19.275754 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:22.400258 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:27.849639 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:29.298290 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:30.906739 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:32.410233 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:37.122924 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:39.448504 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:41.004837 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:42.544985 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:44.186741 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:48.789311 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:56.349534 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:57.939083 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:59.470493 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:01.029200 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:05.642239 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:18.934258 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:20.435657 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:23.509403 arp who-has 20.50.0.1 tell 20.50.0.2#netstat -nrf inet
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.50.0.1 UGS 0 13993 le0
10.50.0.0/16 link#1 UC 0 0 le0
10.50.0.1 00:0c:29:32:bc:0c UHLW 2 85 le0 840
20.50.0.0/16 link#3 UC 0 31 le2
20.50.0.3 00:0c:29:6c:39:c3 UHLW 1 80 le2 1166
127.0.0.1 127.0.0.1 UH 0 1699 lo0
192.168.233.0/24 link#2 UC 0 0 le1
192.168.233.1 00:50:56:c0:00:01 UHLW 1 6932 le1 1140I tried the same config with openbsd, freebsd 6.2 and freebsd 7.0 (in place of pfsense)
Both freebsd 7.0 and pfsense 1.2.1rc4 had the same problem all other systems worked as expected.
I forgot to mention that this only happens if the IP trying to access WAN is on the same subnet as OPT1 in my case /16.</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></mss></mss></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp> -
Can you draw a network diagram complete with ips and all and reformulate this?!
There is a fix for allowing that reply-to on WAN to not break certain things and it was not meant for such situations and that might break it.
The network diagram is just to see if this is something that needs to be supported or not or it is just something that needs attention before final release. -
PC1 PC2 PFSENSE PC3
ip 20.50.10.106/16 <–----> ip2 20.50.10.105/16--|--ip1 10.50.0.1/16<----->wan ip 10.50.0.2/16--|--opt1 20.50.0.1/16 <-------> ip 20.50.0.3/16
gw 20.50.10.105 gw 10.50.0.2 gw 10.50.0.1 | gw 20.50.0.3 gw 20.50.0.1
|
|
lan ip 192.168.233.2/24
|
|
|
PC4
ip 192.168.233.1/24
gw 192.168.233.2