Bug in pf on freebsd 7

crtek

I had a problem with creation of repl-to rules on wan in 1.2,
you can see my post here http://forum.pfsense.org/index.php/topic,13024.0.html.

Then I tried to upgrade to 1.2.1rc4
and I can see that those rules on wan are created correctly in this release

User-defined rules follow

pass in quick on $wan reply-to (le0 x.x.x.x) from any to any keep state label "USER_RULE"
pass in quick on $OPT1 reply-to (le2 y.y.y.y) from any to any keep state label "USER_RULE"

but my pfsense did not reply back to source addresses like it should so i did some testing

I created a testing environment with 5 pcs

pc 1 ip 20.50.10.106/16 gw 20.0.0.101

pc2 ip1 10.50.0.1/16 gw 10.0.0.2
ip2 20.50.10.105/16

pfsense wan ip 10.50.0.2/16 gw 10.0.0.1
lan ip 192.168.233.2
opt1 20.50.0.1/16 gw 20.0.0.3

pc3 ip 20.50.0.3/16 gw 20.0.0.1

pc4 ip 192.168.233.1/24 gw 192.168.233.2
on pc4 i run firebird server just to have some service running

I loaded my custom pf.conf
that looks like this (only 4 rules)

rdr on le0 proto tcp from any to 10.50.0.2 port { 3050 } -> 192.168.233.1
rdr on le2 proto tcp from any to 20.50.0.2 port { 3050 } -> 192.168.233.1

pass in quick on le0 reply-to ( le0 10.0.0.1 )from any to any keep state
pass in quick on le2 reply-to ( le2 20.0.0.1 ) from any to any keep state

My test went like this
1. pfsense machine running pfsense 1.2
-loaded rules with this command pfctl -F all -f /root/pf.conf
-from pc1 tried to acces my service on pc4 and there was no problems
pc1 issued command:
#telnet 10.50.0.2 3050
Trying 10.50.0.2…
Connected to 10.50.0.2

pfsense issued command:# tcpdump -ni le0 | grep 3050
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
10:54:09.671074 IP 20.50.0.1.2657 > 10.50.0.2.3050: P 1712515217:1712515222(5) ack 804699406 win 16384 <nop,nop,timestamp 0="" 2445543632="">10:54:09.671276 IP 10.50.0.2.3050 > 20.50.0.1.2657: F 1:1(0) ack 5 win 65530 <nop,nop,timestamp 1176208="" 2445543632="">10:54:09.671490 IP 20.50.0.1.2657 > 10.50.0.2.3050: . ack 2 win 16384 <nop,nop,timestamp 1176208="" 2445543632="">10:54:09.671941 IP 20.50.0.1.2657 > 10.50.0.2.3050: F 5:5(0) ack 2 win 16384 <nop,nop,timestamp 1176208="" 2445543632="">10:54:09.672150 IP 10.50.0.2.3050 > 20.50.0.1.2657: . ack 6 win 65530 <nop,nop,timestamp 1176208="" 2445543632="">10:54:11.382724 IP 20.50.0.1.7338 > 10.50.0.2.3050: S 2478544458:2478544458(0) win 16384 <mss 0="" 3331861149="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">10:54:11.382864 IP 10.50.0.2.3050 > 20.50.0.1.7338: S 2788614409:2788614409(0) ack 2478544459 win 65535 <mss 0="" 1460,nop,wscale="" 0,nop,nop,timestamp="" 0,nop,nop,sackok="">10:54:11.383167 IP 20.50.0.1.7338 > 10.50.0.2.3050: . ack 1 win 16384 <nop,nop,timestamp 0="" 3331861149="">10:54:14.057866 IP 20.50.0.1.7338 > 10.50.0.2.3050: P 1:6(5) ack 1 win 16384 <nop,nop,timestamp 0="" 3331861154="">10:54:14.058088 IP 10.50.0.2.3050 > 20.50.0.1.7338: F 1:1(0) ack 6 win 65530 <nop,nop,timestamp 1176251="" 3331861154="">10:54:14.058325 IP 20.50.0.1.7338 > 10.50.0.2.3050: . ack 2 win 16384 <nop,nop,timestamp 1176251="" 3331861154="">10:54:14.058559 IP 20.50.0.1.7338 > 10.50.0.2.3050: F 6:6(0) ack 2 win 16384 <nop,nop,timestamp 1176251="" 3331861154="">10:54:14.058631 IP 10.50.0.2.3050 > 20.50.0.1.7338: . ack 7 win 65530 <nop,nop,timestamp 1176251="" 3331861154=""># netstat -nrf inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.50.0.1 UGS 0 45 le0
10.50/16 link#1 UC 0 0 le0
10.50.0.1 00:0c:29:32:bc:0c UHLW 2 70 le0 399
20.50/16 link#3 UC 0 0 le2
20.50.0.3 00:0c:29:6c:39:c3 UHLW 1 70 le2 1143
127.0.0.1 127.0.0.1 UH 0 28 lo0
192.168.233 link#2 UC 0 0 le1
192.168.233.1 00:50:56:c0:00:01 UHLW 1 104 le1 901

8801 packets received by filter
1839 packets dropped by kernel
All well than

2. pfsense machine running pfsense 1.2.1rc4
-loaded rules with this command pfctl -F all -f /root/pf.conf
-from pc1 tried to acces my service on pc4 and there was no problems
pc1 issued command:
#telnet 10.50.0.2 3050
Trying 10.50.0.2…

on pfsense i then dumped traffic on all interfaces
WAN interface
pfSense:~# tcpdump -ni le0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
12:12:39.799344 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957288991="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:42.908623 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289003="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:49.101611 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289027="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:55.073943 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467175="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:58.183807 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467187="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:04.420355 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467211="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:17.725867 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467259="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:27.849623 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539422="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:30.906400 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539434="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:37.122792 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539458="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:39.448390 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433760="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:42.544310 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433772="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:48.789239 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433796="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:56.349084 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316660="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:59.470146 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316672="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:05.642091 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316696="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:18.934221 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316744="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">LAN interface
pfSense:~# tcpdump -ni le0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
12:12:39.799344 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957288991="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:42.908623 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289003="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:49.101611 IP 20.50.0.1.17797 > 10.50.0.2.3050: S 3153139180:3153139180(0) win 16384 <mss 0="" 2957289027="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:55.073943 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467175="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:12:58.183807 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467187="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:04.420355 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467211="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:17.725867 IP 20.50.0.1.39691 > 10.50.0.2.3050: S 121285515:121285515(0) win 16384 <mss 0="" 1589467259="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:27.849623 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539422="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:30.906400 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539434="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:37.122792 IP 20.50.0.1.37833 > 10.50.0.2.3050: S 1726406080:1726406080(0) win 16384 <mss 0="" 1700539458="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:39.448390 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433760="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:42.544310 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433772="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:48.789239 IP 20.50.0.1.1411 > 10.50.0.2.3050: S 1221977032:1221977032(0) win 16384 <mss 0="" 2800433796="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:56.349084 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316660="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:13:59.470146 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316672="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:05.642091 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316696="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">12:14:18.934221 IP 20.50.0.1.43369 > 10.50.0.2.3050: S 623197096:623197096(0) win 16384 <mss 0="" 498316744="" 1460,nop,nop,sackok,nop,wscale="" 0,nop,nop,timestamp="">OPT1 interface
pfSense:~# tcpdump -ni le2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le2, link-type EN10MB (Ethernet), capture size 96 bytes
12:12:55.074689 arp who-has 20.50.0.1 tell 20.50.0.2
12:12:56.606935 arp who-has 20.50.0.1 tell 20.50.0.2
12:12:58.184179 arp who-has 20.50.0.1 tell 20.50.0.2
12:12:59.792644 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:04.420392 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:17.725956 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:19.275754 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:22.400258 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:27.849639 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:29.298290 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:30.906739 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:32.410233 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:37.122924 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:39.448504 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:41.004837 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:42.544985 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:44.186741 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:48.789311 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:56.349534 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:57.939083 arp who-has 20.50.0.1 tell 20.50.0.2
12:13:59.470493 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:01.029200 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:05.642239 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:18.934258 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:20.435657 arp who-has 20.50.0.1 tell 20.50.0.2
12:14:23.509403 arp who-has 20.50.0.1 tell 20.50.0.2

#netstat -nrf inet
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.50.0.1 UGS 0 13993 le0
10.50.0.0/16 link#1 UC 0 0 le0
10.50.0.1 00:0c:29:32:bc:0c UHLW 2 85 le0 840
20.50.0.0/16 link#3 UC 0 31 le2
20.50.0.3 00:0c:29:6c:39:c3 UHLW 1 80 le2 1166
127.0.0.1 127.0.0.1 UH 0 1699 lo0
192.168.233.0/24 link#2 UC 0 0 le1
192.168.233.1 00:50:56:c0:00:01 UHLW 1 6932 le1 1140

I tried the same config with openbsd, freebsd 6.2 and freebsd 7.0 (in place of pfsense)
Both freebsd 7.0 and pfsense 1.2.1rc4 had the same problem all other systems worked as expected.
I forgot to mention that this only happens if the IP trying to access WAN is on the same subnet as OPT1 in my case /16.</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></mss></mss></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp>

eri--

Can you draw a network diagram complete with ips and all and reformulate this?!

There is a fix for allowing that reply-to on WAN to not break certain things and it was not meant for such situations and that might break it.
The network diagram is just to see if this is something that needs to be supported or not or it is just something that needs attention before final release.

crtek

PC1    PC2    PFSENSE       PC3
ip 20.50.10.106/16 <–----> ip2 20.50.10.105/16--|--ip1 10.50.0.1/16<----->wan ip 10.50.0.2/16--|--opt1 20.50.0.1/16 <-------> ip 20.50.0.3/16
gw 20.50.10.105    gw 10.50.0.2 gw 10.50.0.1 | gw 20.50.0.3 gw 20.50.0.1
   |
   |
lan ip 192.168.233.2/24
|
|
|
PC4
ip 192.168.233.1/24
gw 192.168.233.2