PfSense only allowing traffic out WAN, not LAN/OPTs
-
Hi all,
I appear to have broken my pfSense installation. :( Hopefully there is a solution so I don't have to hit the "Factory Defaults" button.
The setup I have is a WAN using PPPoE and 4 LANs (lets call them O, S, V and D)
Access to the Internet from any of these networks is fine but I cannot communicate from any internal network to any other network. The problem first exhibited itself when users in LAN D or V couldn't use the printer in LAN O.
I have the default allow access to any rule on the O network and duplicated that to V & D (S is behind a squid+squidguard proxy)
Where this gets interesting is that from the pfSense box itself I cannot ping anything on any of the local networks (I verified that hosts on the same network can ping each other.) Packet Captures show nothing leaving the interface that wasn't solicited from that network.
From pfSense:
[2.0.3-RELEASE][admin@pfSense.sc.local]/root(1): ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2): 56 data bytes
^C
–- 192.168.10.2 ping statistics ---
29 packets transmitted, 0 packets received, 100.0% packet lossand from my PC on the 192.168.10.0/24 network:
C:>ping 192.168.10.2
Pinging 192.168.10.2 with 32 bytes of data:
Reply from 192.168.10.2: bytes=32 time=3ms TTL=64
Reply from 192.168.10.2: bytes=32 time=4ms TTL=64
Reply from 192.168.10.2: bytes=32 time=1ms TTL=64
Reply from 192.168.10.2: bytes=32 time=3ms TTL=64Ping statistics for 192.168.10.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 4ms, Average = 2msAny thoughts would be greatly appreciated & please let me know what other information I can provide to solve this.
Thanks,
Lee. -
please let me know what other information I can provide to solve this.
A good start would be to post the output of pfSense shell commands:```
/etc/rc.banner
ifconfig
netstat -r -n -
Thanks WallabyBob. As requested:
/etc/rc.banner
[2.0.3-RELEASE][admin@pfSense.sc.local]/root(1): /etc/rc.banner
*** Welcome to pfSense 2.0.3-RELEASE-pfSense (i386) on pfSense ***WAN (wan) -> pppoe0 -> 175.100.64.243 (PPPoE)
OFFICELAN (lan) -> em1 -> 192.168.10.254
SCHOOLLAN (opt1) -> em0 -> 192.168.11.254
DEPARTMENTSLAN (opt2) -> em2 -> 192.168.13.254
VOLUNTEERLAN (opt3) -> em3 -> 192.168.12.254ifconfig
[2.0.3-RELEASE][admin@pfSense.sc.local]/root(2): ifconfig
re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=389b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:30:67:6d:6c:02
inet6 fe80::230:67ff:fe6d:6c02%re0 prefixlen 64 scopeid 0x1
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4f:47:66
inet 192.168.11.254 netmask 0xffffff00 broadcast 192.168.11.255
inet6 fe80::202:a5ff:fe4f:4766%em0 prefixlen 64 scopeid 0x2
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4f:47:67
inet 192.168.10.254 netmask 0xffffff00 broadcast 192.168.10.255
inet6 fe80::202:a5ff:fe4f:4767%em1 prefixlen 64 scopeid 0x3
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4f:2c:58
inet 192.168.13.254 netmask 0xffffff00 broadcast 192.168.13.255
inet6 fe80::202:a5ff:fe4f:2c58%em2 prefixlen 64 scopeid 0x4
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4f:2c:59
inet 192.168.12.254 netmask 0xffffff00 broadcast 192.168.12.255
inet6 fe80::202:a5ff:fe4f:2c59%em3 prefixlen 64 scopeid 0x5
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
pfsync0: flags=0<> metric 0 mtu 1460
syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
pflog0: flags=100 <promisc>metric 0 mtu 33200
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
nd6 options=43 <performnud,accept_rtadv>pppoe0: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
inet6 fe80::230:67ff:fe6d:6c02%pppoe0 prefixlen 64 scopeid 0xb
inet 175.100.64.243 –> 175.100.64.1 netmask 0xffffffff
nd6 options=43<performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,noarp,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,simplex,multicast>and netstat -r -n
[2.0.3-RELEASE][admin@pfSense.sc.local]/root(3): netstat -r -n
Routing tablesInternet:
Destination Gateway Flags Refs Use Netif Expire
default 175.100.64.1 UGS 0 3544647 pppoe0
117.120.24.1 175.100.64.1 UGHS 0 22080 pppoe0
127.0.0.1 link#10 UH 0 25925 lo0
175.100.64.1 link#11 UH 0 42848 pppoe0
175.100.64.243 link#11 UHS 0 0 lo0
192.168.10.0/24 link#3 U 0 846539 em1
192.168.10.254 link#3 UHS 0 11 lo0
192.168.11.0/24 link#2 U 0 491875 em0
192.168.11.254 link#2 UHS 0 0 lo0
192.168.12.0/24 link#5 U 0 1023984 em3
192.168.12.254 link#5 UHS 0 0 lo0
192.168.13.0/24 link#4 U 0 2601579 em2
192.168.13.254 link#4 UHS 0 0 lo0
203.113.131.1 175.100.64.1 UGHS 0 22059 pppoe0Internet6:
Destination Gateway Flags Netif Expire
::1 ::1 UH lo0
fe80::%re0/64 link#1 U re0
fe80::230:67ff:fe6d:6c02%re0 link#1 UHS lo0
fe80::%em0/64 link#2 U em0
fe80::202:a5ff:fe4f:4766%em0 link#2 UHS lo0
fe80::%em1/64 link#3 U em1
fe80::202:a5ff:fe4f:4767%em1 link#3 UHS lo0
fe80::%em2/64 link#4 U em2
fe80::202:a5ff:fe4f:2c58%em2 link#4 UHS lo0
fe80::%em3/64 link#5 U em3
fe80::202:a5ff:fe4f:2c59%em3 link#5 UHS lo0
fe80::%lo0/64 link#10 U lo0
fe80::1%lo0 link#10 UHS lo0
fe80::%pppoe0/64 link#11 U pppoe0
fe80::230:67ff:fe6d:6c02%pppoe0 link#11 UHS lo0
ff01:1::/32 fe80::230:67ff:fe6d:6c02%re0 U re0
ff01:2::/32 fe80::202:a5ff:fe4f:4766%em0 U em0
ff01:3::/32 fe80::202:a5ff:fe4f:4767%em1 U em1
ff01:4::/32 fe80::202:a5ff:fe4f:2c58%em2 U em2
ff01:5::/32 fe80::202:a5ff:fe4f:2c59%em3 U em3
ff01
:/32 ::1 U lo0
ff01
:/32 fe80::230:67ff:fe6d:6c02%pppoe0 U pppoe0
ff02::%re0/32 fe80::230:67ff:fe6d:6c02%re0 U re0
ff02::%em0/32 fe80::202:a5ff:fe4f:4766%em0 U em0
ff02::%em1/32 fe80::202:a5ff:fe4f:4767%em1 U em1
ff02::%em2/32 fe80::202:a5ff:fe4f:2c58%em2 U em2
ff02::%em3/32 fe80::202:a5ff:fe4f:2c59%em3 U em3
ff02::%lo0/32 ::1 U lo0
ff02::%pppoe0/32 fe80::230:67ff:fe6d:6c02%pppoe0 U pppoe0Thanks again,
Lee. -
Some of the basics look OK.
I suspect you probably have multiple problems. Lets start with a single simpler problem.
Access to the Internet from any of these networks is fine but I cannot communicate from any internal network to any other network. The problem first exhibited itself when users in LAN D or V couldn't use the printer in LAN O.
How do users in LAN D attempt to access the printer in LAN O? What happens when they attempt such access? Does the access attempt get reported in the firewall log? Does the printer allow access from LAN D? Does the printer respond to pings from LAN O? Does the printer respond to pings from LAN D?
When you added the firewall rules did you also reset states (see Diagnostics -> States, click on Reset States and read the explanation).
Please post a screen shot or other full specification of the firewall rules on the LAN D interface. The information you have provided:
@lee.davis:I have the default allow access to any rule on the O network and duplicated that to V & D
is incomplete. For example, do the rules allow accesses from LAN D IP address (or did you copy a source=LAN O subnet rule)?
-
Hi wallabybob,
unfortunately with mounting pressure from the users I needed a solution for "now" rather than a solution that was "right", so I have restored a backup from 2 weeks ago which seems to have fixed things for the most part. It irks me that I don't know what the actual problem was and printing is still slow from the other subnet. Looks like I'm going to solve that in a different way now.
To answer your questions:
How do users in LAN D attempt to access the printer in LAN O?
Printer drivers were installed on each PC in LAN D. At the time of installation the driver setup was able to communicate with the printer which configured an appropriate printer port on the client PC.
What happens when they attempt such access?
The print job sits in the print queue on the client PC indefinitely
Does the access attempt get reported in the firewall log?
I enabled appropriate logging and saw PASSes noted in the firewall log, however running a packet capture on the LAN O interface of pfSense I did not see any matching packets.
Does the printer allow access from LAN D?
Yes.
Does the printer respond to pings from LAN O?
Pinging from a client on LAN O to the printer was successful. Pinging from the firewall interface LAN O to the printer was NOT successful.
Does the printer respond to pings from LAN D?
No. Firewall Logs show PASSes but again, nothing in a packet capture from LAN O interface
Please post a screen shot or other full specification of the firewall rules on the LAN D interface.
Sorry, as I've restored from backup the rule is the now the same as when it was failing. What i have now is:
I've highlighted the rule that should allow access to the printer (and the file server) on LAN O
The OfficeResources alias contains the IP addresses of the printer and the file server only.
However when the firewall was allowing nothing out its LAN interfaces I had removed all the rules but the last one, which was copied from the LAN O (the "LAN" inferface asopposed to the "OPTn" interfaces) rule and then modified to relate to LAN D.
I hope that's clear, reading back there's a lot in there and it may be moot given I have restored to a backup.
I'm also looking at dropping LAN D and combining the clients with the LAN O. Just need to convince management that the separate LANs are causing more problems than they are solving.
Thanks,
Lee.