Miss-configured Firewall/NAT and (possible) packet loss plus some extras

DemoNIck

I am experiencing some problems with a new installation of pfSense 2.1 (64-bit) on a APU1C4 (PC Engines system: 3 Realtek RTL8111E Gb / 4 GB DRAM / AMD G T40E CPU) regarding some firewall/NAT rules blocking my p2p traffic and probably leading to a big packet loss (>20% during p2p traffic) on the WAN_PPPoE interface.
I am using the pfSense box in order to connect to my ISP via a TP-Link 8616 ADSL2+ modem.

Question No 1:
Is it possible a bad firewall configuration to lead to an increase of Packet Loss (>20%) on the WAN_PPPoE interface and finally/possibly to lead to a disconnection from the ISP, or such a packet loss increase has to do only with the "quality of the line" provided by the ISP (or the bad quality/configuration of the ADSL modem, or the quality of the "telephone" lines inside my home etc)?

The 3 ethernet interfaces are named as RE0, RE1 and RE2

RE0 is assigned to WAN interface and the IPv4 (and IPv6) assignment is obtained from my ISP via PPPoE with the TP-Link modem in bridge mode.

RE1 and RE2 are assigned to LAN interface in Bridge Mode (I want them to act like a switch).
Due to the existence of the Bridge I have also set :
net.link.bridge.pfil_member -> 0 (Set to 0 to disable filtering on the incoming and outgoing member interfaces.)
net.link.bridge.pfil_bridge -> 1 (Set to 1 to enable filtering on the bridge interface)

Question No 2:
Despite the fact that net.link.bridge.pfil_member =0, do I also have to add additional firewall rules allowing "everything" from the RE1 and RE2 (members of the bridge interface) to LAN (the bridge interface) and vice versa ?

NOTE: [1]=Checked, [ ]=Unchecked

WAN:
[1] Block Private Networks
[1] Block bogon Networks

RE1 and RE2:
[1] Enable Interface
IPv4 Configuration type : None
IPv6 Configuration type : None

Block Private Networks

Block bogon Networks

LAN:
[1] Enable Interface
IPv4 Configuration type : Static IPv4
IPv6 Configuration type : None
IPv4 address: 192.168.1.1/24
IPv4 Upstream Gateway: None

Block Private Networks

Block bogon Networks

On "RE1" physical port is connected a FreeBSD server (192.168.1.10) running some services (ssh, MySQL server, SVN Repo, X11 server, transmission p2p daemon etc) some of them I want to access from outside the local lan
On "RE2" physical port is connected a switch in order to provide "internet access" to the rest of the house (HTPC, Wi-Fi AP, laptops etc).

System: Advanced: Firewall and NAT
[1] Clear invalid DF bits instead of dropping the packets
[1] Bypass firewall rules for traffic on the same interface
[1] Disable reply-to on WAN rules
Network Address translation : Enable (NAT + Proxy)
[1] Enable NAT Reflection for 1:1 NAT
[1] Enable automatic outbound NAT for Reflection
Networking
[1] Allow IPv6
[1] Enable device polling
[1] Disable hardware checksum offload
[1] Disable hardware TCP segmentation offload
[1] Disable hardware large receive offload
[1] Suppress ARP messages

Services running on pfSense box:
1. DHCP server on LAN interface (IPv4)
2. DNS forwarder (Interface IPs used by the DNS forwarder for responding to queries from clients : Selected is ALL)
3. Dynamic DNS via the "No-IP" provider
4. UPnP & NAT-PMP
5. Snort on WAN interface (now disabled for the various tests)
6. Traffic shaper (now disabled for the various tests)

So far so good, everything works fine.
I have set up various NAT/Firewall rules in order to access several services from outside (web-gui of the pfSense firewall, ssh on FreeBSD server etc.)

Question No 3:
For the internal "home" lan on the LAN interface I do NOT need to filter any packets. Can I just disable the filtering on the bridge interface by setting net.link.bridge.pfil_bridge->0?
If I do so will I be able to add additional NAT/Firewall rules on the WAN interface in order to access from outside some services running on computers belonging in the 192.168.1.0/24 LAN subnet if the filtering on the LAN interface is disabled?

Regarding the p2p problem. The p2p transmission client is running on the FreeBSD server (192.168.1.10) who is directly connected on the "RE1" physical port of the APU1C4 box.
Under "heavy duty", thus downloading with the maximum bandwidth provided by my ISP, the Packet Loss on the WAN_PPPoE interface increases dramatically (>20%) and after some time I get disconnected.
On the Firewall Log I can see blocking traffic on the WAN interface where sources are "external" IP's and destination my external IP provided by the ISP.

The same thing happened again, blocking IPs on the WAN Interface and increase of the Packet Loss on the WAN_PPPoE interface.

Question No 4:
Do I have to add additional Firewall rule in order to allow "everything" from 192.168.1.10 to WAN?
I guess WAN will always be the dynamically assigned IP from my ISP?
I am probably missing something but is the above needed despite the fact that I have already selected under "Firewall: NAT: Outbound" the setting [1] Automatic outbound NAT rule generation ?

Thank you in advance for your time and effort and accept my apologies for this "long" post.

MindfulCoyote

@DemoNIck:

Question No 1:
Is it possible a bad firewall configuration to lead to an increase of Packet Loss (>20%) on the WAN_PPPoE interface and finally/possibly to lead to a disconnection from the ISP, or such a packet loss increase has to do only with the "quality of the line" provided by the ISP (or the bad quality/configuration of the ADSL modem, or the quality of the "telephone" lines inside my home etc)?

The 3 ethernet interfaces are named as RE0, RE1 and RE2

RE0 is assigned to WAN interface and the IPv4 (and IPv6) assignment is obtained from my ISP via PPPoE with the TP-Link modem in bridge mode.

RE1 and RE2 are assigned to LAN interface in Bridge Mode (I want them to act like a switch).
Due to the existence of the Bridge I have also set :
net.link.bridge.pfil_member -> 0 (Set to 0 to disable filtering on the incoming and outgoing member interfaces.)
net.link.bridge.pfil_bridge -> 1 (Set to 1 to enable filtering on the bridge interface)

Generally not the pfSense software unless it's also associated with high CPU (Status: RRD Graphs: Processor) or buffer overruns (Status: RRD Graphs: Mbuf)… I would try testing the line with a different piece of hardware (ex. your workstation) to see if it's pfSense or it's hardware.

MindfulCoyote

@DemoNIck:

Question No 2:
Despite the fact that net.link.bridge.pfil_member =0, do I also have to add additional firewall rules allowing "everything" from the RE1 and RE2 (members of the bridge interface) to LAN (the bridge interface) and vice versa ?

net.link.bridge.pfil_member=0 should result in any rules on the bridge members being ignored, but since you indicate that net.link.bridge.pfil_bridge=1 then rules applied to the bridge interface will be in effect including the default deny, so yes, with net.link.bridge.pfil_bridge=1 you need to explicitly add rules allowing traffic.

@DemoNIck:

NOTE: [1]=Checked, [ ]=Unchecked

WAN:
[1] Block Private Networks
[1] Block bogon Networks

RE1 and RE2:
[1] Enable Interface
IPv4 Configuration type : None
IPv6 Configuration type : None

Block Private Networks

Block bogon Networks

LAN:
[1] Enable Interface
IPv4 Configuration type : Static IPv4
IPv6 Configuration type : None
IPv4 address: 192.168.1.1/24
IPv4 Upstream Gateway: None

Block Private Networks

Block bogon Networks

On "RE1" physical port is connected a FreeBSD server (192.168.1.10) running some services (ssh, MySQL server, SVN Repo, X11 server, transmission p2p daemon etc) some of them I want to access from outside the local lan
On "RE2" physical port is connected a switch in order to provide "internet access" to the rest of the house (HTPC, Wi-Fi AP, laptops etc).

System: Advanced: Firewall and NAT
[1] Clear invalid DF bits instead of dropping the packets
[1] Bypass firewall rules for traffic on the same interface
[1] Disable reply-to on WAN rules
Network Address translation : Enable (NAT + Proxy)
[1] Enable NAT Reflection for 1:1 NAT
[1] Enable automatic outbound NAT for Reflection
Networking
[1] Allow IPv6
[1] Enable device polling
[1] Disable hardware checksum offload
[1] Disable hardware TCP segmentation offload
[1] Disable hardware large receive offload
[1] Suppress ARP messages

Services running on pfSense box:
1. DHCP server on LAN interface (IPv4)
2. DNS forwarder (Interface IPs used by the DNS forwarder for responding to queries from clients : Selected is ALL)
3. Dynamic DNS via the "No-IP" provider
4. UPnP & NAT-PMP
5. Snort on WAN interface (now disabled for the various tests)
6. Traffic shaper (now disabled for the various tests)

So far so good, everything works fine.
I have set up various NAT/Firewall rules in order to access several services from outside (web-gui of the pfSense firewall, ssh on FreeBSD server etc.)

Very helpful amount of detail! :)

I guess that you have RE1 and RE2 bridged so that the LAN (RE2) will have full layer 2 connectivity to the server on RE1? In this case my only concern would be the server acting as a relay point should it become compromised. Unless you must have layer 2 traffic, you might consider more a DMZ type arrangement.

MindfulCoyote

@DemoNIck:

I am experiencing some problems with a new installation of pfSense 2.1 (64-bit) on a APU1C4 (PC Engines system: 3 Realtek RTL8111E Gb / 4 GB DRAM / AMD G T40E CPU) regarding some firewall/NAT rules blocking my p2p traffic and probably leading to a big packet loss (>20% during p2p traffic) on the WAN_PPPoE interface.
I am using the pfSense box in order to connect to my ISP via a TP-Link 8616 ADSL2+ modem.

Ha. I got so tied up in the individual questions, that I forgot to tackle the real issue, the packet loss. Since you mentioned P2P, this packet loss you are seeing is probably due to one of two things, either outbound saturation of your bandwidth (see Status: RRD Graphs: Traffic: WAN ) or saturation of the state tables (Status: RRD Graphs: System: States). Although the hardware or CPU or mbuf issues still might apply.

Try checking each of these graphs during periods of high packet loss to see if the P2P stuff is overloading anything. A lot of people configure the traffic shaping of the p2p traffic for this exact reason.

DemoNIck

MindfulCoyote thank you very much for your in detail responses.

I will try/test as soon as I get back home on Wednesday and post any results or further questions.

Once again thank you for your time and effort.

DemoNIck

@MindfulCoyote:

I guess that you have RE1 and RE2 bridged so that the LAN (RE2) will have full layer 2 connectivity to the server on RE1?

Yes that is correct.
The "LAN" and the "Server" are on the same subnet (192.168.1.0/24).
Apart from the services I want to access from "outside", the typical use of the server is to act as Network File Sharing server accessed from the "LAN" devices.
The basic idea is that "LAN" is a trusted network.
So I guess that for the bridged LAN interface the following settings are safe to use :

net.link.bridge.pfil_member->0
net.link.bridge.pfil_bridge->0

Without any other rule concerning the filtering from the "Server" and "LAN" to "WAN" and vice versa, with the above two settings, the packets from 192.168.1.0/24 devices to 192.168.1.0/24 devices are NOT filtered (thus there are NO packets blocked by the firewall).
And then with a basic rule allowing ANY from "WAN" to "Server" and vice versa and with simple port forwarding rules I could access any service I want from "outside" to the "Server".
Is that correct?

@MindfulCoyote:

In this case my only concern would be the server acting as a relay point should it become compromised. Unless you must have layer 2 traffic, you might consider more a DMZ type arrangement.

Please forgive my ignorance, but is it possible to be more specific regarding your concern about the "Server" acting as a relay point? If that is actual happening what exactly should I try to track down in the firewall logs?

@MindfulCoyote:

Ha. I got so tied up in the individual questions, that I forgot to tackle the real issue, the packet loss. Since you mentioned P2P, this packet loss you are seeing is probably due to one of two things, either outbound saturation of your bandwidth (see Status: RRD Graphs: Traffic: WAN ) or saturation of the state tables (Status: RRD Graphs: System: States). Although the hardware or CPU or mbuf issues still might apply.

I am attaching some RRD graphs for the last 1 week in case you have the time to check.

Once again thank you in advance for your time and effort.

WAN_1wk.png_thumb

states_1wk.png_thumb

mbuf_1wk.png_thumb

CPU_1wk.png_thumb

WAN_PPOE_QUALITY_1wk.png_thumb

MindfulCoyote

@DemoNIck:

And then with a basic rule allowing ANY from "WAN" to "Server" and vice versa and with simple port forwarding rules I could access any service I want from "outside" to the "Server". Is that correct?

I believe so, but I'm not entirely clear about exactly how your interfaces are configured. It sounds like you've completely disabled LAN filtering.

@DemoNIck:

Please forgive my ignorance, but is it possible to be more specific regarding your concern about the "Server" acting as a relay point? If that is actual happening what exactly should I try to track down in the firewall logs?

I meant in the event the server was compromised. Whenever you give the Internet access to a server inside your firewall, the firewall is bypassed and no longer offers any protection for that server. If an attacker successfully compromised your server then he would "be inside" your LAN. A DMZ arrangement means that the firewall still sits between the server and the LAN so that it can still offer protection if the server is compromised. Here's a good diagram of a DMZ configuration: http://doc.m0n0.ch/handbook/examples.html

@DemoNIck:

I am attaching some RRD graphs for the last 1 week in case you have the time to check.

When you experience the >20% packet loss are there any corresponding entries on these graphs? For example, on the WAN Quality graph I notice two periods of packet loss, Wednesday around 11PM and Sunday around 6AM. I also see some correlation between latency on the quality graph and bandwidth utilization on the WAN traffic graph (Thu, Mon, Tue). To me this could indicate WAN saturation.

DemoNIck

Thank you once again for the response and excuse my delayed answer.

@MindfulCoyote:

I believe so, but I'm not entirely clear about exactly how your interfaces are configured. It sounds like you've completely disabled LAN filtering.

For the time being yes, the LAN filtering is completely disabled in order to clarify that the pfSense system is NOT "responsible" for any packet loss. A DMZ approach (as you also propose) will be preferred in the future as soon as the packet loss problem is solved.

@MindfulCoyote:

When you experience the >20% packet loss are there any corresponding entries on these graphs? For example, on the WAN Quality graph I notice two periods of packet loss, Wednesday around 11PM and Sunday around 6AM. I also see some correlation between latency on the quality graph and bandwidth utilization on the WAN traffic graph (Thu, Mon, Tue). To me this could indicate WAN saturation.

Any packet loss appearing on the graphs is ONLY due to p2p traffic. Now I am almost certain that this "packet loss" has nothing to do with the pfSense system itself (thus this topic will have to be closed). But before we do so could you be so kind and be more specific on the "WAN saturation" indication.

Once again thank you in advance for your time and effort.

MindfulCoyote

@DemoNIck:

Any packet loss appearing on the graphs is ONLY due to p2p traffic. Now I am almost certain that this "packet loss" has nothing to do with the pfSense system itself

Excellent. I suspected this was the case. Now some simple traffic shaping ought to tidy things up so that your high priority traffic isn't impacted.

@DemoNIck:

But before we do so could you be so kind and be more specific on the "WAN saturation" indication.

I apologize for not writing more clearly. I have a bad habit of thinking faster than I type. ;)

By "WAN saturation" I meant the outbound bandwidth utilization on the WAN link reaching or near 100% [due to p2p traffic]. It has been my experience that retail WAN technology (POTS, DSL, Cable, etc.) is subject to extreme performance degradation (i.e. packet loss, etc.) under high bandwidth utilization conditions. When sustained bandwidth utilization exceeds 80% then smaller traffic bursts start hitting the "artificial" bandwidth limit imposed the Internet service provider (ISP). When traffic hits that artificial limit, the ISP begin to "rate limit" the traffic. Often their rate limiting is very "brute force" by simply discarding packets thus creating all sorts of timeouts and retransmits.

Although you never specifically mentioned the bandwidth of your connections, I know from experience that most Internet connections are asynchronous and so are highly sensitive to outbound traffic (i.e. p2p). Your WAN traffic graph shows periods of intense outbound bandwidth utilization with "wan-out-pass" reaching 742kb/s which is at the limit of the most common outbound speed (768kbps) so WAN outbound bandwidth utilization became the primary suspect especially since the rest of the graphs have values in reasonable ranges.

@DemoNIck:

Once again thank you in advance for your time and effort.

It's been my pleasure. Thank you for creating an interesting topic.