Pfsense, two subnets, jumboframes etc.



  • Hi,

    I have a setup that differes quite a bit form the norm (apparently):

    We have 3 subnets: 192.168.140.0/24, 192.168.210.0/24 and 192.168.190.0/24. The .140 subnet is our normal office network, with it's own firewall and normal access to the internet.

    The .210 subnet is our production network and uses jumboframes (mtu 9000) thruout. One layer3 switch and several smaller layer2 switches.

    The .190 subnet is our control network, has one Layer 2 switch.

    Our final setup will be something like this:

    separate fw–-OPT1(.140)
          |                |
          |                |
    Internet--WAN-pfsense---LAN(.210)
                            |
                            |
                        OPT2(.190)

    So the machines in our .210 network will access the internet thru pfsense, the .140 network will access the internet thru it's own firewall. Machines in the .210 network need to access a certain amount of servers in in the .140 network, and vise versa. The .190 network only needs to be accessible from the .210 network.

    At the moment I have everything set everything up like this, just to test things out and see how things work out:

    Internet---separatefw--(.140)--pfsense---LAN(.210)
                                                 |
                                                 |
                                             OPT1(.190)

    Surfing, SSH etc. works fine from the .210 network /w squid (just whitelisting a couple of sites). Now I would like to have the machines in .140 to have access some servers in the .210 network. We have not decided yet will we open access to SMB and AFP or will we stick with sftp and ftp.

    My first problem is booting up the server.

    There is no place for me to set my LAN-interface to use jumboframes automatically, instead after each reboot I need to set it up by hand using ifconfig. Any solutions to this? There are no settings for this in LAN INterface screen. Once set the system works just fine.

    My second problem is NATing/Port Forwarding.

    The WAN interface on the pfsense box has received an IP from the DHCP server on the .140 network: 192.168.140.73.

    The LAN interface a static address 192.168.210.199 (I do not want to set pfsense as the default gw (.254) on the whole network before I have addressed these issues. On the machines I have been testing on, I have set the machines to use .210.199 as their default gw.)

    I have a server in the LAN network (.210) which is on, doesn't have a firewall enabled and works just fine. The server has the address 192.168.210.113.

    I have tried two different ways, and all permutations with each setup.

    The first try was normal Port Forwarding:

    • I first remove all old NAT rules and FW rules except the LAN to everywhere rule.
    • Open up Firewall->NAT-> Port Forwarding
    • Add new:
              - Interface: WAN
    • EXT add: Interface Add
    • Protocol TCP
    • Ext port range: SSH (22)
    • NAT IP: 192.168.210.113
    • Local Port: SSH (22)
    • enable "Auto Add a firewall rule…"
    • I edit the fw rule to do logging

    When I try to open a SSH connection from the .140 network (on a machine that got it's IP from DHCP), The connection times out.

    I see that the firewall passes the traffic on:

    Oct 3 16:35:16 WAN 192.168.140.77:4136 192.168.210.113:22 TCP
    Clicking on the green arrow gives some additional info:

    @39 pass in log quick on fxp0 reply-to (fxp0 192.168.140.254) inet proto tcp from any to 192.168.210.113 port = ssh flags S/SA keep state label "USER_RULE: NAT "

    I dont get any traffic into the logs of the server I am trying to access. Am I missing something? Before this I have mainly used ZyXel commercial FW:s, and never ran into problems like this there.

    My third problem is that I have very limited access to some of the machines on the .190 network. One machine works perfectly, I can access the web GUI just fine, but another machine does not even respond to ping. I have the following firewwall rule set up:

    TCP  LAN net  *  Hallinta net  *  *

    Some pre-emptive answers, if anybody cares:

    • The hardware is an extra P4 1.6Ghz machine we had lying around. the WAN is in the built-in Ethernet, the LAN is on a D-Link Gigabit Ether NIC that supports jumboframes. If everything works in the "proof of concept setup", we will be investing in a decent 1U rack server (HP with Intel procs and a 2 port Intel server NIC).

    • I do not want to mess with the other fw (the office network one). In fact, I'd like to stay away from the office network alltogether, and have my own FW where I can set up my own VPN and SSH settings and access.

    • All the systems I use here respond to ping from the pfsense unit.

    Thanks to anyone who has even bothered to read this far, I really any ideas anyone can give me. I just wouldnt want to go to iptables and fwbuilder…

    Added: I forgot to mention, I am running 1.2.1-RC1 built on Fri Sep 26 03:31:11 EDT 2008. I would very much prefer to stay on something that is based on freebsd 7, but if this can be caused by the fact that I am not using the stable version, I guess I'll have to revert to 1.2.

    /jussi
    Helsinki/Finland



  • Have you checked the firewall logs (Status -> System Logs Firewall tab)?

    tcpdump can be very helpful to verify packets arriving at a particular interface or going out a particular interface.

    On pfSense and other unix like systems,

    pfsense -i <interface>will display traffic on the specified interface. You can use this to verify that the packets are being seen at each of the hops on the route. Its best done from the console but can be be done in a ssh session though it can become confusing if your ssh traffic is travelling over the interface you are tracing. You should also check the target to make sure its responding (I have recollections of sshd configuration not being "obvious".)

    Concerning jumbo frames: In your case, wouldn't you also need to have jumbo frames enabled on the opt1 network? (I'm not familiar with the interactions of MTU discovery, and fragmentation in pfSense). But first you want to be sure your ssh connection is getting to its target and the target is responding.</interface>



  • @wallabybob:

    Have you checked the firewall logs (Status -> System Logs Firewall tab)?

    Yes, I have also monitored what's happening via ssh from the pfsense box. The problem here is that there is an obscene amount of things in the log, due to several windows machines and servers sending broadcast and udp traffic everywhere on the network. But I can see the initial ssh connection passing thru, like I mentioned in the first post.

    tcpdump can be very helpful to verify packets arriving at a particular interface or going out a particular interface.

    On pfSense and other unix like systems,

    pfsense -i <interface>will display traffic on the specified interface. You can use this to verify that the packets are being seen at each of the hops on the route. Its best done from the console but can be be done in a ssh session though it can become confusing if your ssh traffic is travelling over the interface you are tracing. You should also check the target to make sure its responding (I have recollections of sshd configuration not being "obvious".)</interface>

    I'll check this on monday.

    Concerning jumbo frames: In your case, wouldn't you also need to have jumbo frames enabled on the opt1 network? (I'm not familiar with the interactions of MTU discovery, and fragmentation in pfSense). But first you want to be sure your ssh connection is getting to its target and the target is responding.

    I hope not, as that would force me to use something else than pfsense. I dont want to have several boxes doing different things, just one neat package doing the stuff I need.

    /jussi



  • This is what I see in the filter logs:

    pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

    And nothing after that, form either of those IP's.

    /jussi



  • @juosukai:

    This is what I see in the filter logs:

    pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

    And nothing after that, form either of those IP's.

    /jussi

    This might be the reason the box was "lying around".

    I presume fxp0 is the motherboard NIC. That trace would lead me to suspect that NIC was broken or you had a memory problem. Might be time to try a known working box OR try another NIC and run some memory diagnostics. Quite a few of the LINUX Live CDs have the option of booting the memtest86 memory diagnostics or you can probably download a CD image. (Google for memtest86 or memtest86+). You might also want to hook another system up to than LAN port and ping the port while watching the interface traffic with tcpdump, just to get an idea how frequently this sort of thing happens.

    That an apparently well formed TCP packet passes the received FCS check but has a bad header length suggests to me either the NIC is broken (writing bad data to memory) OR the NIC is OK but some part of the memory is broken (returning bad data on reads), OR both.

    If you dust the box out, take out all the cards including memory sticks, clean out the slots with a vacuum cleaner and reseat all the cards things may come good.



  • Hi
    One quick. Put ifconfig in the very last of /etc/rc.

    instead after each reboot I need to set it up by hand using ifconfig. Any solutions to this?

    cheers,



  • pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

    first, disable hardware checksum offloading (in advanced)
    second - try different NIC



  • Hi Guys, thanks for the tips.

    @pan_2:

    pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

    first, disable hardware checksum offloading (in advanced)
    second - try different NIC

    Tried this. Still not succeeding with the NAT. I also changed a new 3COM NIC in instead of the built in NIC. I'm still getting the fragmentation errors:

    000000 rule 44/0(match): pass in on xl0: 192.168.140.77.1317 > 192.168.210.113.80:  tcp 28 [bad hdr length 0 - too short, < 20]

    I discovered something else, that might explain the issues here:

    When I look up the routing tables from the shell (using SSH) I get this:

    netstat -r -f inet -i

    Name    Mtu Network      Address              Ipkts Ierrs    Opkts Oerrs  Coll
    xl0    1500 192.168.140.0 192.168.140.250      1975    -      714    -    -
    re0    1500 192.168.190.0 192.168.190.254          3    -        3    -    -
    stge0  9000 192.168.210.0 ihantala            33097    -    34218    -    -
    lo0  16384 your-net      localhost              339    -      339    -    -

    netstat -ri

    Name    Mtu Network      Address              Ipkts Ierrs    Opkts Oerrs  Coll
    xl0    1500 <link#1>00:10:5a:39:b0:ea    53105    1    25977    0    0
    xl0    1500 192.168.140.0 192.168.140.250      2022    -      736    -    -
    xl0    1500 fe80:1::210:5 fe80:1::210:5aff:        0    -        2    -    -
    fxp0*  1500 <link#2>00:08:02:1a:c4:02        0    0        0    0    0
    re0    1500 <link#3>00:e0:4c:69:77:5b      193    0      145    0    0
    re0    1500 192.168.190.0 192.168.190.254          3    -        3    -    -
    re0    1500 fe80:3::2e0:4 fe80:3::2e0:4cff:        0    -        2    -    -
    stge0  9000 <link#4>00:05:5d:88:8b:7e    78411    0    78213    0    0
    stge0  9000 192.168.210.0 ihantala            33167    -    34288    -    -
    stge0  9000 fe80:4::205:5 fe80:4::205:5dff:        0    -        2    -    -
    plip0  1500 <link#5>0    0        0    0    0
    pflog 33204 <link#6>0    0    4548    0    0
    lo0  16384 <link#7>344    0      344    0    0
    lo0  16384 your-net      localhost              344    -      344    -    -
    lo0  16384 localhost    ::1                      0    -        0    -    -
    lo0  16384 fe80:7::1    fe80:7::1                0    -        0    -    -
    enc0*  1536 <link#8>0    0        0    0    0
    pfsyn  1460 <link#9>0    0        0    0    0

    stge0 has the correct MTU, no problems. But when I look at the routing tables from the web-interface (Diagnostic->Routes)

    IPv4
    Destination Gateway Flags Refs Use Mtu Netif Expire
    default 192.168.140.254 UGS 0 24739 1500 xl0
    127.0.0.1 127.0.0.1 UH 0 347 16384 lo0
    192.168.140.0/24 link#1 UC 0 1 1500 xl0
    192.168.140.13 00:1b:78:93:1d:2a UHLW 1 605 1500 xl0 1167
    192.168.140.14 00:1b:78:95:4a:30 UHLW 1 91 1500 xl0 1104
    192.168.140.77 00:e0:00:9b:9e:66 UHLW 1 0 1500 xl0 934
    192.168.140.254 00:90:7f:3c:89:30 UHLW 2 475 1500 xl0 1194
    192.168.190.0/24 link#3 UC 0 7 1500 re0
    192.168.210.0/24 link#4 UC 0 0 1500 stge0
    192.168.210.38 00:e0:ed:0a:49:60 UHLW 1 1 1500 stge0 66
    192.168.210.49 00:1f:5b:35:9a:61 UHLW 1 13 1500 stge0 1080
    192.168.210.50 00:1b:63:b7:4c:2c UHLW 1 78261 1500 stge0 405
    192.168.210.103 00:0a:5e:20:a7:ed UHLW 1 6 1500 stge0 1161
    192.168.210.109 00:14:51:65:02:82 UHLW 1 2 1500 stge0 1170
    192.168.210.113 00:1f:5b:31:40:20 UHLW 1 21 1500 stge0 64

    The systems shows all routes as MTU=1500! Is my problem here? Should I be trying to use another version or build of pfsense? Any suggestions to why I am getting this kind of problems?

    The box was lying around because we replaced it with higher-end machine. There were no actual stability issues with it (not more than your average XP box, that is). I am loathe to put 1000 euros into a decent FW machine before I know I can actually achieve what I am getting at.</link#9></link#8></link#7></link#6></link#5></link#4></link#3></link#2></link#1>



  • Hi,

    Just MTU thingy put a side for now and focusing on port forwarding issue, I've experienced most likely the same issue with the latest 1.3-AA build yesterday. Everything on the WAN gets port forwarded by the rule are dropped with the [too short] messages. Although yours isn't 1.3 nor the latest 1.2.1 so it might not your case but I strongly recommend that you try some OLDER builds.

    BTW, do you still see the issue even you revert all the MTU setting to default?

    cheers,



  • Are you allowing icmp otherwise you break path mtu discovery?!

    Can you monitor if RST packets are being sent by pfsense to the .140 network?

    Are you sure that ssh on pfSense is not running on that port(ssh:22)?



  • @ermal:

    Are you allowing icmp otherwise you break path mtu discovery?!

    Yes, I have tried to let all icmp traffic thru on the interfaces.

    Can you monitor if RST packets are being sent by pfsense to the .140 network?

    How can I monitor them? I know for sure that nothing is reaching the www or ftp servers.

    Are you sure that ssh on pfSense is not running on that port(ssh:22)?

    I have tied ftp, ssh and http, and always with the same results.

    I swapped hardware as well, I am know using a HP DL145 server with a 4 port Intel Server NIC (we are going to put server to other use in a week or so, I just want to verify that pfsense will work in our environment before I splash the cash on some new hardware). Should be no issues with the hardware this time…

    I also decided to go with the stable 1.2 build at this point, there shouldnt be a lot of changes is basic FW rules and NATing from 1.2 to 1.2.1, right?

    /jussi


Log in to reply