Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Pfsense, two subnets, jumboframes etc.

    Scheduled Pinned Locked Moved General pfSense Questions
    11 Posts 5 Posters 7.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W
      wallabybob
      last edited by

      Have you checked the firewall logs (Status -> System Logs Firewall tab)?

      tcpdump can be very helpful to verify packets arriving at a particular interface or going out a particular interface.

      On pfSense and other unix like systems,

      pfsense -i <interface>will display traffic on the specified interface. You can use this to verify that the packets are being seen at each of the hops on the route. Its best done from the console but can be be done in a ssh session though it can become confusing if your ssh traffic is travelling over the interface you are tracing. You should also check the target to make sure its responding (I have recollections of sshd configuration not being "obvious".)

      Concerning jumbo frames: In your case, wouldn't you also need to have jumbo frames enabled on the opt1 network? (I'm not familiar with the interactions of MTU discovery, and fragmentation in pfSense). But first you want to be sure your ssh connection is getting to its target and the target is responding.</interface>

      1 Reply Last reply Reply Quote 0
      • J
        juosukai
        last edited by

        @wallabybob:

        Have you checked the firewall logs (Status -> System Logs Firewall tab)?

        Yes, I have also monitored what's happening via ssh from the pfsense box. The problem here is that there is an obscene amount of things in the log, due to several windows machines and servers sending broadcast and udp traffic everywhere on the network. But I can see the initial ssh connection passing thru, like I mentioned in the first post.

        tcpdump can be very helpful to verify packets arriving at a particular interface or going out a particular interface.

        On pfSense and other unix like systems,

        pfsense -i <interface>will display traffic on the specified interface. You can use this to verify that the packets are being seen at each of the hops on the route. Its best done from the console but can be be done in a ssh session though it can become confusing if your ssh traffic is travelling over the interface you are tracing. You should also check the target to make sure its responding (I have recollections of sshd configuration not being "obvious".)</interface>

        I'll check this on monday.

        Concerning jumbo frames: In your case, wouldn't you also need to have jumbo frames enabled on the opt1 network? (I'm not familiar with the interactions of MTU discovery, and fragmentation in pfSense). But first you want to be sure your ssh connection is getting to its target and the target is responding.

        I hope not, as that would force me to use something else than pfsense. I dont want to have several boxes doing different things, just one neat package doing the stuff I need.

        /jussi

        1 Reply Last reply Reply Quote 0
        • J
          juosukai
          last edited by

          This is what I see in the filter logs:

          pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

          And nothing after that, form either of those IP's.

          /jussi

          1 Reply Last reply Reply Quote 0
          • W
            wallabybob
            last edited by

            @juosukai:

            This is what I see in the filter logs:

            pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

            And nothing after that, form either of those IP's.

            /jussi

            This might be the reason the box was "lying around".

            I presume fxp0 is the motherboard NIC. That trace would lead me to suspect that NIC was broken or you had a memory problem. Might be time to try a known working box OR try another NIC and run some memory diagnostics. Quite a few of the LINUX Live CDs have the option of booting the memtest86 memory diagnostics or you can probably download a CD image. (Google for memtest86 or memtest86+). You might also want to hook another system up to than LAN port and ping the port while watching the interface traffic with tcpdump, just to get an idea how frequently this sort of thing happens.

            That an apparently well formed TCP packet passes the received FCS check but has a bad header length suggests to me either the NIC is broken (writing bad data to memory) OR the NIC is OK but some part of the memory is broken (returning bad data on reads), OR both.

            If you dust the box out, take out all the cards including memory sticks, clean out the slots with a vacuum cleaner and reseat all the cards things may come good.

            1 Reply Last reply Reply Quote 0
            • N
              nocer
              last edited by

              Hi
              One quick. Put ifconfig in the very last of /etc/rc.

              instead after each reboot I need to set it up by hand using ifconfig. Any solutions to this?

              cheers,

              1 Reply Last reply Reply Quote 0
              • S
                Soyokaze
                last edited by

                pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

                first, disable hardware checksum offloading (in advanced)
                second - try different NIC

                Need full pfSense in a cloud? PM for details!

                1 Reply Last reply Reply Quote 0
                • J
                  juosukai
                  last edited by

                  Hi Guys, thanks for the tips.

                  @pan_2:

                  pass in on fxp0: 192.168.140.77.1549 > 192.168.210.113.22:  tcp 28 [bad hdr length 0 - too short, < 20]

                  first, disable hardware checksum offloading (in advanced)
                  second - try different NIC

                  Tried this. Still not succeeding with the NAT. I also changed a new 3COM NIC in instead of the built in NIC. I'm still getting the fragmentation errors:

                  000000 rule 44/0(match): pass in on xl0: 192.168.140.77.1317 > 192.168.210.113.80:  tcp 28 [bad hdr length 0 - too short, < 20]

                  I discovered something else, that might explain the issues here:

                  When I look up the routing tables from the shell (using SSH) I get this:

                  netstat -r -f inet -i

                  Name    Mtu Network      Address              Ipkts Ierrs    Opkts Oerrs  Coll
                  xl0    1500 192.168.140.0 192.168.140.250      1975    -      714    -    -
                  re0    1500 192.168.190.0 192.168.190.254          3    -        3    -    -
                  stge0  9000 192.168.210.0 ihantala            33097    -    34218    -    -
                  lo0  16384 your-net      localhost              339    -      339    -    -

                  netstat -ri

                  Name    Mtu Network      Address              Ipkts Ierrs    Opkts Oerrs  Coll
                  xl0    1500 <link#1>00:10:5a:39:b0:ea    53105    1    25977    0    0
                  xl0    1500 192.168.140.0 192.168.140.250      2022    -      736    -    -
                  xl0    1500 fe80:1::210:5 fe80:1::210:5aff:        0    -        2    -    -
                  fxp0*  1500 <link#2>00:08:02:1a:c4:02        0    0        0    0    0
                  re0    1500 <link#3>00:e0:4c:69:77:5b      193    0      145    0    0
                  re0    1500 192.168.190.0 192.168.190.254          3    -        3    -    -
                  re0    1500 fe80:3::2e0:4 fe80:3::2e0:4cff:        0    -        2    -    -
                  stge0  9000 <link#4>00:05:5d:88:8b:7e    78411    0    78213    0    0
                  stge0  9000 192.168.210.0 ihantala            33167    -    34288    -    -
                  stge0  9000 fe80:4::205:5 fe80:4::205:5dff:        0    -        2    -    -
                  plip0  1500 <link#5>0    0        0    0    0
                  pflog 33204 <link#6>0    0    4548    0    0
                  lo0  16384 <link#7>344    0      344    0    0
                  lo0  16384 your-net      localhost              344    -      344    -    -
                  lo0  16384 localhost    ::1                      0    -        0    -    -
                  lo0  16384 fe80:7::1    fe80:7::1                0    -        0    -    -
                  enc0*  1536 <link#8>0    0        0    0    0
                  pfsyn  1460 <link#9>0    0        0    0    0

                  stge0 has the correct MTU, no problems. But when I look at the routing tables from the web-interface (Diagnostic->Routes)

                  IPv4
                  Destination Gateway Flags Refs Use Mtu Netif Expire
                  default 192.168.140.254 UGS 0 24739 1500 xl0
                  127.0.0.1 127.0.0.1 UH 0 347 16384 lo0
                  192.168.140.0/24 link#1 UC 0 1 1500 xl0
                  192.168.140.13 00:1b:78:93:1d:2a UHLW 1 605 1500 xl0 1167
                  192.168.140.14 00:1b:78:95:4a:30 UHLW 1 91 1500 xl0 1104
                  192.168.140.77 00:e0:00:9b:9e:66 UHLW 1 0 1500 xl0 934
                  192.168.140.254 00:90:7f:3c:89:30 UHLW 2 475 1500 xl0 1194
                  192.168.190.0/24 link#3 UC 0 7 1500 re0
                  192.168.210.0/24 link#4 UC 0 0 1500 stge0
                  192.168.210.38 00:e0:ed:0a:49:60 UHLW 1 1 1500 stge0 66
                  192.168.210.49 00:1f:5b:35:9a:61 UHLW 1 13 1500 stge0 1080
                  192.168.210.50 00:1b:63:b7:4c:2c UHLW 1 78261 1500 stge0 405
                  192.168.210.103 00:0a:5e:20:a7:ed UHLW 1 6 1500 stge0 1161
                  192.168.210.109 00:14:51:65:02:82 UHLW 1 2 1500 stge0 1170
                  192.168.210.113 00:1f:5b:31:40:20 UHLW 1 21 1500 stge0 64

                  The systems shows all routes as MTU=1500! Is my problem here? Should I be trying to use another version or build of pfsense? Any suggestions to why I am getting this kind of problems?

                  The box was lying around because we replaced it with higher-end machine. There were no actual stability issues with it (not more than your average XP box, that is). I am loathe to put 1000 euros into a decent FW machine before I know I can actually achieve what I am getting at.</link#9></link#8></link#7></link#6></link#5></link#4></link#3></link#2></link#1>

                  1 Reply Last reply Reply Quote 0
                  • N
                    nocer
                    last edited by

                    Hi,

                    Just MTU thingy put a side for now and focusing on port forwarding issue, I've experienced most likely the same issue with the latest 1.3-AA build yesterday. Everything on the WAN gets port forwarded by the rule are dropped with the [too short] messages. Although yours isn't 1.3 nor the latest 1.2.1 so it might not your case but I strongly recommend that you try some OLDER builds.

                    BTW, do you still see the issue even you revert all the MTU setting to default?

                    cheers,

                    1 Reply Last reply Reply Quote 0
                    • E
                      eri--
                      last edited by

                      Are you allowing icmp otherwise you break path mtu discovery?!

                      Can you monitor if RST packets are being sent by pfsense to the .140 network?

                      Are you sure that ssh on pfSense is not running on that port(ssh:22)?

                      1 Reply Last reply Reply Quote 0
                      • J
                        juosukai
                        last edited by

                        @ermal:

                        Are you allowing icmp otherwise you break path mtu discovery?!

                        Yes, I have tried to let all icmp traffic thru on the interfaces.

                        Can you monitor if RST packets are being sent by pfsense to the .140 network?

                        How can I monitor them? I know for sure that nothing is reaching the www or ftp servers.

                        Are you sure that ssh on pfSense is not running on that port(ssh:22)?

                        I have tied ftp, ssh and http, and always with the same results.

                        I swapped hardware as well, I am know using a HP DL145 server with a 4 port Intel Server NIC (we are going to put server to other use in a week or so, I just want to verify that pfsense will work in our environment before I splash the cash on some new hardware). Should be no issues with the hardware this time…

                        I also decided to go with the stable 1.2 build at this point, there shouldnt be a lot of changes is basic FW rules and NATing from 1.2 to 1.2.1, right?

                        /jussi

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.