Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense and IPSEC lan to lan: a big doubt about the correct implementation

    General pfSense Questions
    3
    83
    6.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mauro.tridici @mauro.tridici
      last edited by

      VPCS on the left can ping every hosts on the schema (except the 192.168.120.1)
      VPCs on the left is not able to ping 192.168.120.0/24

      VPCS on the right can ping every hosts on the schema (except the 10.99.0.0/24)
      VPCs on the left is not able to ping 192.168.118.0/24

      If I'm not wrong, it seems that the ping traffic is not passing through the tunnel.
      Sorry for this stupid information...

      1 Reply Last reply Reply Quote 0
      • M
        mauro.tridici @stephenw10
        last edited by

        @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

        If there are still no icmp states then either there are firewall rules blocking that traffic or the pings never arrive at pfSense. A pcap on the pfSense B em2 interface should show that.

        I'm going to check this. I will let you know

        1 Reply Last reply Reply Quote 0
        • M
          mauro.tridici @stephenw10
          last edited by

          @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

          If there are still no icmp states then either there are firewall rules blocking that traffic or the pings never arrive at pfSense. A pcap on the pfSense B em2 interface should show that.

          pcap on pfsense B em2 interface is completely blank/empty

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Then the client PC there is not attached to em2 as it shows in the disagram or it is sending pings some other way, it has the wrong default route.

            M 1 Reply Last reply Reply Quote 0
            • M
              mauro.tridici @stephenw10
              last edited by

              @stephenw10

              VPCS on the left:

              VPCS> show

              NAME IP/MASK GATEWAY MAC LPORT RHOST:PORT
              VPCS1 192.168.118.10/24 192.168.118.1 00:50:79:66:68:01 20096 127.0.0.1:20097
              fe80::250:79ff:fe66:6801/64

              VPCs on the right:

              VPCS> show

              NAME IP/MASK GATEWAY MAC LPORT RHOST:PORT
              VPCS1 192.168.120.10/24 192.168.120.1 00:50:79:66:68:00 20098 127.0.0.1:20099
              fe80::250:79ff:fe66:6800/64

              Both the client PC an reach internet and other machines in the schema

              For example, the client PC on the left:

              VPCS> ping www.google.it
              www.google.it resolved to 142.250.184.35

              84 bytes from 142.250.184.35 icmp_seq=1 ttl=126 time=22.655 ms
              ^C
              VPCS> ping 10.99.0.2

              84 bytes from 10.99.0.2 icmp_seq=1 ttl=63 time=3.166 ms
              ^C
              VPCS> ping 10.99.0.1

              84 bytes from 10.99.0.1 icmp_seq=1 ttl=64 time=1.025 ms
              ^C
              VPCS> ping 192.168.122.180

              84 bytes from 192.168.122.180 icmp_seq=1 ttl=63 time=2.851 ms

              But if I try to make a ping from one client PC to the other one the pcap is empty.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Do you see those successful pings in pcaps?

                Those VPCS client machines only have one NIC?

                Either they are not sending the pings via pfSense or the pcap is filtering them out.

                M 1 Reply Last reply Reply Quote 0
                • M
                  mauro.tridici @stephenw10
                  last edited by

                  @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

                  Do you see those successful pings in pcaps?

                  Yes, I can see them in caps

                  Those VPCS client machines only have one NIC?

                  Yes, each VPCS has only one NIC

                  Either they are not sending the pings via pfSense or the pcap is filtering them out.

                  I don't think that pcap is filtering the pings.
                  Maybe something related to IPSEC tunnel because I just tried to ping pfsense B LAN IP (192.168.120.1) directly from pfsense A LAN IP (192.168.118.1) with no success.

                  During the next minutes, I would like to verify that static route on routerA is working as expected.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    If you try to ping 192.168.118.1 from the client behind pfSense B you must see those ping packets entering the interface at pfSense B.

                    What have you named that interface? (em2) Because it isn't the LAN. What firewall rules have you added on it?

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      mauro.tridici @stephenw10
                      last edited by

                      @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

                      If you try to ping 192.168.118.1 from the client behind pfSense B you must see those ping packets entering the interface at pfSense B.

                      Yes, I can confirm that. But the ping packets have not a response

                      Screenshot 2022-11-30 at 23.27.59.png

                      What have you named that interface? (em2) Because it isn't the LAN. What firewall rules have you added on it?

                      • em2 interface on pfsenseB is named "ACCESS"
                      • I added a very open rule "from any to any"

                      Screenshot 2022-11-30 at 23.33.01.png

                      I don't know if it can help, but please note that, in the IPSEC configuration page, I didn't set the name of the subnet "ACCESS" but the networks:

                      Screenshot 2022-11-30 at 23.35.08.png

                      This is for the pfsenseB. I did the same for the pfsenseA (changing the order).

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Ok, that looks good.

                        Earlier though you said those pings did not appear in a pcap in pfSense on the Access interface. Is that still the case?
                        It looks like that pcap was done on the client directly.

                        With that rule if they are arriving it should definitely create at least a state on the Access interface. And since the P2 matches it it should go out over the tunnel creating a state on IPSec too.

                        Steve

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          mauro.tridici @stephenw10
                          last edited by mauro.tridici

                          @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

                          Ok, that looks good.
                          Earlier though you said those pings did not appear in a pcap in pfSense on the Access interface. Is that still the case?

                          it is no longer happening.

                          With that rule if they are arriving it should definitely create at least a state on the Access interface. And since the P2 matches it it should go out over the tunnel creating a state on IPSec too.

                          I just repeated the test pinging 192.168.118.1 from PC with IP 192.168.120.10.
                          Ping is not working and the states for ACCESS interface are 0.

                          Screenshot 2022-12-01 at 11.30.24.png

                          Attaching the virtual Wireshark to the em0 port of RouterA I see these lines :

                          168 2022-12-01 11:35:34.467546 192.168.120.10 192.168.118.1 ICMP 146 Echo (ping) request id=0x004b, seq=956/48131, ttl=63 (no response found!)
                          169 2022-12-01 11:35:34.468024 192.168.122.230 192.168.122.180 ICMP 174 Destination unreachable (Protocol unreachable)

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by stephenw10

                            Ok, the states exist though and on both ACESS and IPSec. You can see both show 193 packets outbound and no replies.

                            So check the packet counters in the IPSec status again.

                            I would not expect to be able to see that traffic on the WAN dircetly, it should be encrypted.

                            However that 193 packet AH state on WAN looks suspicious. Do you have the Phase 2 protocol set to AH instead of ESP at one end?
                            It should always be ESP there, AH is unencrypted.

                            M 1 Reply Last reply Reply Quote 0
                            • M
                              mauro.tridici @stephenw10
                              last edited by mauro.tridici

                              @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

                              Ok, the states exist though and on both ACESS and IPSec. You can see both show 193 packets outbound and no replies.
                              So check the packet counters in the IPSec status again.
                              I would not expect to be able to see that traffic on the WAN dircetly, it should be encrypted.

                              Yes, you are right.

                              However that 193 packet AH state on WAN looks suspicious. Do you have the Phase 2 protocol set to AH instead of ESP at one end?
                              It should always be ESP there, AH is unencrypted.

                              P2 protocol was the same on both endpoints (it was AH, you are right).

                              Anyway, everything works as expected now. It was a Nat problem on the RouterA.
                              Many thanks for your help. You are a genius :)

                              As soon as possible I will share what was happening. right now I'm travelling

                              M 1 Reply Last reply Reply Quote 1
                              • M
                                mauro.tridici @mauro.tridici
                                last edited by

                                Hello Steve, @stephenw10

                                a last question for you about this case.
                                I was able to deploy the IPSEC Lan to Lan successfully (thanks to you).
                                Now it works using the transport network as you suggested. It is almost done.

                                Anyway, the throughput is a little bit slow (280Mbits/s but 1Gb/s link is available). I noticed that, during the previous tests between two PFsense 2.6CE instances, the throughput was about 900 Mbits/s. The test instances had each one a public IP.

                                Now, as in the previous test, AES-NI is enabled on both the endpoints, but the pfsense CE versions are different (2.6 and 2.5.2).
                                Do you think that this could be the reason of the poor performance?

                                In one of my previous post, a user said:

                                " If AES_NI is in the game pfSense since 2.6 CE or Plus version will benefit from that"

                                Thanks

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  It's probably just a factor of the latency between the sites. I imagine it's significantly higher than in your test setup?

                                  How are you testing the throughput?

                                  Steve

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    mauro.tridici @stephenw10
                                    last edited by mauro.tridici

                                    Good Morning Steve,

                                    here some additional info about the test and real scenario.

                                    TEST environment details:

                                    • n. 2 virtual machines with pfSense 2.6 CE with public IPs on WAN interfaces (let's say, 1.1.1.1/25 and 2.2.2.2/25);
                                    • both the virtual machines have been deployed on two VMware ESXi 6.7 U3 hypervisors (one per site);
                                    • the test has been made in order to test IPSEC feature and connectivity
                                    • both the hypervisors have AES-NI enabled on bios
                                    • both the virtual machines have the NetGate suggested configuration (AES-GCM, and so on).
                                    • the throughput has been checked using iperf2 (iperf2 client running in the LAN at site A, iperf2 server running in the LAN at site B)
                                    • the throughput is very near the 1Gb/s link capacity.

                                    REAL/PRODUCTION environment:

                                    • n. 2 virtual machines, one with pfSense 2.6 CE and the other with 2.5.2 CE;
                                    • only one pfsense (at site B) has a public IPs on WAN interface (let's say, 2.2.2.3/25). The other one is behind the router (please, refer to the schema above).
                                    • both the virtual machines have been deployed on two VMware ESXi 6.7 U3 hypervisors (one per site) - the same hypervisors mentioned above;
                                    • both the hypervisors have AES-NI enabled on bios
                                    • both the virtual machines have the NetGate suggested configuration (AES-GCM, and so on).
                                    • the throughput has been checked using iperf2 (iperf2 client running in the LAN at site A, iperf2 server running in the LAN at site B)
                                    • the throughput is about 280Mbits/s.
                                    • please, note that the iperf test between the router WAN interface (1.1.1.2/25) at site A and the pfsense Wan interface (2.2.2.3/25) at site B returns 890 Mbits/s.

                                    Do you think that the router at site A is doing something bad? Or it is really due to the different versions of pfSense?

                                    Thank you,
                                    Mauro

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Are those two environments actually between the same sites?

                                      What is the latency across the tunnel in each case? Or just between the WANs outside the tunnel?

                                      What were the iperf test conditions? Single TCP stream?

                                      If you can pass close to 900Mbps outside the tunnel I would expect to see better inside it. That test does not include pfSense at all though at site A? The router at site A could be restricting something.

                                      Is there any reason one of those VMs is still running 2.5.2?

                                      Steve

                                      M 1 Reply Last reply Reply Quote 0
                                      • M
                                        mauro.tridici @stephenw10
                                        last edited by mauro.tridici

                                        @stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

                                        Are those two environments actually between the same sites?

                                        Yes, the two environments are between the same sites (site A and site B) and I'm using the same hypervisors on each site.

                                        What is the latency across the tunnel in each case? Or just between the WANs outside the tunnel?

                                        I made a ping between one host in the LAN at site A (192.168.118.13/24) and one host in the LAN at site B (192.168.120.114/24) and viceversa.

                                        ping from 192.168.118.13 and 192.168.120.114

                                        PING 192.168.120.114 (192.168.120.114) 56(84) bytes of data.
                                        64 bytes from 192.168.120.114: icmp_seq=1 ttl=61 time=1.93 ms
                                        64 bytes from 192.168.120.114: icmp_seq=2 ttl=61 time=1.57 ms
                                        64 bytes from 192.168.120.114: icmp_seq=3 ttl=61 time=1.67 ms

                                        ping from 192.168.120.114 and 192.168.118.13

                                        PING 192.168.118.13 (192.168.118.13) 56(84) bytes of data.
                                        64 bytes from 192.168.118.13: icmp_seq=1 ttl=61 time=1.46 ms
                                        64 bytes from 192.168.118.13: icmp_seq=2 ttl=61 time=1.86 ms
                                        64 bytes from 192.168.118.13: icmp_seq=3 ttl=61 time=1.59 ms

                                        Or just between the WANs outside the tunnel?

                                        This is the ping output between the WAN interface of the router at site A and the WAN interface of pFSense at site B:

                                        ping x.x.177.2
                                        PING x.x.177.2 (x.x.177.2) 56(84) bytes of data.
                                        64 bytes from x.x.177.2: icmp_seq=1 ttl=57 time=1.06 ms
                                        64 bytes from x.x.177.2: icmp_seq=2 ttl=57 time=1.13 ms
                                        64 bytes from x.x.177.2: icmp_seq=3 ttl=57 time=1.18 ms
                                        64 bytes from x.x.177.2: icmp_seq=4 ttl=57 time=1.25 ms
                                        64 bytes from x.x.177.2: icmp_seq=5 ttl=57 time=0.947 ms

                                        This is the ping output between the WAN interface of the pfsense behind the router at site A and the WAN interface of pFSense at site B:

                                        [2.6.0-RELEASE][admin@L2L01.home.arpa]/root: ping x.x.177.2
                                        PING x.x.177.2 (x.x.177.2): 56 data bytes
                                        64 bytes from x.x.177.2: icmp_seq=0 ttl=56 time=1.381 ms
                                        64 bytes from x.x.177.2: icmp_seq=1 ttl=56 time=1.488 ms
                                        64 bytes from x.x.177.2: icmp_seq=2 ttl=56 time=1.465 ms
                                        64 bytes from x.x.177.2: icmp_seq=3 ttl=56 time=1.222 ms
                                        64 bytes from x.x.177.2: icmp_seq=4 ttl=56 time=1.143 ms

                                        What were the iperf test conditions? Single TCP stream?

                                        I made an iperf test (between the same hosts mentioned above) using single TCP stream and multiple TCP stream.

                                        This is the result with a single TCP stream:

                                        iperf2 -c 192.168.120.114

                                        Client connecting to 192.168.120.114, TCP port 5001
                                        TCP window size: 85.3 KByte (default)

                                        [ 3] local 192.168.118.13 port 42192 connected with 192.168.120.114 port 5001
                                        [ ID] Interval Transfer Bandwidth
                                        [ 3] 0.0-10.0 sec 286 MBytes 240 Mbits/sec

                                        And these are the results with 4, 8 and 16 parallel processes:

                                        iperf2 -c 192.168.120.114 -P 4

                                        Client connecting to 192.168.120.114, TCP port 5001
                                        TCP window size: 390 KByte (default)

                                        [ 5] local 192.168.118.13 port 43148 connected with 192.168.120.114 port 5001
                                        [ 3] local 192.168.118.13 port 43146 connected with 192.168.120.114 port 5001
                                        [ 4] local 192.168.118.13 port 43150 connected with 192.168.120.114 port 5001
                                        [ 6] local 192.168.118.13 port 43152 connected with 192.168.120.114 port 5001
                                        [ ID] Interval Transfer Bandwidth
                                        [ 4] 0.0-10.0 sec 153 MBytes 128 Mbits/sec
                                        [ 5] 0.0-10.0 sec 152 MBytes 127 Mbits/sec
                                        [ 3] 0.0-10.0 sec 154 MBytes 129 Mbits/sec
                                        [ 6] 0.0-10.0 sec 152 MBytes 128 Mbits/sec
                                        [SUM] 0.0-10.0 sec 611 MBytes 512 Mbits/sec

                                        iperf2 -c 192.168.120.114 -P 8

                                        Client connecting to 192.168.120.114, TCP port 5001
                                        TCP window size: 390 KByte (default)

                                        [ 8] local 192.168.118.13 port 43168 connected with 192.168.120.114 port 5001
                                        [ 6] local 192.168.118.13 port 43166 connected with 192.168.120.114 port 5001
                                        [ 7] local 192.168.118.13 port 43164 connected with 192.168.120.114 port 5001
                                        [ 3] local 192.168.118.13 port 43160 connected with 192.168.120.114 port 5001
                                        [ 5] local 192.168.118.13 port 43162 connected with 192.168.120.114 port 5001
                                        [ 9] local 192.168.118.13 port 43172 connected with 192.168.120.114 port 5001
                                        [ 10] local 192.168.118.13 port 43170 connected with 192.168.120.114 port 5001
                                        [ 4] local 192.168.118.13 port 43158 connected with 192.168.120.114 port 5001
                                        [ ID] Interval Transfer Bandwidth
                                        [ 9] 0.0-10.0 sec 61.1 MBytes 51.3 Mbits/sec
                                        [ 4] 0.0-10.0 sec 61.1 MBytes 51.3 Mbits/sec
                                        [ 6] 0.0-10.0 sec 59.0 MBytes 49.4 Mbits/sec
                                        [ 7] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
                                        [ 3] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
                                        [ 5] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
                                        [ 8] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
                                        [ 10] 0.0-10.0 sec 60.9 MBytes 51.0 Mbits/sec
                                        [SUM] 0.0-10.0 sec 486 MBytes 407 Mbits/sec

                                        iperf2 -c 192.168.120.114 -P 16

                                        Client connecting to 192.168.120.114, TCP port 5001
                                        TCP window size: 390 KByte (default)

                                        [ 18] local 192.168.118.13 port 43204 connected with 192.168.120.114 port 5001
                                        [ 6] local 192.168.118.13 port 43178 connected with 192.168.120.114 port 5001
                                        [ 4] local 192.168.118.13 port 43174 connected with 192.168.120.114 port 5001
                                        [ 7] local 192.168.118.13 port 43182 connected with 192.168.120.114 port 5001
                                        [ 9] local 192.168.118.13 port 43184 connected with 192.168.120.114 port 5001
                                        [ 3] local 192.168.118.13 port 43180 connected with 192.168.120.114 port 5001
                                        [ 5] local 192.168.118.13 port 43176 connected with 192.168.120.114 port 5001
                                        [ 8] local 192.168.118.13 port 43186 connected with 192.168.120.114 port 5001
                                        [ 10] local 192.168.118.13 port 43188 connected with 192.168.120.114 port 5001
                                        [ 14] local 192.168.118.13 port 43196 connected with 192.168.120.114 port 5001
                                        [ 12] local 192.168.118.13 port 43192 connected with 192.168.120.114 port 5001
                                        [ 15] local 192.168.118.13 port 43198 connected with 192.168.120.114 port 5001
                                        [ 13] local 192.168.118.13 port 43194 connected with 192.168.120.114 port 5001
                                        [ 16] local 192.168.118.13 port 43200 connected with 192.168.120.114 port 5001
                                        [ 11] local 192.168.118.13 port 43190 connected with 192.168.120.114 port 5001
                                        [ 17] local 192.168.118.13 port 43202 connected with 192.168.120.114 port 5001
                                        [ ID] Interval Transfer Bandwidth
                                        [ 7] 0.0-10.0 sec 41.0 MBytes 34.4 Mbits/sec
                                        [ 5] 0.0-10.0 sec 41.0 MBytes 34.4 Mbits/sec
                                        [ 14] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
                                        [ 12] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
                                        [ 15] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
                                        [ 4] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
                                        [ 3] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
                                        [ 8] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
                                        [ 13] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
                                        [ 17] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
                                        [ 6] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
                                        [ 9] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
                                        [ 10] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
                                        [ 11] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
                                        [ 18] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
                                        [ 16] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
                                        [SUM] 0.0-10.0 sec 658 MBytes 550 Mbits/sec

                                        Please, note that the results are not the always the same. Sometimes, It seems to be unstable. The basic value is always about 240/280 Mbits/s.
                                        Sometimes, "write failed: Connection reset by peer" message appears in the iperf2 output when I use parallel streams.
                                        The error message doesn't aappear if I use a single stream, but the bitrate is very low.

                                        iperf2 -c 192.168.120.114 -P 16

                                        Client connecting to 192.168.120.114, TCP port 5001
                                        TCP window size: 390 KByte (default)

                                        [ 16] local 192.168.118.13 port 43260 connected with 192.168.120.114 port 5001
                                        [ 3] local 192.168.118.13 port 43238 connected with 192.168.120.114 port 5001
                                        [ 8] local 192.168.118.13 port 43242 connected with 192.168.120.114 port 5001
                                        [ 4] local 192.168.118.13 port 43234 connected with 192.168.120.114 port 5001
                                        [ 5] local 192.168.118.13 port 43236 connected with 192.168.120.114 port 5001
                                        [ 9] local 192.168.118.13 port 43246 connected with 192.168.120.114 port 5001
                                        [ 7] local 192.168.118.13 port 43240 connected with 192.168.120.114 port 5001
                                        [ 6] local 192.168.118.13 port 43244 connected with 192.168.120.114 port 5001
                                        [ 11] local 192.168.118.13 port 43250 connected with 192.168.120.114 port 5001
                                        [ 13] local 192.168.118.13 port 43254 connected with 192.168.120.114 port 5001
                                        [ 14] local 192.168.118.13 port 43258 connected with 192.168.120.114 port 5001
                                        [ 15] local 192.168.118.13 port 43256 connected with 192.168.120.114 port 5001
                                        [ 12] local 192.168.118.13 port 43252 connected with 192.168.120.114 port 5001
                                        [ 10] local 192.168.118.13 port 43248 connected with 192.168.120.114 port 5001
                                        [ 18] local 192.168.118.13 port 43264 connected with 192.168.120.114 port 5001
                                        [ 17] local 192.168.118.13 port 43262 connected with 192.168.120.114 port 5001
                                        write failed: Connection reset by peer
                                        write failed: Connection reset by peer
                                        [ ID] Interval Transfer Bandwidth
                                        [ 18] 0.0- 0.0 sec 358 KBytes 229 Mbits/sec
                                        [ 17] 0.0- 0.0 sec 358 KBytes 220 Mbits/sec
                                        [ 16] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
                                        [ 3] 0.0-10.7 sec 3.64 MBytes 2.86 Mbits/sec
                                        [ 4] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
                                        [ 5] 0.0-10.7 sec 3.44 MBytes 2.70 Mbits/sec
                                        [ 9] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
                                        [ 7] 0.0-10.7 sec 3.41 MBytes 2.67 Mbits/sec
                                        [ 6] 0.0-10.7 sec 3.59 MBytes 2.82 Mbits/sec
                                        [ 11] 0.0-10.7 sec 3.62 MBytes 2.83 Mbits/sec
                                        [ 13] 0.0-10.7 sec 3.64 MBytes 2.86 Mbits/sec
                                        [ 14] 0.0-10.7 sec 2.52 MBytes 1.97 Mbits/sec
                                        [ 15] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
                                        [ 10] 0.0-10.7 sec 3.41 MBytes 2.68 Mbits/sec
                                        [ 8] 0.0-10.7 sec 3.62 MBytes 2.83 Mbits/sec
                                        [ 12] 0.0-10.8 sec 3.46 MBytes 2.69 Mbits/sec
                                        [SUM] 0.0-10.8 sec 49.5 MBytes 38.5 Mbits/sec

                                        I also tried to change the net.core.somaxconn in sysctl.conf on both the endpoints (as suggested in the link below), but the behaviour didn't change.

                                        https://spyff.github.io/net/2019/07/14/many-tcp-flow-with-iperf/#the-problem-of-small-backlog

                                        If you can pass close to 900Mbps outside the tunnel I would expect to see better inside it.

                                        Yes, it is exactly what I would expect.

                                        That test does not include pfSense at all though at site A?

                                        Yes, some test I made include pfsense at site A (iperf test through the tunnel)

                                        The router at site A could be restricting something.

                                        I hope that the router is not the cause...I'm your hands :)

                                        Is there any reason one of those VMs is still running 2.5.2?

                                        No, it has been created some months ago and it is in production. If you think that it should updated, we will schedule a downtime to do it.

                                        Steve

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          You probably need to do a test between the pfSense instances but outside the tunnel. Doing that would prove you are not being throttled by Router A for some reason. You can run iperf dircetly on pfSense A to test that. The speed to/from iperf running in pfSense will always be lower that through it but it should easily pass >280Mbps across a 2ms link.

                                          M 1 Reply Last reply Reply Quote 0
                                          • M
                                            mauro.tridici @stephenw10
                                            last edited by

                                            @stephenw10

                                            Thank you for your reply, Steve.

                                            This is the result of the iperf3 test between the pfsense at site A (10.99.0.2) and the pfsense at site B (x.x.177.2/25). It is a test outside the tunnel as you suggested.

                                            [2.6.0-RELEASE][admin@L2L01.home.arpa]/root: iperf3 -c x.x.177.2
                                            Connecting to host x.x.177.2, port 5201
                                            [ 5] local 10.99.0.2 port 24218 connected to 90.147.177.2 port 5201
                                            [ ID] Interval Transfer Bitrate Retr Cwnd
                                            [ 5] 0.00-1.00 sec 83.5 MBytes 701 Mbits/sec 77 283 KBytes
                                            [ 5] 1.00-2.00 sec 104 MBytes 872 Mbits/sec 7 428 KBytes
                                            [ 5] 2.00-3.00 sec 103 MBytes 867 Mbits/sec 4 488 KBytes
                                            [ 5] 3.00-4.00 sec 101 MBytes 847 Mbits/sec 126 381 KBytes
                                            [ 5] 4.00-5.00 sec 105 MBytes 883 Mbits/sec 0 673 KBytes
                                            [ 5] 5.00-6.00 sec 103 MBytes 861 Mbits/sec 9 539 KBytes
                                            [ 5] 6.00-7.00 sec 99.3 MBytes 833 Mbits/sec 12 535 KBytes
                                            [ 5] 7.00-8.00 sec 99.0 MBytes 831 Mbits/sec 35 426 KBytes
                                            [ 5] 8.00-9.00 sec 99.9 MBytes 838 Mbits/sec 0 689 KBytes
                                            [ 5] 9.00-10.00 sec 98.3 MBytes 825 Mbits/sec 0 872 KBytes

                                            [ ID] Interval Transfer Bitrate Retr
                                            [ 5] 0.00-10.00 sec 996 MBytes 836 Mbits/sec 270 sender
                                            [ 5] 0.00-10.01 sec 995 MBytes 834 Mbits/sec receiver

                                            It seems to be a good result.
                                            So, if I'm not wrong, the problem is not the router, right?

                                            Thank you,
                                            Mauro

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.