2.0-RELEASE: Performance oddity?
-
Hi Y'all,
I'm running a 2.0-RELEASE pfSense box on a Dell R610 with four 3.5GHz bleeding edge Xeon cores. It's got a top-o-the-line MyriCom 10G NIC in it.
For the past couple days I've been moving ~2Gb/s data inbound. CPU is low on the box (like 3-4%) and the interrupts are also fairly low (like 15-18% or so). Then I did something that must have freaked it out in some way. I VPN'd in to the pfSense OpenVPN server. This was fine, barely any traffic there, just an SSH shell into a server through the VPN. Then I tried rsyncing a 100MB file from a server to another server outside the pfSense firewall in another city (10Gb/s path end-to-end). The file, astonishingly enough, copied at like 500Kb/s, and meanwhile, my 2Gb/s traffic inbound to the firewall slowed to like 100Mb/s. After what seemed like ages, the 100MB file finished transferring, and magically the inbound traffic went from 100Mb/s back to 2Gb/s. During the 100MB file transfer outbound, even my shell connection via the VPN was responding slowly.
During the weird "slow" period CPU was not any different, nor were the interrupts. I have attached a CPU graph showing only nominal activity. The RRD "traffic" graph, weirdly enough, does not even show the 2Gb/s traffic although I've been watching it on my router graphs, it shows barely any traffic. I attached an image of that as well. Finally, I also attached the live traffic graph showing ~2Gb/s just to prove I'm not insane.
I guess my question is: Why, when I copy a little 100MB file outbound from the firewall, does my inbound traffic tank?
Thanks for any advice!
-
I just tried it again - totally reproducible, every time I try an outbound rsync, it's slow as mud and my inbound traffic tanks at that very moment. Very odd.
-
Other notes: The 100MB file transfer I was speaking of before was NOT through the VPN connection I mentioned earlier. I was just VPN'd in to get to the servers inside. The file transfer was a simple file transfer outbound through the firewall to a host on the internet.
I'm doing some 1:1 NAT as well, don't know if that makes a difference.
-
Did you check your duplex setting on your NIC? I have heard that sometimes not hard setting it can cause issues.
Are you doing any traffic shaping?Can you send me your setup …. so jealous.
-
Hmm… Duplex? Didn't even think about that, but it makes sense. Looks like things are indeed full duplex:
$ ifconfig mxge0
mxge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
options=400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso>ether 00:60:dd:45:5a:a2
inet 111.0.0.11 netmask 0xffffff00 broadcast 111.0.0.255
inet6 fe80::260:ddff:fe45:5aa2%mxge0 prefixlen 64 scopeid 0x5
nd6 options=3 <performnud,accept_rtadv>media: Ethernet 10Gbase-SR <full-duplex>status: activeSo I guess that isn't the problem... Although it's got me thinking, because it sure does seem like a duplex problem in some way, shape or form. However, the weird thing is that both sides drop bandwidth at the same time when I send egress traffic, it's not exactly a tradeoff of one or the other that is fast.
I'm not using traffic shaping. Just whatever the defaults are.</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
-
I don't know too much about 10G.
Do you have the NIC in promiscuous mode? If so, why?
Is the switch and all network nodes setup for jumbo frames (MTU 9000)?Are you running 64 bit pfsense or 32bit pfsense? How much memory does the system have?
-
Why, when I copy a little 100MB file outbound from the firewall, does my inbound traffic tank?
What happens if you copy a file over the same path (ssh & openvpn) but in the other direction, i.e. from your remote server -> firewall ?
Also may also want to try running vmstat 2 10 from the pfsense shell, while doing the rsync transfer.
And vmstat -z before starting and after the end of the rsync transfer.PS: Also try a search in http://lists.freebsd.org/mailman/listinfo/freebsd-net
-
Hmm, I don't think I have the NIC in promiscuous mode, at least I didn't do it on purpose. That's mostly used when doing packet tracing on the NIC right? I haven't been doing any of that.
The firewall box is indeed using 9000 byte frames, and it has 24GB RAM.
Because the box doesn't seem to increase in load during an outbound rsync, but instead traffic and load drop, I'm kind of wondering if the router right outside of my firewall is the real problem, and not the firewall. The jumbo frame issue is an interesting one that I have not thought of. Incoming packets are not jumbo, probably the usual 1500 bytes. However outgoing ones are indeed 9000 bytes. The NOC tells me that the path to their gateway from my firewall is jumbo clean, but I wonder if some router in the middle is having to fragment the packets and that's where the problem is happening? Like, maybe the router is choking when trying to fragment that many packets?
I wonder what would happen if I changed my WAN facing interface MTU on my firewall back to 1500 bytes (the LAN interface is 9000) and let my firewall do the fragmentation? Hmm… Lemme try that and see if the CPU freaks out...
I will also try running the vmstat test as suggested.
-
I ran the vmstat commands while transferring, here they are:
[2.0-RELEASE][admin@fire01.xxx]/root(1): vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 108, 11, 108, 0 UMA Zones: 448, 0, 108, 4, 108, 0 UMA Slabs: 568, 0, 1634, 4, 10719, 0 UMA RCntSlabs: 568, 0, 14027, 1, 14027, 0 UMA Hash: 256, 0, 1, 14, 3, 0 16 Bucket: 152, 0, 206, 19, 206, 0 32 Bucket: 280, 0, 316, 6, 316, 0 64 Bucket: 536, 0, 338, 5, 338, 79 128 Bucket: 1048, 0, 323, 1, 323, 877 VM OBJECT: 216, 0, 2170, 1070, 21533629, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 786253, 64, 494, 25805, 0 MAP ENTRY: 120, 0, 1628, 1100, 45538101, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 313, 14, 313, 0 16: 16, 0, 2252, 1444, 2984997, 0 32: 32, 0, 3286, 1360, 7405041, 0 64: 64, 0, 3287, 1585, 57246324442, 0 128: 128, 0, 4456, 1518, 3529376, 0 256: 256, 0, 715, 1355, 9017640, 0 512: 512, 0, 856, 880, 2098526, 0 1024: 1024, 0, 57, 219, 245486, 0 2048: 2048, 0, 104, 148, 26522, 0 4096: 4096, 0, 412, 832, 1056263, 0 Files: 80, 0, 161, 784, 7208760, 0 TURNSTILE: 136, 0, 517, 103, 517, 0 umtx pi: 96, 0, 0, 0, 0, 0 PROC: 1120, 0, 58, 329, 985343, 0 THREAD: 984, 0, 461, 55, 461, 0 SLEEPQUEUE: 80, 0, 517, 121, 517, 0 VMSPACE: 392, 0, 35, 465, 985065, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 512, 0, 16320, 2112, 2086392, 0 mbuf: 512, 0, 2051, 2813, 59098587505, 0 mbuf_cluster: 2048, 25600, 18434, 1092, 1847171876, 0 mbuf_jumbo_page: 4096, 12800, 0, 215, 5241, 0 mbuf_jumbo_9k: 9216, 6400, 1024, 3025, 26158196178, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 NetGraph items: 72, 4118, 0, 290, 826, 0 NetGraph data items: 72, 522, 0, 0, 0, 0 g_bio: 232, 0, 0, 800, 2215015, 0 ttyinq: 160, 0, 30, 114, 165, 0 ttyoutq: 256, 0, 16, 59, 88, 0 ata_request: 320, 0, 0, 36, 35, 0 ata_composite: 336, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0 VNODE: 472, 0, 951, 681, 35521, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 870, 549, 394338, 0 L VFS Cache: 328, 0, 36, 96, 49, 0 NAMEI: 1024, 0, 0, 256, 18976211, 0 NFSMOUNT: 616, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 25, 25, 0 pipe: 728, 0, 5, 340, 652566, 0 ksiginfo: 112, 0, 374, 682, 14651, 0 itimer: 344, 0, 0, 0, 0, 0 ng_pipe: 64, 0, 0, 0, 0, 0 bridge_rtnode: 64, 0, 0, 0, 0, 0 KNOTE: 128, 0, 11, 424, 371227, 0 socket: 680, 25602, 53, 283, 191779, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 11, 231, 117670, 0 udpcb: 16, 25704, 11, 1165, 117670, 0 tcp_inpcb: 336, 25608, 15, 271, 5548, 0 tcpcb: 880, 25600, 11, 149, 5548, 0 tcptw: 80, 5130, 4, 581, 5481, 0 syncache: 144, 15366, 0, 234, 5490, 0 hostcache: 136, 15372, 3, 165, 10, 0 tcpreass: 40, 1680, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0 sctp_ep: 1272, 25602, 0, 0, 0, 0 sctp_asoc: 2232, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 432, 18, 0 sctp_raddr: 616, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 25608, 2, 141, 288, 0 unpcb: 240, 25600, 23, 377, 67205, 0 rtentry: 200, 0, 59, 321, 1679, 0 pfsrctrpl: 152, 2369000, 0, 0, 0, 0 pfrulepl: 992, 0, 125, 355, 15293, 0 pfstatepl: 400, 2369007, 122, 1039, 389896, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 55, 449, 6552, 0 pfrktable: 1296, 1002, 9, 123, 2896, 0 pfrkentry: 216, 100008, 43, 227, 2587, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 606, 52, 0 pffrag: 80, 0, 0, 270, 26, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 3, 183, 338, 0 pfospfen: 112, 0, 696, 195, 117624, 0 pfosfp: 40, 0, 407, 517, 68783, 0 selfd: 56, 0, 602, 1162, 14536063, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 3, 12, 5, 0 FFS inode: 168, 0, 900, 552, 35469, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 900, 570, 35469, 0 md0: 512, 0, 298, 45, 298, 0 [2.0-RELEASE][admin@fire01.xxx]/root(3): vmstat 2 10 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mf0 md0 in sy cs us sy id 0 0 0 591M 23G 389 0 0 0 394 0 0 0 2603 457 11606 0 7 93 0 0 0 591M 23G 2 0 0 3 1 0 20 0 45 294 2251 0 0 100 0 0 0 591M 23G 5580 0 0 0 5328 0 15 0 98 5260 2567 1 1 98 0 0 0 591M 23G 199 0 0 0 178 0 0 0 104 443 2441 0 0 100 0 0 0 685M 23G 3760 0 0 0 1056 0 12 1 40 3235 2328 1 0 99 0 0 0 591M 23G 5890 0 0 0 8161 0 15 2 55 5844 2459 1 1 97 0 0 0 591M 23G 0 0 0 0 0 0 0 0 696 272 3826 0 0 100 0 0 0 591M 23G 7184 0 0 0 6670 0 13 43 2628 7939 9498 1 1 98 0 0 0 591M 23G 4265 0 0 0 4064 0 12 2 2236 3930 8134 1 1 99 0 0 0 591M 23G 0 0 0 0 0 0 0 0 35 133 2231 0 0 100 [2.0-RELEASE][admin@fire01.xxx]/root(4): vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 108, 11, 108, 0 UMA Zones: 448, 0, 108, 4, 108, 0 UMA Slabs: 568, 0, 1634, 4, 10720, 0 UMA RCntSlabs: 568, 0, 14027, 1, 14027, 0 UMA Hash: 256, 0, 1, 14, 3, 0 16 Bucket: 152, 0, 206, 19, 206, 0 32 Bucket: 280, 0, 317, 5, 317, 0 64 Bucket: 536, 0, 338, 5, 338, 79 128 Bucket: 1048, 0, 323, 1, 323, 877 VM OBJECT: 216, 0, 2190, 1050, 21560351, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 786253, 64, 494, 25807, 0 MAP ENTRY: 120, 0, 1660, 1130, 45601264, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 313, 14, 313, 0 16: 16, 0, 2252, 1444, 2989568, 0 32: 32, 0, 3277, 1369, 7414238, 0 64: 64, 0, 3288, 1584, 57246971299, 0 128: 128, 0, 4456, 1518, 3580711, 0 256: 256, 0, 699, 1371, 9043599, 0 512: 512, 0, 857, 879, 2101141, 0 1024: 1024, 0, 57, 219, 246302, 0 2048: 2048, 0, 104, 148, 27604, 0 4096: 4096, 0, 413, 831, 1058172, 0 Files: 80, 0, 159, 786, 7223293, 0 TURNSTILE: 136, 0, 517, 103, 517, 0 umtx pi: 96, 0, 0, 0, 0, 0 PROC: 1120, 0, 59, 328, 986350, 0 THREAD: 984, 0, 461, 55, 461, 0 SLEEPQUEUE: 80, 0, 517, 121, 517, 0 VMSPACE: 392, 0, 36, 464, 986069, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 512, 0, 16320, 2112, 2087752, 0 mbuf: 512, 0, 2051, 2813, 59099224460, 0 mbuf_cluster: 2048, 25600, 18434, 1092, 1847178111, 0 mbuf_jumbo_page: 4096, 12800, 0, 215, 5241, 0 mbuf_jumbo_9k: 9216, 6400, 1024, 3025, 26158415019, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 NetGraph items: 72, 4118, 0, 290, 850, 0 NetGraph data items: 72, 522, 0, 0, 0, 0 g_bio: 232, 0, 0, 800, 2219245, 0 ttyinq: 160, 0, 30, 114, 165, 0 ttyoutq: 256, 0, 16, 59, 88, 0 ata_request: 320, 0, 0, 36, 35, 0 ata_composite: 336, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0 VNODE: 472, 0, 951, 681, 35688, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 869, 550, 395855, 0 L VFS Cache: 328, 0, 36, 96, 49, 0 NAMEI: 1024, 0, 0, 256, 19024705, 0 NFSMOUNT: 616, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 25, 25, 0 pipe: 728, 0, 6, 339, 653310, 0 ksiginfo: 112, 0, 374, 682, 14673, 0 itimer: 344, 0, 0, 0, 0, 0 ng_pipe: 64, 0, 0, 0, 0, 0 bridge_rtnode: 64, 0, 0, 0, 0, 0 KNOTE: 128, 0, 6, 429, 371606, 0 socket: 680, 25602, 44, 292, 193187, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 11, 231, 118898, 0 udpcb: 16, 25704, 11, 1165, 118898, 0 tcp_inpcb: 336, 25608, 6, 280, 5548, 0 tcpcb: 880, 25600, 6, 154, 5548, 0 tcptw: 80, 5130, 0, 585, 5486, 0 syncache: 144, 15366, 0, 234, 5490, 0 hostcache: 136, 15372, 3, 165, 10, 0 tcpreass: 40, 1680, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0 sctp_ep: 1272, 25602, 0, 0, 0, 0 sctp_asoc: 2232, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 432, 18, 0 sctp_raddr: 616, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 25608, 2, 141, 298, 0 unpcb: 240, 25600, 23, 377, 67339, 0 rtentry: 200, 0, 59, 321, 1715, 0 pfsrctrpl: 152, 2369000, 0, 0, 0, 0 pfrulepl: 992, 0, 128, 352, 15937, 0 pfstatepl: 400, 2369007, 77, 1084, 390181, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 54, 450, 6825, 0 pfrktable: 1296, 1002, 9, 123, 2959, 0 pfrkentry: 216, 100008, 43, 227, 2692, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 606, 52, 0 pffrag: 80, 0, 0, 270, 26, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 4, 182, 359, 0 pfospfen: 112, 0, 696, 195, 122496, 0 pfosfp: 40, 0, 407, 517, 71632, 0 selfd: 56, 0, 601, 1163, 14558370, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 3, 12, 5, 0 FFS inode: 168, 0, 900, 552, 35636, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 900, 570, 35636, 0 md0: 512, 0, 301, 70, 301, 0 -
And I just tried setting the WAN MTU to 1500 and trying again, same results… What a mystery. So I reset it to 9000.
-
And another interesting factoid to consider…. Whenever I try to rsync that file outbound to the internet from inside the firewall, it seems the firewall state table shoots way up. I don't know why this would be since the rsync is a single socket connection, I can't imagine why it would generate 200+ states in the state table... Screenshot of the state graph attached. The activity on the far right of the graph was generated by me copying the file outbound.
-
Sorry, forgot to answer an earlier question… Yes, this is the amd64 version. And all the hosts and switches are dialed for jumbo frames. Tested and verified.
-
What is your provided speed (up and down) by your ISP?
http://cable-dsl.navasgroup.com/#Asymmetry
-
The outbound and inbound speed is 10Gb/s (full duplex synchronous) via CENIC in California.
-
Hmm, I don't think I have the NIC in promiscuous mode, at least I didn't do it on purpose. That's mostly used when doing packet tracing on the NIC right? I haven't been doing any of that.
Well, after consulting with a very knowledgeable friend of mine, it appears I do have the WAN interface in promiscuous mode, but it appears that it is that way because of pfsync/pflog because I'm using virtual floating IPs via CARP or something. Perhaps the pfSense devs know more on that. But I don't know if that is the root of the problem? Maybe… But I need CARP and virtual IPs so I hope it isn't the problem... ;)
-
Well, it is generally a bad idea to put that on WAN or LAN. It is better to have that on a dedicated NIC. As a test, disable pfsync. Promiscuous is not enabled by CARP VIPs only.
-
Hmm.. I do have a dedicated NIC for pfSync/CARP already? And it's not the WAN one. ;) I'm just using RFC1918 space for the dedicated CARP interface. I attached 2 screenshots showing my CARP config. It should be noted that I'm syncing everything available however.
But maybe I did something incorrectly there, let me know if so!
-
Is CARP a vLAN on on the WAN or LAN interface?
-
No, I'm not using VLANs in this model. The CARP interface uses a dedicated CAT5e cable connected in a crossover fashion from one physical interface of the "left" firewall to one physical interface on the "right" firewall, on a uniquely defined network that does not overlap any other nearby network.
-
Hmm.. I do have a dedicated NIC for pfSync/CARP already? And it's not the WAN one. ;) I'm just using RFC1918 space for the dedicated CARP interface. I attached 2 screenshots showing my CARP config. It should be noted that I'm syncing everything available however.
But maybe I did something incorrectly there, let me know if so!
CARP DOC says that it needs public ip to function
