2.0-RELEASE: Performance oddity?
-
Hmm, I don't think I have the NIC in promiscuous mode, at least I didn't do it on purpose. That's mostly used when doing packet tracing on the NIC right? I haven't been doing any of that.
The firewall box is indeed using 9000 byte frames, and it has 24GB RAM.
Because the box doesn't seem to increase in load during an outbound rsync, but instead traffic and load drop, I'm kind of wondering if the router right outside of my firewall is the real problem, and not the firewall. The jumbo frame issue is an interesting one that I have not thought of. Incoming packets are not jumbo, probably the usual 1500 bytes. However outgoing ones are indeed 9000 bytes. The NOC tells me that the path to their gateway from my firewall is jumbo clean, but I wonder if some router in the middle is having to fragment the packets and that's where the problem is happening? Like, maybe the router is choking when trying to fragment that many packets?
I wonder what would happen if I changed my WAN facing interface MTU on my firewall back to 1500 bytes (the LAN interface is 9000) and let my firewall do the fragmentation? Hmm… Lemme try that and see if the CPU freaks out...
I will also try running the vmstat test as suggested.
-
I ran the vmstat commands while transferring, here they are:
[2.0-RELEASE][admin@fire01.xxx]/root(1): vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 108, 11, 108, 0 UMA Zones: 448, 0, 108, 4, 108, 0 UMA Slabs: 568, 0, 1634, 4, 10719, 0 UMA RCntSlabs: 568, 0, 14027, 1, 14027, 0 UMA Hash: 256, 0, 1, 14, 3, 0 16 Bucket: 152, 0, 206, 19, 206, 0 32 Bucket: 280, 0, 316, 6, 316, 0 64 Bucket: 536, 0, 338, 5, 338, 79 128 Bucket: 1048, 0, 323, 1, 323, 877 VM OBJECT: 216, 0, 2170, 1070, 21533629, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 786253, 64, 494, 25805, 0 MAP ENTRY: 120, 0, 1628, 1100, 45538101, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 313, 14, 313, 0 16: 16, 0, 2252, 1444, 2984997, 0 32: 32, 0, 3286, 1360, 7405041, 0 64: 64, 0, 3287, 1585, 57246324442, 0 128: 128, 0, 4456, 1518, 3529376, 0 256: 256, 0, 715, 1355, 9017640, 0 512: 512, 0, 856, 880, 2098526, 0 1024: 1024, 0, 57, 219, 245486, 0 2048: 2048, 0, 104, 148, 26522, 0 4096: 4096, 0, 412, 832, 1056263, 0 Files: 80, 0, 161, 784, 7208760, 0 TURNSTILE: 136, 0, 517, 103, 517, 0 umtx pi: 96, 0, 0, 0, 0, 0 PROC: 1120, 0, 58, 329, 985343, 0 THREAD: 984, 0, 461, 55, 461, 0 SLEEPQUEUE: 80, 0, 517, 121, 517, 0 VMSPACE: 392, 0, 35, 465, 985065, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 512, 0, 16320, 2112, 2086392, 0 mbuf: 512, 0, 2051, 2813, 59098587505, 0 mbuf_cluster: 2048, 25600, 18434, 1092, 1847171876, 0 mbuf_jumbo_page: 4096, 12800, 0, 215, 5241, 0 mbuf_jumbo_9k: 9216, 6400, 1024, 3025, 26158196178, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 NetGraph items: 72, 4118, 0, 290, 826, 0 NetGraph data items: 72, 522, 0, 0, 0, 0 g_bio: 232, 0, 0, 800, 2215015, 0 ttyinq: 160, 0, 30, 114, 165, 0 ttyoutq: 256, 0, 16, 59, 88, 0 ata_request: 320, 0, 0, 36, 35, 0 ata_composite: 336, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0 VNODE: 472, 0, 951, 681, 35521, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 870, 549, 394338, 0 L VFS Cache: 328, 0, 36, 96, 49, 0 NAMEI: 1024, 0, 0, 256, 18976211, 0 NFSMOUNT: 616, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 25, 25, 0 pipe: 728, 0, 5, 340, 652566, 0 ksiginfo: 112, 0, 374, 682, 14651, 0 itimer: 344, 0, 0, 0, 0, 0 ng_pipe: 64, 0, 0, 0, 0, 0 bridge_rtnode: 64, 0, 0, 0, 0, 0 KNOTE: 128, 0, 11, 424, 371227, 0 socket: 680, 25602, 53, 283, 191779, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 11, 231, 117670, 0 udpcb: 16, 25704, 11, 1165, 117670, 0 tcp_inpcb: 336, 25608, 15, 271, 5548, 0 tcpcb: 880, 25600, 11, 149, 5548, 0 tcptw: 80, 5130, 4, 581, 5481, 0 syncache: 144, 15366, 0, 234, 5490, 0 hostcache: 136, 15372, 3, 165, 10, 0 tcpreass: 40, 1680, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0 sctp_ep: 1272, 25602, 0, 0, 0, 0 sctp_asoc: 2232, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 432, 18, 0 sctp_raddr: 616, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 25608, 2, 141, 288, 0 unpcb: 240, 25600, 23, 377, 67205, 0 rtentry: 200, 0, 59, 321, 1679, 0 pfsrctrpl: 152, 2369000, 0, 0, 0, 0 pfrulepl: 992, 0, 125, 355, 15293, 0 pfstatepl: 400, 2369007, 122, 1039, 389896, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 55, 449, 6552, 0 pfrktable: 1296, 1002, 9, 123, 2896, 0 pfrkentry: 216, 100008, 43, 227, 2587, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 606, 52, 0 pffrag: 80, 0, 0, 270, 26, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 3, 183, 338, 0 pfospfen: 112, 0, 696, 195, 117624, 0 pfosfp: 40, 0, 407, 517, 68783, 0 selfd: 56, 0, 602, 1162, 14536063, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 3, 12, 5, 0 FFS inode: 168, 0, 900, 552, 35469, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 900, 570, 35469, 0 md0: 512, 0, 298, 45, 298, 0 [2.0-RELEASE][admin@fire01.xxx]/root(3): vmstat 2 10 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mf0 md0 in sy cs us sy id 0 0 0 591M 23G 389 0 0 0 394 0 0 0 2603 457 11606 0 7 93 0 0 0 591M 23G 2 0 0 3 1 0 20 0 45 294 2251 0 0 100 0 0 0 591M 23G 5580 0 0 0 5328 0 15 0 98 5260 2567 1 1 98 0 0 0 591M 23G 199 0 0 0 178 0 0 0 104 443 2441 0 0 100 0 0 0 685M 23G 3760 0 0 0 1056 0 12 1 40 3235 2328 1 0 99 0 0 0 591M 23G 5890 0 0 0 8161 0 15 2 55 5844 2459 1 1 97 0 0 0 591M 23G 0 0 0 0 0 0 0 0 696 272 3826 0 0 100 0 0 0 591M 23G 7184 0 0 0 6670 0 13 43 2628 7939 9498 1 1 98 0 0 0 591M 23G 4265 0 0 0 4064 0 12 2 2236 3930 8134 1 1 99 0 0 0 591M 23G 0 0 0 0 0 0 0 0 35 133 2231 0 0 100 [2.0-RELEASE][admin@fire01.xxx]/root(4): vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 108, 11, 108, 0 UMA Zones: 448, 0, 108, 4, 108, 0 UMA Slabs: 568, 0, 1634, 4, 10720, 0 UMA RCntSlabs: 568, 0, 14027, 1, 14027, 0 UMA Hash: 256, 0, 1, 14, 3, 0 16 Bucket: 152, 0, 206, 19, 206, 0 32 Bucket: 280, 0, 317, 5, 317, 0 64 Bucket: 536, 0, 338, 5, 338, 79 128 Bucket: 1048, 0, 323, 1, 323, 877 VM OBJECT: 216, 0, 2190, 1050, 21560351, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 786253, 64, 494, 25807, 0 MAP ENTRY: 120, 0, 1660, 1130, 45601264, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 313, 14, 313, 0 16: 16, 0, 2252, 1444, 2989568, 0 32: 32, 0, 3277, 1369, 7414238, 0 64: 64, 0, 3288, 1584, 57246971299, 0 128: 128, 0, 4456, 1518, 3580711, 0 256: 256, 0, 699, 1371, 9043599, 0 512: 512, 0, 857, 879, 2101141, 0 1024: 1024, 0, 57, 219, 246302, 0 2048: 2048, 0, 104, 148, 27604, 0 4096: 4096, 0, 413, 831, 1058172, 0 Files: 80, 0, 159, 786, 7223293, 0 TURNSTILE: 136, 0, 517, 103, 517, 0 umtx pi: 96, 0, 0, 0, 0, 0 PROC: 1120, 0, 59, 328, 986350, 0 THREAD: 984, 0, 461, 55, 461, 0 SLEEPQUEUE: 80, 0, 517, 121, 517, 0 VMSPACE: 392, 0, 36, 464, 986069, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 512, 0, 16320, 2112, 2087752, 0 mbuf: 512, 0, 2051, 2813, 59099224460, 0 mbuf_cluster: 2048, 25600, 18434, 1092, 1847178111, 0 mbuf_jumbo_page: 4096, 12800, 0, 215, 5241, 0 mbuf_jumbo_9k: 9216, 6400, 1024, 3025, 26158415019, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 NetGraph items: 72, 4118, 0, 290, 850, 0 NetGraph data items: 72, 522, 0, 0, 0, 0 g_bio: 232, 0, 0, 800, 2219245, 0 ttyinq: 160, 0, 30, 114, 165, 0 ttyoutq: 256, 0, 16, 59, 88, 0 ata_request: 320, 0, 0, 36, 35, 0 ata_composite: 336, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0 VNODE: 472, 0, 951, 681, 35688, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 869, 550, 395855, 0 L VFS Cache: 328, 0, 36, 96, 49, 0 NAMEI: 1024, 0, 0, 256, 19024705, 0 NFSMOUNT: 616, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 25, 25, 0 pipe: 728, 0, 6, 339, 653310, 0 ksiginfo: 112, 0, 374, 682, 14673, 0 itimer: 344, 0, 0, 0, 0, 0 ng_pipe: 64, 0, 0, 0, 0, 0 bridge_rtnode: 64, 0, 0, 0, 0, 0 KNOTE: 128, 0, 6, 429, 371606, 0 socket: 680, 25602, 44, 292, 193187, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 11, 231, 118898, 0 udpcb: 16, 25704, 11, 1165, 118898, 0 tcp_inpcb: 336, 25608, 6, 280, 5548, 0 tcpcb: 880, 25600, 6, 154, 5548, 0 tcptw: 80, 5130, 0, 585, 5486, 0 syncache: 144, 15366, 0, 234, 5490, 0 hostcache: 136, 15372, 3, 165, 10, 0 tcpreass: 40, 1680, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0 sctp_ep: 1272, 25602, 0, 0, 0, 0 sctp_asoc: 2232, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 432, 18, 0 sctp_raddr: 616, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 25608, 2, 141, 298, 0 unpcb: 240, 25600, 23, 377, 67339, 0 rtentry: 200, 0, 59, 321, 1715, 0 pfsrctrpl: 152, 2369000, 0, 0, 0, 0 pfrulepl: 992, 0, 128, 352, 15937, 0 pfstatepl: 400, 2369007, 77, 1084, 390181, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 54, 450, 6825, 0 pfrktable: 1296, 1002, 9, 123, 2959, 0 pfrkentry: 216, 100008, 43, 227, 2692, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 606, 52, 0 pffrag: 80, 0, 0, 270, 26, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 4, 182, 359, 0 pfospfen: 112, 0, 696, 195, 122496, 0 pfosfp: 40, 0, 407, 517, 71632, 0 selfd: 56, 0, 601, 1163, 14558370, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 3, 12, 5, 0 FFS inode: 168, 0, 900, 552, 35636, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 900, 570, 35636, 0 md0: 512, 0, 301, 70, 301, 0 -
And I just tried setting the WAN MTU to 1500 and trying again, same results… What a mystery. So I reset it to 9000.
-
And another interesting factoid to consider…. Whenever I try to rsync that file outbound to the internet from inside the firewall, it seems the firewall state table shoots way up. I don't know why this would be since the rsync is a single socket connection, I can't imagine why it would generate 200+ states in the state table... Screenshot of the state graph attached. The activity on the far right of the graph was generated by me copying the file outbound.
-
Sorry, forgot to answer an earlier question… Yes, this is the amd64 version. And all the hosts and switches are dialed for jumbo frames. Tested and verified.
-
What is your provided speed (up and down) by your ISP?
http://cable-dsl.navasgroup.com/#Asymmetry
-
The outbound and inbound speed is 10Gb/s (full duplex synchronous) via CENIC in California.
-
Hmm, I don't think I have the NIC in promiscuous mode, at least I didn't do it on purpose. That's mostly used when doing packet tracing on the NIC right? I haven't been doing any of that.
Well, after consulting with a very knowledgeable friend of mine, it appears I do have the WAN interface in promiscuous mode, but it appears that it is that way because of pfsync/pflog because I'm using virtual floating IPs via CARP or something. Perhaps the pfSense devs know more on that. But I don't know if that is the root of the problem? Maybe… But I need CARP and virtual IPs so I hope it isn't the problem... ;)
-
Well, it is generally a bad idea to put that on WAN or LAN. It is better to have that on a dedicated NIC. As a test, disable pfsync. Promiscuous is not enabled by CARP VIPs only.
-
Hmm.. I do have a dedicated NIC for pfSync/CARP already? And it's not the WAN one. ;) I'm just using RFC1918 space for the dedicated CARP interface. I attached 2 screenshots showing my CARP config. It should be noted that I'm syncing everything available however.
But maybe I did something incorrectly there, let me know if so!
-
Is CARP a vLAN on on the WAN or LAN interface?
-
No, I'm not using VLANs in this model. The CARP interface uses a dedicated CAT5e cable connected in a crossover fashion from one physical interface of the "left" firewall to one physical interface on the "right" firewall, on a uniquely defined network that does not overlap any other nearby network.
-
Hmm.. I do have a dedicated NIC for pfSync/CARP already? And it's not the WAN one. ;) I'm just using RFC1918 space for the dedicated CARP interface. I attached 2 screenshots showing my CARP config. It should be noted that I'm syncing everything available however.
But maybe I did something incorrectly there, let me know if so!
CARP DOC says that it needs public ip to function
-
It needs 3 Internet IPs
1 for Physical Connection on Master
1 for Physical Connection on Backup
1 that is shared between the 2 (CARP Interface)It also needs 3 IPs per LAN interface for the same purposes.
It is highly recommended that you have dedicated NICs for pfsync and settings sync. This interface does not need internet route-able addresses. It is only to sync settings and and states.
-
My summary of interfaces are as follows:
firewall #1:
WAN - public IP 199.22.33.4/24
LAN - private IP 172.16.0.2/16
CARP - private IP 192.168.100.1/24 (connected directly to CARP interface on firewall #2, dedicated)firewall #2:
WAN - public IP 199.22.33.5/24
LAN - private IP 172.16.0.3/16
CARP - private IP 192.168.100.2/24 (connected directly to CARP interface on firewall #1, dedicated)Again, the CARP cable is a dedicated crossover cable at 1Gb/s ethernet. It is on a network that does not overlap with either the WAN or LAN networks. I am telling CARP/pfSync to use the dedicated CARP interface only.
One of the things CARP is doing is managing the virtual public IPs on the WAN interfaces. Such that if firewall #1 dies, firewall #2 would bring over the virtual IPs (on the WAN interface). Is that what is causing my WAN interfaces to be operating in promiscuous mode?
-
Another thing that is weird is that while I can see 2Gb/s on the live bandwidth graph, the RRD graphs don't show anything that high (maybe 20Mb/s or something). Is it possible the RRD graphs have upper limits and my traffic is above those limits, and therefore being ignored?
