2.0-RELEASE: Performance oddity?
-
I ran the vmstat commands while transferring, here they are:
[2.0-RELEASE][admin@fire01.xxx]/root(1): vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 108, 11, 108, 0 UMA Zones: 448, 0, 108, 4, 108, 0 UMA Slabs: 568, 0, 1634, 4, 10719, 0 UMA RCntSlabs: 568, 0, 14027, 1, 14027, 0 UMA Hash: 256, 0, 1, 14, 3, 0 16 Bucket: 152, 0, 206, 19, 206, 0 32 Bucket: 280, 0, 316, 6, 316, 0 64 Bucket: 536, 0, 338, 5, 338, 79 128 Bucket: 1048, 0, 323, 1, 323, 877 VM OBJECT: 216, 0, 2170, 1070, 21533629, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 786253, 64, 494, 25805, 0 MAP ENTRY: 120, 0, 1628, 1100, 45538101, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 313, 14, 313, 0 16: 16, 0, 2252, 1444, 2984997, 0 32: 32, 0, 3286, 1360, 7405041, 0 64: 64, 0, 3287, 1585, 57246324442, 0 128: 128, 0, 4456, 1518, 3529376, 0 256: 256, 0, 715, 1355, 9017640, 0 512: 512, 0, 856, 880, 2098526, 0 1024: 1024, 0, 57, 219, 245486, 0 2048: 2048, 0, 104, 148, 26522, 0 4096: 4096, 0, 412, 832, 1056263, 0 Files: 80, 0, 161, 784, 7208760, 0 TURNSTILE: 136, 0, 517, 103, 517, 0 umtx pi: 96, 0, 0, 0, 0, 0 PROC: 1120, 0, 58, 329, 985343, 0 THREAD: 984, 0, 461, 55, 461, 0 SLEEPQUEUE: 80, 0, 517, 121, 517, 0 VMSPACE: 392, 0, 35, 465, 985065, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 512, 0, 16320, 2112, 2086392, 0 mbuf: 512, 0, 2051, 2813, 59098587505, 0 mbuf_cluster: 2048, 25600, 18434, 1092, 1847171876, 0 mbuf_jumbo_page: 4096, 12800, 0, 215, 5241, 0 mbuf_jumbo_9k: 9216, 6400, 1024, 3025, 26158196178, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 NetGraph items: 72, 4118, 0, 290, 826, 0 NetGraph data items: 72, 522, 0, 0, 0, 0 g_bio: 232, 0, 0, 800, 2215015, 0 ttyinq: 160, 0, 30, 114, 165, 0 ttyoutq: 256, 0, 16, 59, 88, 0 ata_request: 320, 0, 0, 36, 35, 0 ata_composite: 336, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0 VNODE: 472, 0, 951, 681, 35521, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 870, 549, 394338, 0 L VFS Cache: 328, 0, 36, 96, 49, 0 NAMEI: 1024, 0, 0, 256, 18976211, 0 NFSMOUNT: 616, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 25, 25, 0 pipe: 728, 0, 5, 340, 652566, 0 ksiginfo: 112, 0, 374, 682, 14651, 0 itimer: 344, 0, 0, 0, 0, 0 ng_pipe: 64, 0, 0, 0, 0, 0 bridge_rtnode: 64, 0, 0, 0, 0, 0 KNOTE: 128, 0, 11, 424, 371227, 0 socket: 680, 25602, 53, 283, 191779, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 11, 231, 117670, 0 udpcb: 16, 25704, 11, 1165, 117670, 0 tcp_inpcb: 336, 25608, 15, 271, 5548, 0 tcpcb: 880, 25600, 11, 149, 5548, 0 tcptw: 80, 5130, 4, 581, 5481, 0 syncache: 144, 15366, 0, 234, 5490, 0 hostcache: 136, 15372, 3, 165, 10, 0 tcpreass: 40, 1680, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0 sctp_ep: 1272, 25602, 0, 0, 0, 0 sctp_asoc: 2232, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 432, 18, 0 sctp_raddr: 616, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 25608, 2, 141, 288, 0 unpcb: 240, 25600, 23, 377, 67205, 0 rtentry: 200, 0, 59, 321, 1679, 0 pfsrctrpl: 152, 2369000, 0, 0, 0, 0 pfrulepl: 992, 0, 125, 355, 15293, 0 pfstatepl: 400, 2369007, 122, 1039, 389896, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 55, 449, 6552, 0 pfrktable: 1296, 1002, 9, 123, 2896, 0 pfrkentry: 216, 100008, 43, 227, 2587, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 606, 52, 0 pffrag: 80, 0, 0, 270, 26, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 3, 183, 338, 0 pfospfen: 112, 0, 696, 195, 117624, 0 pfosfp: 40, 0, 407, 517, 68783, 0 selfd: 56, 0, 602, 1162, 14536063, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 3, 12, 5, 0 FFS inode: 168, 0, 900, 552, 35469, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 900, 570, 35469, 0 md0: 512, 0, 298, 45, 298, 0 [2.0-RELEASE][admin@fire01.xxx]/root(3): vmstat 2 10 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mf0 md0 in sy cs us sy id 0 0 0 591M 23G 389 0 0 0 394 0 0 0 2603 457 11606 0 7 93 0 0 0 591M 23G 2 0 0 3 1 0 20 0 45 294 2251 0 0 100 0 0 0 591M 23G 5580 0 0 0 5328 0 15 0 98 5260 2567 1 1 98 0 0 0 591M 23G 199 0 0 0 178 0 0 0 104 443 2441 0 0 100 0 0 0 685M 23G 3760 0 0 0 1056 0 12 1 40 3235 2328 1 0 99 0 0 0 591M 23G 5890 0 0 0 8161 0 15 2 55 5844 2459 1 1 97 0 0 0 591M 23G 0 0 0 0 0 0 0 0 696 272 3826 0 0 100 0 0 0 591M 23G 7184 0 0 0 6670 0 13 43 2628 7939 9498 1 1 98 0 0 0 591M 23G 4265 0 0 0 4064 0 12 2 2236 3930 8134 1 1 99 0 0 0 591M 23G 0 0 0 0 0 0 0 0 35 133 2231 0 0 100 [2.0-RELEASE][admin@fire01.xxx]/root(4): vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 108, 11, 108, 0 UMA Zones: 448, 0, 108, 4, 108, 0 UMA Slabs: 568, 0, 1634, 4, 10720, 0 UMA RCntSlabs: 568, 0, 14027, 1, 14027, 0 UMA Hash: 256, 0, 1, 14, 3, 0 16 Bucket: 152, 0, 206, 19, 206, 0 32 Bucket: 280, 0, 317, 5, 317, 0 64 Bucket: 536, 0, 338, 5, 338, 79 128 Bucket: 1048, 0, 323, 1, 323, 877 VM OBJECT: 216, 0, 2190, 1050, 21560351, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 786253, 64, 494, 25807, 0 MAP ENTRY: 120, 0, 1660, 1130, 45601264, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 313, 14, 313, 0 16: 16, 0, 2252, 1444, 2989568, 0 32: 32, 0, 3277, 1369, 7414238, 0 64: 64, 0, 3288, 1584, 57246971299, 0 128: 128, 0, 4456, 1518, 3580711, 0 256: 256, 0, 699, 1371, 9043599, 0 512: 512, 0, 857, 879, 2101141, 0 1024: 1024, 0, 57, 219, 246302, 0 2048: 2048, 0, 104, 148, 27604, 0 4096: 4096, 0, 413, 831, 1058172, 0 Files: 80, 0, 159, 786, 7223293, 0 TURNSTILE: 136, 0, 517, 103, 517, 0 umtx pi: 96, 0, 0, 0, 0, 0 PROC: 1120, 0, 59, 328, 986350, 0 THREAD: 984, 0, 461, 55, 461, 0 SLEEPQUEUE: 80, 0, 517, 121, 517, 0 VMSPACE: 392, 0, 36, 464, 986069, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 512, 0, 16320, 2112, 2087752, 0 mbuf: 512, 0, 2051, 2813, 59099224460, 0 mbuf_cluster: 2048, 25600, 18434, 1092, 1847178111, 0 mbuf_jumbo_page: 4096, 12800, 0, 215, 5241, 0 mbuf_jumbo_9k: 9216, 6400, 1024, 3025, 26158415019, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 NetGraph items: 72, 4118, 0, 290, 850, 0 NetGraph data items: 72, 522, 0, 0, 0, 0 g_bio: 232, 0, 0, 800, 2219245, 0 ttyinq: 160, 0, 30, 114, 165, 0 ttyoutq: 256, 0, 16, 59, 88, 0 ata_request: 320, 0, 0, 36, 35, 0 ata_composite: 336, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0 VNODE: 472, 0, 951, 681, 35688, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 869, 550, 395855, 0 L VFS Cache: 328, 0, 36, 96, 49, 0 NAMEI: 1024, 0, 0, 256, 19024705, 0 NFSMOUNT: 616, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 23, 25, 25, 0 pipe: 728, 0, 6, 339, 653310, 0 ksiginfo: 112, 0, 374, 682, 14673, 0 itimer: 344, 0, 0, 0, 0, 0 ng_pipe: 64, 0, 0, 0, 0, 0 bridge_rtnode: 64, 0, 0, 0, 0, 0 KNOTE: 128, 0, 6, 429, 371606, 0 socket: 680, 25602, 44, 292, 193187, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 11, 231, 118898, 0 udpcb: 16, 25704, 11, 1165, 118898, 0 tcp_inpcb: 336, 25608, 6, 280, 5548, 0 tcpcb: 880, 25600, 6, 154, 5548, 0 tcptw: 80, 5130, 0, 585, 5486, 0 syncache: 144, 15366, 0, 234, 5490, 0 hostcache: 136, 15372, 3, 165, 10, 0 tcpreass: 40, 1680, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0 sctp_ep: 1272, 25602, 0, 0, 0, 0 sctp_asoc: 2232, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 432, 18, 0 sctp_raddr: 616, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 25608, 2, 141, 298, 0 unpcb: 240, 25600, 23, 377, 67339, 0 rtentry: 200, 0, 59, 321, 1715, 0 pfsrctrpl: 152, 2369000, 0, 0, 0, 0 pfrulepl: 992, 0, 128, 352, 15937, 0 pfstatepl: 400, 2369007, 77, 1084, 390181, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 54, 450, 6825, 0 pfrktable: 1296, 1002, 9, 123, 2959, 0 pfrkentry: 216, 100008, 43, 227, 2692, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 606, 52, 0 pffrag: 80, 0, 0, 270, 26, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 4, 182, 359, 0 pfospfen: 112, 0, 696, 195, 122496, 0 pfosfp: 40, 0, 407, 517, 71632, 0 selfd: 56, 0, 601, 1163, 14558370, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 3, 12, 5, 0 FFS inode: 168, 0, 900, 552, 35636, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 900, 570, 35636, 0 md0: 512, 0, 301, 70, 301, 0 -
And I just tried setting the WAN MTU to 1500 and trying again, same results… What a mystery. So I reset it to 9000.
-
And another interesting factoid to consider…. Whenever I try to rsync that file outbound to the internet from inside the firewall, it seems the firewall state table shoots way up. I don't know why this would be since the rsync is a single socket connection, I can't imagine why it would generate 200+ states in the state table... Screenshot of the state graph attached. The activity on the far right of the graph was generated by me copying the file outbound.
-
Sorry, forgot to answer an earlier question… Yes, this is the amd64 version. And all the hosts and switches are dialed for jumbo frames. Tested and verified.
-
What is your provided speed (up and down) by your ISP?
http://cable-dsl.navasgroup.com/#Asymmetry
-
The outbound and inbound speed is 10Gb/s (full duplex synchronous) via CENIC in California.
-
Hmm, I don't think I have the NIC in promiscuous mode, at least I didn't do it on purpose. That's mostly used when doing packet tracing on the NIC right? I haven't been doing any of that.
Well, after consulting with a very knowledgeable friend of mine, it appears I do have the WAN interface in promiscuous mode, but it appears that it is that way because of pfsync/pflog because I'm using virtual floating IPs via CARP or something. Perhaps the pfSense devs know more on that. But I don't know if that is the root of the problem? Maybe… But I need CARP and virtual IPs so I hope it isn't the problem... ;)
-
Well, it is generally a bad idea to put that on WAN or LAN. It is better to have that on a dedicated NIC. As a test, disable pfsync. Promiscuous is not enabled by CARP VIPs only.
-
Hmm.. I do have a dedicated NIC for pfSync/CARP already? And it's not the WAN one. ;) I'm just using RFC1918 space for the dedicated CARP interface. I attached 2 screenshots showing my CARP config. It should be noted that I'm syncing everything available however.
But maybe I did something incorrectly there, let me know if so!
-
Is CARP a vLAN on on the WAN or LAN interface?
-
No, I'm not using VLANs in this model. The CARP interface uses a dedicated CAT5e cable connected in a crossover fashion from one physical interface of the "left" firewall to one physical interface on the "right" firewall, on a uniquely defined network that does not overlap any other nearby network.
-
Hmm.. I do have a dedicated NIC for pfSync/CARP already? And it's not the WAN one. ;) I'm just using RFC1918 space for the dedicated CARP interface. I attached 2 screenshots showing my CARP config. It should be noted that I'm syncing everything available however.
But maybe I did something incorrectly there, let me know if so!
CARP DOC says that it needs public ip to function
-
It needs 3 Internet IPs
1 for Physical Connection on Master
1 for Physical Connection on Backup
1 that is shared between the 2 (CARP Interface)It also needs 3 IPs per LAN interface for the same purposes.
It is highly recommended that you have dedicated NICs for pfsync and settings sync. This interface does not need internet route-able addresses. It is only to sync settings and and states.
-
My summary of interfaces are as follows:
firewall #1:
WAN - public IP 199.22.33.4/24
LAN - private IP 172.16.0.2/16
CARP - private IP 192.168.100.1/24 (connected directly to CARP interface on firewall #2, dedicated)firewall #2:
WAN - public IP 199.22.33.5/24
LAN - private IP 172.16.0.3/16
CARP - private IP 192.168.100.2/24 (connected directly to CARP interface on firewall #1, dedicated)Again, the CARP cable is a dedicated crossover cable at 1Gb/s ethernet. It is on a network that does not overlap with either the WAN or LAN networks. I am telling CARP/pfSync to use the dedicated CARP interface only.
One of the things CARP is doing is managing the virtual public IPs on the WAN interfaces. Such that if firewall #1 dies, firewall #2 would bring over the virtual IPs (on the WAN interface). Is that what is causing my WAN interfaces to be operating in promiscuous mode?
-
Another thing that is weird is that while I can see 2Gb/s on the live bandwidth graph, the RRD graphs don't show anything that high (maybe 20Mb/s or something). Is it possible the RRD graphs have upper limits and my traffic is above those limits, and therefore being ignored?
