Infiniband cards with pfSense 2.2
-
… Thread Resurrection...
I'm building a home lab and looking at QDR infiniband within my rack and ethernet coming in from the outside (wan/lan)
I have the occassional need for very high bandwidth between my storage and my compute nodes.
Am I correct in understanding that pfSense can be compiled with IP over IB and can include a subnet manager?
Can pfSense concurrently route RDMA and IP over IB? Does this question even make sense?Would I be better off with a point to point connection between my storage node(s)/san and my compute nodes and/or adding a 4036 (4036E has a 10gb ethernet and subnet manager onboard but I'm poor) switch and effectively running both gigabit ethernet for ip traffic and QDR infiniband for NFS over RDMA?
Thanks,
Robert. -
Hello,
in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
attached directly from card to card. And this extra or separate network will be then connected to
the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
a separate router that is doing anything in this "game". -
@BlueKobold:
Hello,
in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
attached directly from card to card. And this extra or separate network will be then connected to
the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
a separate router that is doing anything in this "game".I am building a small compute network (single rack)
Kvm/Centos / dual xeon / 4 xeon phi - compute nodes
Kvm/Solaris / zfs mirrored / striped ssds - data server (San - napp-it)Remote VPN
Development VMs
LAMP, misc servers smtp, ftp, etc…Qdr infiniband - cheap, high bandwidth, low latency
Ib/nfs/rdmaIf pfsense can support ipoib and bridge wan to lan
then I don't have to buy a bridging switch in the short term. I will likely vm pfsense as well.A Voltaire 4036 is relatively inexpensive.
This is being done as a home basement project, low budget, incremental build out.... Data mining, machine learning, parallel programming, networking lab/sandbox, kaggle contests
Matlab, pysci, R, OpenMP, openacc, c++, Fortran, etc.
a few remote developers, analysts, VPN / ssh.