TIP: If you have an IPMI motherboard and constantly pull an internal IP on WAN
-
I made this mistake the hard way when first implementing my pfSense box. Using a Supermicro motherboard with the new IPMI (management interface over IP basically) feature. I wasn't using IPMI at all yet and didn't have a cable plugged into the port. For the life of me I couldn't figure out why WAN was pulling an internal IP from my Motorola cable modem instead of a public IP.
It turns out, IPMI is sneaky and WANTS to be found, even if you ignore the dedicated IPMI port. It'll tag along on the regular Intel NIC ports. Most consumer cable Internet connections only allow one public IP from your ISP's DHCP server, which is bridged through the cable modem to your edge router's WAN interface. It turns out, the virtual IPMI interface was receiving the PUBLIC IP first, leaving my cable modem no choice but to give my WAN interface an internal IP from the modem's DHCP service.
In other words, my IPMI and major BIOS settings got a public IP and was wide open to the world, unprotected. Heck, I bet even Telnet and TFTP and who knows what other stuff was open (with default passwords). Meaning anybody who accessed http/80 or https/443 on my public IP would get the login page to my pfSense box's motherboard. Holy crap!
SOLUTION: Don't think that just because you don't have a physical cable plugged into the IPMI port that it won't try to piggyback onto another physical port, such as WAN. What I did was go into the BIOS and statically assign IPMI all 0.0.0.0 addresses for now.
I hope this made some sense to you guys who use Supermicro or any other newer motherboard with IPMI. (Which I think wasn't really built with security in mind and scares me).
-
For the less overkill solution, log on to the IPMI web interface, go to the network settings and disable "network bonding" (which as OP experienced bonds the IPMI NIC to one of the LOM ports by default for some motherboards).
-
This post is deleted! -
For the less overkill solution, log on to the IPMI web interface, go to the network settings and disable "network bonding" (which as OP experienced bonds the IPMI NIC to one of the LOM ports by default for some motherboards).
Thanks. At this stage I had not gone into the IPMI interface on the mobo and just wanted to get pfSense running. Thanks for letting me know about a "network bonding" setting!
-
@SunCatalyst:
we at work test hundreds of servers a year (in the lab) … with dedicated IPMI ports.
NEVER have seen this behavior.
sounds like a BIOS bug or a configuration error.
never seen a dedicated ethernet port IPMI card decide cause it doesnt have IP to overlap itself on top of the motherboard
ethernet ports.at work we run 100's of Supermicro servers all with IPMI cards. and i run them personally as well.
Just because you haven't seen (or configured) it doesn't mean it doesn't exist. I've seen lots of IPMI implementations that can be explicitly configured to share one of the on-board ports if using a dedicated port is inconvenient (random examples of boards I have on hand: Tyan K8HM, Supermicro X8SIL, ASRock E3C226D2I); newer implementations (just checked on both the X8SIL and the E3C226D2I) will also let you configure automatic fail-over from the dedicated management port to the LOM ports (which BTW is tied to having a link, not having an IP). Now, up until recently, I had not seen this as the default behavior, but that's what it is at least on the E3C226D2I, and apparently on the original poster's board as well.
-
Thanks. At this stage I had not gone into the IPMI interface on the mobo and just wanted to get pfSense running. Thanks for letting me know about a "network bonding" setting!
Note that the setting you're looking for may have different names depending on your BMC. As I mentioned in my previous posts, on some of my boards the corresponding setting is called "LAN interface" (and should be set to "dedicated," not "shared" or "failover," to avoid what you're seeing), and obviously the terminology on yours may be different yet. The general idea is that most IPMI implementations allow you to configure which network port(s) to use, and you'll want to force it to use the dedicated management port only.
-
This post is deleted! -
@SunCatalyst:
i work in one of the top 100 corporations in america. we test more servers in a year (and crush 99% of them) than most people will ever see in there complete lifetime. all of the stuff we test is all Xeon based servers from many manufacturers.
apparently you didnt read "bios bug OR a configuration error". could be EITHER or BOTH. if you dont know what your doing with IPMI and dont
RTFM , then its your own fault.'nuff said.
Good for you, but none of that changes the fact that the ability to failover to the LOM ports is a feature that is commonly (and very intentionally) offered by current IPMI implementations, not some kind of "bios bug" nor something that only happens accidentally when you mess up your configuration. Now, obviously the OP wasn't aware that this particular feature was enabled by default on his board, and RTFM would have helped; however, it's just silly to claim that no IPMI implementation would ever behave this way in normal operation.
-
this took a year to bite me! I've been running pfsense on a C2558F for the last year with the igb0 interface attached to my cable modem. The other day I lost internet and couldn't figure out why only this interface couldn't get a real internet IP assigned. ipmi failover hit me today while listening to TechSNAP.
What a random turn of events, lol
-
But why on earth would anybody leave the IPMI port configured as DHCP?
If pfSense is the DHCP server, but it's not booted up yet, one might end up not being able to connect to IPMI either, since it's got no valid internal IP. In most cases just setting a normal internal IP address to IPMI, and connecting the port to a switch will save from situations like this… -
I leave all my IPMI ports configured to DHCP with no problems, yet. IPMI network config is very sticky, meaning, it will hang unto its DHCP settings until you completely disconnect all power to the motherboard. AFAIK, even the DHCP lease time expiring doesn't force IPMI to query the DHCP server.
I can have pfsense shutdown and still access the IPMI interface. However, I have yet to try disconnecting the power cord when my pfSense box is shutdown to see what happens when IPMI issue a DHCP request when power is restored.
-
And the "advantage" of this setup is exactly what? Prey that it doesn't break in the worst possible moment?
-
Yep, It solved the issue that pfSense won't get WAN IP address from my bridged cable modem (WAN was assigned to igb0) after a shut down. This problem was puzzled me for several days until I saw this post. and now it works. Thanks.
-
But why on earth would anybody leave the IPMI port configured as DHCP?
If pfSense is the DHCP server, but it's not booted up yet, one might end up not being able to connect to IPMI either, since it's got no valid internal IP. In most cases just setting a normal internal IP address to IPMI, and connecting the port to a switch will save from situations like this…This is exactly what I did when I put my system together. I set the bmc ip statically to get 192.168.1.2 and I have had no problems whatsoever.
-
But why on earth would anybody leave the IPMI port configured as DHCP?
I consider, any IPMI interface must have his own static IP address.
We have all connected to the management VLAN to get a dedicated contact. -
We have dozens of SuperMicro servers and they all do this by default, you must switch to only use the dedicated IPMI interface.
If you are using the nic in a virtualization environment, it will be wide open as well.
-
We have dozens of SuperMicro servers and they all do this by default, you must switch to only use the dedicated IPMI interface.
Yep, they do. On the ones we sell, as part of the installation process that "feature" gets disabled so it only uses the dedicated IPMI port. It wants to use what most people assign as the WAN port as its fallback, which is potentially a very serious security issue if we didn't configure it more sensibly.
-
Honestly, there are so many bugs/security issues with Supermicro's IPMI interfaces that I wanted to just disable them all anyway.
-
Honestly, there are so many bugs/security issues with Supermicro's IPMI interfaces that I wanted to just disable them all anyway.
In our company we had not only one times a problem with this boards or their IPMI LAN Port
and friends of mine connect them to Aten KVM Switches up to >100 devices with any kind of problem!For sure I will consider that this ports must be configured as well as all new boards are arriving.
-
@BlueKobold:
In our company we had not only one times a problem with this boards or their IPMI LAN Port
Maybe you just don't know about the problem.
https://www.thomas-krenn.com/en/wiki/Supermicro_IPMI_Security_Updates_July_2014
https://www.thomas-krenn.com/en/wiki/Supermicro_IPMI_Security_Updates_November_2013
