Replace 2 port with 4 port nic card

ady2 · Oct 4, 2019, 1:55 AM

I bought a used 4 port intel nic card and was trying to replace the old 2 port card and have some question regarding the assignments of the new nic ports (igb0-igb3) instead of the old (em0-em1) to the existing interfaces. I have OpenVpn clients and vpn server and also some of my interfaces are using VLANs.
What I did:

I have backup my existing configuration and shutdown the firewall;
replaced the nic card and restarted the pfsense computer;
during the booting pfsense detects the new nic card and proposing to assign the VLANs and interfaces.

I have followed the VLAN assignment to nic ports (i.e. igb0 >> VLAN 100)
then I linked the interfaces with the ports (ex: LAN >> igb 1 ) and do I need to link the interface to the specific VLAN instead of port if the interface is using VLAN?
also there are vpn_client_interfaces and vpn_server_interfaces, here is not quite clear what port to link, I suppose that I need to link respective em0 WAN port (em0 in my case will be used by WAN interface as it will switch to em0 from em2 as I remove the old 2 nic card) to all clients and server?
After I have tried the above steps but probably I messed something in between as after assigning all VLANs and interfaces I reached again the start assignment screen that was saying to assign the VLANs and interfaces.

Will appreciate any help
Thanks

kiokoman · Oct 4, 2019, 7:48 AM

take your backup configuration, edit it with notepad++ for example
search and replace em0 with igb0
search and replace em1 with igb1
save it to a new copy
assuming there is no problem with your pfsense installation just configure WAN and LAN without any other particular settings. go to the webgui and
restore the configuration of pfsense with the new file. reboot

stephenw10 · Oct 4, 2019, 2:09 PM

Yeah, almost certainly easier to manually edit the config in this case.

I would not use find-and-replace directly though as there's a good chance you could have some em0s and em1s in your certificates and keys and you obviously don't want that replacing.

Also watch out for the port order not being what you expect. igb0 might be the opposite end of the card for example. Run ifconfg at the console with one port connected to check which is which if you have any doubt.

Steve

ady2 · Oct 4, 2019, 8:21 PM

@stephenw10
This worked - Thanks a lot.

If somebody will have same issue, here is how I did: I have edited the config file in sublime text editor and replaced the necessary ports (e.g. em0 with igb1) for interfaces and VLANs. After I have restore to the edited config and I get a message on the top that there are some discrepancies for my interfaces (that was expected as my config was for new nic card and I have still the old one ). I have shut down the pfsense computer, replaced the old nic with the new one and restarted and everything was working fine from the first time (no other settings were necessary except just to find the right ports that corresponds to the new nic card.

Thanks

Sergei_Shablovsky · Mar 27, 2022, 9:26 PM

@stephenw10 said in Replace 2 port with 4 port nic card:

Yeah, almost certainly easier to manually edit the config in this case.

I would not use find-and-replace directly though as there's a good chance you could have some em0s and em1s in your certificates and keys and you obviously don't want that replacing.

Also watch out for the port order not being what you expect. igb0 might be the opposite end of the card for example. Run ifconfg at the console with one port connected to check which is which if you have any doubt.

Steve

Dear Steve!

Could You be so please to extend/modify this suggestion in case when:

WE HAVE:

the pfSense server have 6 slots and all slots filled by identical 1Gb 2-head NICs (all NICs are Intel, so we have igb0-igb11 interfaces in a system);
there are a lot of settings (WireGuard, OpenVPN, multi-WAN balancing, etc, etc...) already made;

WE NEED:

replace two NICs on new version (the same slots, the same 2-head NICs, the same Intel vendor), 1Gb/s -> 10Gb/s;

How not to loose something in settings? (Because from FreeBSD / pfSense point of view some device disappear, new device appear, MAC changed, Device ID changed, etc....)

Thank You so much for attention and detailed answering! ;)

Gertjan · Mar 28, 2022, 8:48 AM

@sergei_shablovsky

Added to what was said above :

Open your config.xml file with an editor like Notepad++.

Goto the <interfaces> section.
You will find sub sections like

<interfaces>
		<wan>
			<if>em0</if>
			<disableftpproxy></disableftpproxy>
			<bandwidth>100</bandwidth>
			<bandwidthtype>Mb</bandwidthtype>
			<descr><![CDATA[WAN]]></descr>
			<alias-address></alias-address>
			<alias-subnet>32</alias-subnet>
			<spoofmac></spoofmac>
			<enable></enable>
			<ipaddr>dhcp</ipaddr>
			<dhcphostname></dhcphostname>
			<dhcprejectfrom></dhcprejectfrom>
......
		</wan>
		<lan>
			<if>em1</if>
			<bandwidth>100</bandwidth>
			<bandwidthtype>Mb</bandwidthtype>
			<enable></enable>
			<descr><![CDATA[LAN]]></descr>
			<spoofmac></spoofmac>
			<ipaddr>192.168.1.1</ipaddr>
			<subnet>24</subnet>
....
		</lan>
		<opt1>
			<if>em2</if>
			<descr><![CDATA[Portal]]></descr>
			<spoofmac></spoofmac>
			<enable></enable>
			<ipaddr>192.168.2.1</ipaddr>
			<subnet>24</subnet>
.....
		</opt1>
		<opt2>
			<descr><![CDATA[HeNetv6]]></descr>
			<if>gif0</if>
			<spoofmac></spoofmac>
			<mtu>1472</mtu>
			<enable></enable>
		</opt2>

If you want to replace em0 and em1 with a new card :
First, replace the old cards with the new cards. pfSense could ask you to assign the new interfaces, as I guess this will happen when the 'first' WAN and first 'LAN' can't be found == new cards are inserted with other driver names (like my emx).

When done, you know the driver name - my "em" and the order number, my 0 and 1.

Now, in the configure.xml, replace the interface you want replace the driver name and their number.

There always might be some trial and error, as it's unknown how freebsd upon boot numbers the devices. If your new cards have another driver name, like "igb", the other driver name order might change.
In my my case, em3 becomes em0 em4 becomes em1 as the orginbal em0 and em1 are now vanished (replaced).

Sergei_Shablovsky · Mar 28, 2022, 11:11 AM

@gertjan

How all of this changes impact on safety of other settings (vpn, bridges, etc...)

stephenw10 · Mar 28, 2022, 4:29 PM

It shouldn't affect VPNs or bridges, those will just get created on the new interfaces.

It affects sub-interface types that use the NICs directly, so VLANs, LAGGs, QinQ etc If you have those you would have to edit those sections in the config also.

Steve

Sergei_Shablovsky · Mar 28, 2022, 4:29 PM

Thank You for answering, Steve!

@stephenw10 said in Replace 2 port with 4 port nic card:

It shouldn't affect VPNs or bridges, those will just get created on the new interfaces.

It affects sub-interface types that use the NICs directly, so VLANs, LAGGs, QinQ etc If you have those you would have to edit those sections in the config also.

May be better to transform my question to next one:
which parts of pfSense CE operate with interface name (I mean “igb0”, etc...)
and
which part of pfSense CE operate with NICs “close-to-hardware level” (like MAC address, FreeBSD device ID, etc...)

Thank You so much for detailed answer if possible!

Sergei_Shablovsky · Mar 28, 2022, 4:39 PM

@gertjan said in Replace 2 port with 4 port nic card:

There always might be some trial and error, as it's unknown how freebsd upon boot numbers the devices. If your new cards have another driver name, like "igb", the other driver name order might change.
In my my case, em3 becomes em0 em4 becomes em1 as the orginbal em0 and em1 are now vanished (replaced).

Thank You for detailed answer!

Could You be so please to explain:
On which principles FreeBSD (because pfSense are on top of FreeBSD) giving the NICs ports identity?

From my knowledge, there are involved (from HIGHT IMPACT to LOW):
motherboard schematics,
BIOS subsystem,
NICs mechanical interface (PCI, PCI-e, etc...),
NICs order in motherboard slots,
FreeBSD drivers,
...
?

Thank You for help and opinions!

stephenw10 · Mar 28, 2022, 4:41 PM

The only things that use the hardware device name directly are things that can be (or are required to be) added to an unassigned parent interface.
So VLANs, QinQ, LAGG, PPPoE.

Everything else is abstracted to the assigned interface names to make re-assigning much easier.

Steve

Sergei_Shablovsky · Mar 29, 2022, 3:11 AM

@stephenw10 said in Replace 2 port with 4 port nic card:

The only things that use the hardware device name directly are things that can be (or are required to be) added to an unassigned parent interface.
So VLANs, QinQ, LAGG, PPPoE.

Everything else is abstracted to the assigned interface names to make re-assigning much easier.

Thank You, Steve!

I need to add useful utility (from a FreeBSD userforum answer:

=====================================
Two options come to mind.

Manually set interface names in /boot/device.hints
Code:

hint.igb.0.at="pci0:6:0:3"
hint.igb.1.at="pci0:6:0:2"
hint.igb.2.at="pci0:6:0:1"
hint.igb.3.at="pci0:6:0:0"
hint.igb.4.at="pci0:9:0:0"

Drawback here is any PCIe card install/removal could change pci bus address of your interfaces and need fixup.

Then Eric A. Borisch's port.
sysutils/ethname

[NEW PORT] sysutils/ethname: boot-time (re)naming of ethernet devices by MAC

An rc-script for pinning an ethernet network name to a MAC address.

--

*This isn't typically needed on PCIe systems, but for systems with multiple
USB ethernet (ue) devices, which seem to like to come up in
non-deterministic order, it is very helpful; doubly so when the system in
question is a router / firewall where the network config and security
concerns vary wildly from one device to the next.

It could also be of use for traditional NICs (PCIe) when adding a new card
to a system, for example, and ensuring that the existing, previously
configured device sticks to the MAC address, and not having to worry about
which ends up /dev/xxxN vs /dev/xxxM.

The script inserts itself before netif, waits an adjustable delay for the
expected devices to appear, and then renames them as requested by the user.
All of the device configuration, pf, etc., can be written with the new
names. It does not attempt to automatically handle devices added after boot.*

--

Usage is described in the script, but effectively:

$ cat /etc/rc.conf
ethname_enable="YES"
ethname_foo0_mac="aa:bb:cc:dd:ee:00"
ethname_bar_mac="aa:bb:cc:dd:ee:01"

# Optionally, otherwise it'll just enumerate all ethname_*_mac vars:
ethname_names="foo0 bar"

stephenw10 · Mar 29, 2022, 1:27 PM

Yes, you can use PCI device wiring to set the NICs by their bus location as shown there.

But if you do that and have to re-install ever those values will be lost the interfaces re-ordered. It's better to just configure them in the enumerated order IMO.

Steve

Sergei_Shablovsky · Mar 29, 2022, 10:43 PM

@stephenw10 said in Replace 2 port with 4 port nic card:

Yes, you can use PCI device wiring to set the NICs by their bus location as shown there.

But if you do that and have to re-install ever those values will be lost the interfaces re-ordered. It's better to just configure them in the enumerated order IMO.

Steve
Thank You for answering, Steve!

So, If I understand correctly, NetGate have no any step-by-step user guide for most frequent situation:

replace one NIC to another in case increasing port speed (for example from 100Mb/s -> 1Gb/s, with a same number of ports);
replace one NIC to 2(two) NICs in case increasing port speed & numbers of ports (for example 4 x 100Mb/s -> 2 x 1Gb/s + 4 x 1Gb/s);

with minimal configuration editing and minimal risks to losing settings ?

If so, this is VERY strange situation from user care point of view, because of MILLIONS of users of pfSense CE worldwide have the identical difficulties in case when they need upgrade NICs on their installations...

stephenw10 · Mar 30, 2022, 7:35 AM

For most users in that situation:

Shutdown
Replace the NIC(s)
Boot back up
Re-assign the interfaces

Done.

It's far more common to replace the hardware entirely but the same applies really if the interfaces are different.

Steve

Gertjan · Mar 30, 2022, 7:35 AM

I did exactly that half a year ago :
I replace my 3 single 'realtek' 100 Mbits NICs for one quad NIC 1 Gbit Intel NIC.

I rebooted after the replacement ones, to see what happens, and assigned my new drivers em0 to em3. replacing the old re0, re1 and re3 with a new em0 em1 and a em2 - the em3 being a non assigned spare.
With this info, the current config file, I edited my previous backup up config, and imported it.
Done.

Btw using the CE version.

Sergei_Shablovsky · Apr 11, 2022, 1:51 AM

@stephenw10
Thank You! :)

Sergei_Shablovsky · Apr 11, 2022, 1:51 AM

@gertjan
Thank You!

Sergei_Shablovsky · Apr 27, 2022, 7:37 AM

@gertjan said in Replace 2 port with 4 port nic card:

I did exactly that half a year ago :
I replace my 3 single 'realtek' 100 Mbits NICs for one quad NIC 1 Gbit Intel NIC.

I rebooted after the replacement ones, to see what happens, and assigned my new drivers em0 to em3. replacing the old re0, re1 and re3 with a new em0 em1 and a em2 - the em3 being a non assigned spare.
With this info, the current config file, I edited my previous backup up config, and imported it.
Done.

Btw using the CE version.

So, because I have several NICs installed on one motherboard, and need to replace only 2(TWO) of them,
At the first step I change one NIC, then correct Interface assigning, then reboot. All working properly on that stage.

At the second stage I replace another NIC and... pfSense stuck on booting on Starting DNS Resolver...

Rolling back on a previous NIC not changing situation: just stuck on the same Starting DNS Resolver....

Please help :)

Sergei_Shablovsky · Apr 26, 2022, 10:21 AM

@stephenw10 said in Replace 2 port with 4 port nic card:

For most users in that situation:

Shutdown

Replace the NIC(s)

Boot back up

Re-assign the interfaces

Done.

It's far more common to replace the hardware entirely but the same applies really if the interfaces are different.

Steve

System stuck on a “Starting DNS Resolver...”

Replace 2 port with 4 port nic card

Then Eric A. Borisch's port. sysutils/ethname

Then Eric A. Borisch's port.
sysutils/ethname