Netgate Device ID changing on power loss
-
https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T
Using the onboard X550-AT2
I've attempted a bios update, still happens every hard power loss.
-
What does
ifconfig -vmashow at the CLI?Does it have a USB IPMI NIC maybe? Or a cellular modem?
-
Also check the boot logs for any errors from the NIC drivers.
-
@stephenw10 said in Netgate Device ID changing on power loss:
What does
ifconfig -vmashow at the CLI?Does it have a USB IPMI NIC maybe? Or a cellular modem?
I've made the NDI change, checking the mac addresses of the interfaces before and after, they haven't changed.
Looking through the IPMI i don't see any mention of a NIC being added via it (nor does ifconfig -vma show it)
One thing I did notice just now is that network bonding is turned on in the IPMI. Maybe that has something to do with it? I'll disable it and test again.
I forgot I have an I226 in there as well for my backup WAN.
Here's the output of the ifconfig:
/root: ifconfig -vma igc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: LSWAN options=4e020bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 88:c9:b3:bf:75:60 inet [IP_REMOVED] netmask 0xffffffc0 broadcast 255.255.255.255 inet6 [IP_REMOVED]%igc0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (2500Base-T <full-duplex>) status: active supported media: media autoselect media 2500Base-T media 1000baseT media 1000baseT mediaopt full-duplex media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: igc0 ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether d0:50:99:de:c1:1b inet [IP_REMOVED] netmask 0xffffff00 broadcast [IP_REMOVED] inet6 [IP_REMOVED]%ix0 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect media 5000Base-T media 2500Base-T media 100baseTX media 1000baseT media 10Gbase-T nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ix0 ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: LAN options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether d0:50:99:de:c1:1c inet6 [IP_REMOVED]%ix1 prefixlen 64 scopeid 0x3 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect media 5000Base-T media 2500Base-T media 100baseTX media 1000baseT media 10Gbase-T nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ix1 enc0: flags=0 metric 0 mtu 1536 options=0 capabilities=0 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: enc0 lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0x0 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: lo0 pflog0: flags=100<PROMISC> metric 0 mtu 33152 options=0 capabilities=0 groups: pflog drivername: pflog0 pfsync0: flags=0 metric 0 mtu 1500 options=0 capabilities=0 maxupd: 128 defer: off version: 1400 syncok: 1 groups: pfsync drivername: pfsync0 ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=0 capabilities=0 ether 9a:95:62:30:fa:d5 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ue0 ix1.99: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: TransitVLAN options=4600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> capabilities=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> ether d0:50:99:de:c1:1c inet [IP_REMOVED] netmask 0xfffffff8 broadcast [IP_REMOVED] inet6 [IP_REMOVED]%ix1.99 prefixlen 64 scopeid 0xc groups: vlan vlan: 99 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: vlan0 ix1.25: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: DMZ options=4600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> capabilities=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> ether d0:50:99:de:c1:1c inet [IP_REMOVED] netmask 0xffffff00 broadcast [IP_REMOVED] inet6 [IP_REMOVED]%ix1.25 prefixlen 64 scopeid 0xd groups: vlan vlan: 25 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: vlan1 ovpns1: flags=1008043<UP,BROADCAST,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP_REMOVED] netmask 0xffffff00 broadcast [IP_REMOVED] inet6 [IP_REMOVED]%ovpns1 prefixlen 64 scopeid 0xe groups: tun openvpn nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Opened by PID 72330 drivername: tun1 tun_wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: VPN options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP_REMOVED] netmask 0xffffffff groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD> drivername: wg0 tun_wg1: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WSVPN options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP_REMOVED] netmask 0xffffffff groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD> drivername: wg1 tun_wg2: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: TUNNEL options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP_REMOVED] netmask 0xffffffff groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD> drivername: wg2 tailscale0: flags=1008003<UP,BROADCAST,MULTICAST,LOWER_UP> metric 0 mtu 1280 options=80000<LINKSTATE> capabilities=80000<LINKSTATE> groups: tun Tailscale nd6 options=101<PERFORMNUD,NO_DAD> Opened by PID 44535 drivername: tun0 -
Ok it's almost certainly this:
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=0 capabilities=0 ether 9a:95:62:30:fa:d5 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ue0I'd bet that MAC changes between boots. Check the boot log for errors about the MAC there.
Do you know what that is?
If you don't need it you can probably disable it with a loader hint.
-
@stephenw10 So from what I can gather, this Asrockrack board can share the ix0 port (which happens to be my wan) in addition to the dedicated IPMI port.
Posting the thread that I found this info on in case someone else has this issue in the future:
https://www.reddit.com/r/homelab/comments/ph9wgx/help_with_restricting_asrock_rack_x570d4u2l2t/
Running the following disables that sharing (which also happened to remove the ipmi from broadcasting on my WAN yikes)
ipmitool raw 0x32 0x71 0x00 0x01 0x00As for that ue0 interface:
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=0 capabilities=0 ether 42:29:ea:ff:8a:1a nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ue0I don't know what that interface is, so it must be this "virtual" ipmi interface thingy, but it 100% changed after the hard power reset.
Can you let me know what the loader hint would be to just entirely disable it?
After running that IPMI tool cmd, here's the latest
/root: ifconfig -vma igc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: LSWAN options=4e020bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 88:c9:b3:bf:75:60 inet [IP ADDRESS REMOVED] netmask 0xffffffc0 broadcast 255.255.255.255 inet6 fe80::8ac9:b3ff:febf:7560%igc0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (2500Base-T <full-duplex>) status: active supported media: media autoselect media 2500Base-T media 1000baseT media 1000baseT mediaopt full-duplex media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: igc0 ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether d0:50:99:de:c1:1b inet [IP ADDRESS REMOVED] netmask 0xffffff00 broadcast [IP ADDRESS REMOVED] inet6 fe80::d250:99ff:fede:c11b%ix0 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect media 5000Base-T media 2500Base-T media 100baseTX media 1000baseT media 10Gbase-T nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ix0 ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: LAN options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether d0:50:99:de:c1:1c inet6 fe80::d250:99ff:fede:c11c%ix1 prefixlen 64 scopeid 0x3 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect media 5000Base-T media 2500Base-T media 100baseTX media 1000baseT media 10Gbase-T nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ix1 enc0: flags=0 metric 0 mtu 1536 options=0 capabilities=0 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: enc0 lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet [IP ADDRESS REMOVED] netmask 0x0 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: lo0 pflog0: flags=100<PROMISC> metric 0 mtu 33152 options=0 capabilities=0 groups: pflog drivername: pflog0 pfsync0: flags=0 metric 0 mtu 1500 options=0 capabilities=0 maxupd: 128 defer: off version: 1400 syncok: 1 groups: pfsync drivername: pfsync0 ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=0 capabilities=0 ether 42:29:ea:ff:8a:1a nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: ue0 ix1.99: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: TransitVLAN options=4600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> capabilities=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> ether d0:50:99:de:c1:1c inet [IP ADDRESS REMOVED] netmask 0xfffffff8 broadcast [IP ADDRESS REMOVED] inet6 fe80::d250:99ff:fede:c11c%ix1.99 prefixlen 64 scopeid 0xc groups: vlan vlan: 99 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: vlan0 ix1.25: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: DMZ options=4600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> capabilities=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> ether d0:50:99:de:c1:1c inet [IP ADDRESS REMOVED] netmask 0xffffff00 broadcast [IP ADDRESS REMOVED] inet6 fe80::d250:99ff:fede:c11c%ix1.25 prefixlen 64 scopeid 0xd groups: vlan vlan: 25 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> drivername: vlan1 ovpns1: flags=1008043<UP,BROADCAST,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP ADDRESS REMOVED] netmask 0xffffff00 broadcast [IP ADDRESS REMOVED] inet6 fe80::8ac9:b3ff:febf:7560%ovpns1 prefixlen 64 scopeid 0xe groups: tun openvpn nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Opened by PID 67240 drivername: tun1 tun_wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: VPN options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP ADDRESS REMOVED] netmask 0xffffffff groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD> drivername: wg0 tun_wg1: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WSVPN options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP ADDRESS REMOVED] netmask 0xffffffff groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD> drivername: wg1 tun_wg2: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: TUNNEL options=80000<LINKSTATE> capabilities=80000<LINKSTATE> inet [IP ADDRESS REMOVED] netmask 0xffffffff groups: wg WireGuard nd6 options=101<PERFORMNUD,NO_DAD> drivername: wg2 tailscale0: flags=1008003<UP,BROADCAST,MULTICAST,LOWER_UP> metric 0 mtu 1280 options=80000<LINKSTATE> capabilities=80000<LINKSTATE> groups: tun Tailscale nd6 options=101<PERFORMNUD,NO_DAD> Opened by PID 7833 drivername: tun0 -
Run at the command line:
echo 'hint.ue.0.disabled="1"' >> /boot/loader.conf.localThat should prevent the device attaching.
You may be able to disable it in the BIOS which would probably be a better solution as it would survive a reinstall. And would not affect any USB devices added in the future.
-
@stephenw10 Thanks for that, unfortunately the NDI still changed again after a hard reset. (I double checked, its gone in the ifconfig command now)
Although that interface was probably part of the problem, it wasn't the only problem.
Do you have any other ideas? For now I've downgraded back to CE until I can get this solved.
-
@n1ck31 said in Netgate Device ID changing on power loss:
@stephenw10 Thanks for that, unfortunately the NDI still changed again after a hard reset. (I double checked, its gone in the ifconfig command now)
Although that interface was probably part of the problem, it wasn't the only problem.
Do you have any other ideas? For now I've downgraded back to CE until I can get this solved.
Never mind, after another reboot, it's back. (ue0)
I've checked BIOS, nowhere to disable this that I can see.
-
Are you bridging interfaces? If not, never mind.
-
Hmm, it didn't disable it?
Do you see the line as expected in /boot/loader.conf.local?
Try running:
usbconfig dump_device_descThat might show you what that device actually is.
-
@provels Not intentionally, asrock has no options for it other than this network bond config which isn't the same thing.
@stephenw10 Yep, I can see it there in the file, and nope it didn't disable it.
This is the output of that command:
I don't see it there... (yet its still in the system)
/root: usbconfig dump_device_desc ugen1.1: <AMD XHCI root HUB> at usbus1, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SA VE (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0300 bDeviceClass = 0x0009 <HUB> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0003 bMaxPacketSize0 = 0x0009 idVendor = 0x0000 idProduct = 0x0000 bcdDevice = 0x0100 iManufacturer = 0x0001 <AMD> iProduct = 0x0002 <XHCI root HUB> iSerialNumber = 0x0000 <no string> bNumConfigurations = 0x0001 ugen0.1: <AMD XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SA VE (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0300 bDeviceClass = 0x0009 <HUB> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0003 bMaxPacketSize0 = 0x0009 idVendor = 0x0000 idProduct = 0x0000 bcdDevice = 0x0100 iManufacturer = 0x0001 <AMD> iProduct = 0x0002 <XHCI root HUB> iSerialNumber = 0x0000 <no string> bNumConfigurations = 0x0001 ugen2.1: <AMD XHCI root HUB> at usbus2, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SA VE (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0300 bDeviceClass = 0x0009 <HUB> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0003 bMaxPacketSize0 = 0x0009 idVendor = 0x0000 idProduct = 0x0000 bcdDevice = 0x0100 iManufacturer = 0x0001 <AMD> iProduct = 0x0002 <XHCI root HUB> iSerialNumber = 0x0000 <no string> bNumConfigurations = 0x0001 ugen0.2: <American Megatrends Inc. Virtual Hub> at usbus0, cfg=0 md=HOST spd=HIG H (480Mbps) pwr=SAVE (100mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x0009 <HUB> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0001 bMaxPacketSize0 = 0x0040 idVendor = 0x046b idProduct = 0xff01 bcdDevice = 0x0100 iManufacturer = 0x0001 <American Megatrends Inc.> iProduct = 0x0002 <Virtual Hub> iSerialNumber = 0x0003 <serial> bNumConfigurations = 0x0001 ugen0.3: <American Megatrends Inc. Virtual Ethernet> at usbus0, cfg=0 md=HOST sp d=HIGH (480Mbps) pwr=ON (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x0002 <Communication device> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0000 bMaxPacketSize0 = 0x0040 idVendor = 0x046b idProduct = 0xffb0 bcdDevice = 0x0100 iManufacturer = 0x0001 <American Megatrends Inc.> iProduct = 0x0002 <Virtual Ethernet> iSerialNumber = 0x0003 <1234567890> bNumConfigurations = 0x0002 ugen0.4: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0110 bDeviceClass = 0x0000 <Probed by interface class> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0000 bMaxPacketSize0 = 0x0008 idVendor = 0x046b idProduct = 0xff10 bcdDevice = 0x0100 iManufacturer = 0x0001 <American Megatrends Inc.> iProduct = 0x0002 <Virtual Keyboard and Mouse> iSerialNumber = 0x0000 <no string> bNumConfigurations = 0x0001 ugen0.5: <American Megatrends Inc. Virtual Cdrom Device> at usbus0, cfg=0 md=HOS T spd=HIGH (480Mbps) pwr=ON (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x0000 <Probed by interface class> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0000 bMaxPacketSize0 = 0x0040 idVendor = 0x046b idProduct = 0xff20 bcdDevice = 0x0100 iManufacturer = 0x0001 <American Megatrends Inc.> iProduct = 0x0002 <Virtual Cdrom Device> iSerialNumber = 0x0003 <AAAABBBBCCCC1> bNumConfigurations = 0x0001 ugen0.6: <American Megatrends Inc. Virtual HardDisk Device> at usbus0, cfg=0 md= HOST spd=HIGH (480Mbps) pwr=ON (0mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x0000 <Probed by interface class> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0000 bMaxPacketSize0 = 0x0040 idVendor = 0x046b idProduct = 0xff31 bcdDevice = 0x0100 iManufacturer = 0x0001 <American Megatrends Inc.> iProduct = 0x0002 <Virtual HardDisk Device> iSerialNumber = 0x0003 <AAAABBBBCCCC3> bNumConfigurations = 0x0001EDIT: Nevermind, here it is:
ugen0.3: <American Megatrends Inc. Virtual Ethernet> at usbus0, cfg=0 md=HOST sp d=HIGH (480Mbps) pwr=ON (0mA) -
Yup so it's a virtual device. If there's nothing in the BIOS then maybe something in the IPMI setup?
How does it appear in the boot log when it attaches? You might need to try disabling cdce0 instead. If that's how it appears.
-
@stephenw10 Nothing in both places. Only things I can disable are the virtual media devices & KBM emulation. No mention of any virtual network adapters at all.
This is all thats mentioned of it in the boot log:
uhub2: 6 ports with 6 removable, self powered uhub1: 6 ports with 6 removable, self powered uhub0: 22 ports with 22 removable, self powered Root mount waiting for: usbus0 ugen0.2: <American Megatrends Inc. Virtual Hub> at usbus0 uhub3 on uhub0 uhub3: <7-port Hub> on usbus0 uhub3: 5 ports with 5 removable, self powered Root mount waiting for: usbus0 ugen0.3: <American Megatrends Inc. Virtual Ethernet> at usbus0 Root mount waiting for: usbus0 ugen0.4: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0 ukbd0 on uhub3 ukbd0: <Keyboard Interface> on usbus0 kbd2 at ukbd0 -
-
It must list the device and the driver attaching to it in the boot log. Something like:
ugen0.3: <Realtek USB 10/100/1G/2.5G LAN> at usbus0 cdce0 on uhub0 cdce0: <CDC Communications Control> on usbus0 ue0: <USB Ethernet> on cdce0 ue0: Ethernet address: 00:e0:81:21:78:eaAnd if it does appear like that try adding to loader.conf.local:
hint.cdce.0.disabled="1" -
@Bob-Dig said in Netgate Device ID changing on power loss:
@n1ck31 Maybe try disabling the BMC with a jumper and see if this helps.
Although that probably will solve the issue, that negates the scalped price I paid for this board originally - if i were to go this route I'd just buy a regular consumer board and swap it out.
-
@stephenw10 said in Netgate Device ID changing on power loss:
It must list the device and the driver attaching to it in the boot log. Something like:
ugen0.3: <Realtek USB 10/100/1G/2.5G LAN> at usbus0 cdce0 on uhub0 cdce0: <CDC Communications Control> on usbus0 ue0: <USB Ethernet> on cdce0 ue0: Ethernet address: 00:e0:81:21:78:eaAnd if it does appear like that try adding to loader.conf.local:
hint.cdce.0.disabled="1"Here's the full log file
It seems that it's usbus0 on xhci0, so maybe disabling xhci0?
Copyright (c) 1992-2023 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64 FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) VT(efifb): resolution 1024x768 CPU: AMD Ryzen 5 5600G with Radeon Graphics (3900.23-MHz K8-class CPU) Origin="AuthenticAMD" Id=0xa50f00 Family=0x19 Model=0x50 Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x7ef8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX> Structured Extended Features=0x219c97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> Structured Extended Features2=0x40069c<UMIP,PKU,OSPKE,VAES,VPCLMULQDQ,RDPID> Structured Extended Features3=0x10<FSRM> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> AMD Extended Feature Extensions ID EBX=0x191ef657<CLZERO,IRPerf,XSaveErPtr,RDPRU,WBNOINVD,IBPB,IBRS,STIBP,STIBP_ALWAYSON,PREFER_IBRS,SSBD> SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 15924834304 (15187 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <ALASKA A M I > FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 <Version 2.1> irqs 0-23 ioapic1 <Version 2.1> irqs 24-55 Launching APs: 1 3 2 10 7 11 6 8 9 4 5 TCP_ratelimit: Is now initialized wlan: mac acl policy registered random: entropy device external interface kbd1 at kbdmux0 WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0. netgate0: <unknown hardware> efirtc0: <EFI Realtime Clock> efirtc0: registered as a time-of-day clock, resolution 1.000000s smbios0: <System Management BIOS> at iomem 0xbcd72000-0xbcd7201e smbios0: Version: 3.3, BCD Revision: 3.3 acpi0: <ALASKA A M I > acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 350 Event timer "HPET2" frequency 14318180 Hz quality 350 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) pcib1: <ACPI PCI-PCI bridge> at device 2.1 on pci0 pci1: <ACPI PCI bus> on pcib1 xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfc6a0000-0xfc6a7fff irq 36 at device 0.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 ahci0: <AHCI SATA controller> mem 0xfc680000-0xfc69ffff irq 37 at device 0.1 on pci1 ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ahcich6: <AHCI channel> at channel 6 on ahci0 ahcich7: <AHCI channel> at channel 7 on ahci0 pcib2: <ACPI PCI-PCI bridge> irq 38 at device 0.2 on pci1 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> irq 36 at device 0.0 on pci2 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> irq 36 at device 0.0 on pci3 pci4: <ACPI PCI bus> on pcib4 vgapci0: <VGA-compatible display> port 0xf000-0xf07f mem 0xfb000000-0xfbffffff,0xfc000000-0xfc01ffff irq 36 at device 0.0 on pci4 vgapci0: Boot video device pcib5: <ACPI PCI-PCI bridge> irq 37 at device 1.0 on pci2 pci5: <ACPI PCI bus> on pcib5 pcib6: <ACPI PCI-PCI bridge> irq 38 at device 2.0 on pci2 pci6: <ACPI PCI bus> on pcib6 pcib7: <ACPI PCI-PCI bridge> irq 39 at device 3.0 on pci2 pci7: <ACPI PCI bus> on pcib7 igc0: <Intel(R) Ethernet Controller I225-V> mem 0xfc300000-0xfc3fffff,0xfc400000-0xfc403fff irq 39 at device 0.0 on pci7 igc0: Using 1024 TX descriptors and 1024 RX descriptors igc0: Using 4 RX queues 4 TX queues igc0: Using MSI-X interrupts with 5 vectors igc0: Ethernet address: 88:c9:b3:bf:75:60 igc0: netmap queues/slots: TX 4/1024, RX 4/1024 pcib8: <ACPI PCI-PCI bridge> irq 36 at device 4.0 on pci2 pci8: <ACPI PCI bus> on pcib8 nvme0: <Generic NVMe Device> mem 0xfc500000-0xfc503fff,0xfc504000-0xfc5040ff irq 36 at device 0.0 on pci8 pcib9: <ACPI PCI-PCI bridge> irq 36 at device 8.0 on pci2 pci9: <ACPI PCI bus> on pcib9 pcib10: <ACPI PCI-PCI bridge> at device 2.2 on pci0 pci10: <ACPI PCI bus> on pcib10 ix0: <Intel(R) X550-T2> mem 0xe0800000-0xe0bfffff,0xe0c04000-0xe0c07fff irq 40 at device 0.0 on pci10 ix0: Using 2048 TX descriptors and 2048 RX descriptors ix0: Using 6 RX queues 6 TX queues ix0: Using MSI-X interrupts with 7 vectors ix0: allocated for 6 queues ix0: allocated for 6 rx queues ix0: Ethernet address: d0:50:99:de:c1:1b ix0: PCI Express Bus: Speed 8.0GT/s Width x1 ix0: Option ROM V1-b1927-p0 eTrack 0x80000aee PHY FW V523 ix0: netmap queues/slots: TX 6/2048, RX 6/2048 ix1: <Intel(R) X550-T2> mem 0xe0400000-0xe07fffff,0xe0c00000-0xe0c03fff irq 41 at device 0.1 on pci10 ix1: Using 2048 TX descriptors and 2048 RX descriptors ix1: Using 6 RX queues 6 TX queues ix1: Using MSI-X interrupts with 7 vectors ix1: allocated for 6 queues ix1: allocated for 6 rx queues ix1: Ethernet address: d0:50:99:de:c1:1c ix1: PCI Express Bus: Speed 8.0GT/s Width x1 ix1: Option ROM V1-b1927-p0 eTrack 0x80000aee PHY FW V523 ix1: netmap queues/slots: TX 6/2048, RX 6/2048 pcib11: <ACPI PCI-PCI bridge> at device 8.1 on pci0 pci11: <ACPI PCI bus> on pcib11 vgapci1: <VGA-compatible display> port 0xe000-0xe0ff mem 0xd0000000-0xdfffffff,0xe0000000-0xe01fffff,0xfcb00000-0xfcb7ffff irq 38 at device 0.0 on pci11 hdac0: <ATI (0x1637) HDA Controller> mem 0xfcb88000-0xfcb8bfff irq 39 at device 0.1 on pci11 pci11: <encrypt/decrypt> at device 0.2 (no driver attached) xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfc900000-0xfc9fffff irq 37 at device 0.3 on pci11 xhci1: 64 bytes context size, 64-bit DMA usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 xhci2: <XHCI (generic) USB 3.0 controller> mem 0xfc800000-0xfc8fffff irq 38 at device 0.4 on pci11 xhci2: 64 bytes context size, 64-bit DMA usbus2 on xhci2 usbus2: 5.0Gbps Super Speed USB v3.0 hdac1: <AMD Raven HDA Controller> mem 0xfcb80000-0xfcb87fff irq 36 at device 0.6 on pci11 pcib12: <ACPI PCI-PCI bridge> at device 8.2 on pci0 pci12: <ACPI PCI bus> on pcib12 ahci1: <AMD KERNCZ AHCI SATA controller> mem 0xfcc01000-0xfcc017ff irq 34 at device 0.0 on pci12 ahci1: AHCI v1.31 with 1 6Gbps ports, Port Multiplier supported with FBS ahcich8: <AHCI channel> at channel 0 on ahci1 ahci2: <AMD KERNCZ AHCI SATA controller> mem 0xfcc00000-0xfcc007ff irq 35 at device 0.1 on pci12 ahci2: AHCI v1.31 with 1 6Gbps ports, Port Multiplier supported with FBS ahcich9: <AHCI channel> at channel 0 on ahci2 isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 acpi_button0: <Power Button> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 hwpstate0: <Cool`n'Quiet 2.0> on cpu0 Timecounter "TSC-low" frequency 1949999786 Hz quality 1000 Timecounters tick every 1.000 msec ZFS filesystem version: 5 ZFS storage pool version: features support (5000) ugen1.1: <AMD XHCI root HUB> at usbus1 ugen2.1: <AMD XHCI root HUB> at usbus2 ugen0.1: <AMD XHCI root HUB> at usbus0 nvme0: Allocated 200MB host memory buffer uhub0 on usbus1 uhub0: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 uhub1 on usbus2 uhub1: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 uhub2 on usbus0 uhub2: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 hdacc0: <ATI R6xx HDA CODEC> at cad 0 on hdac0 hdaa0: <ATI R6xx Audio Function Group> at nid 1 on hdacc0 pcm0: <ATI R6xx (HDMI)> at nid 3 on hdaa0 pcm1: <ATI R6xx (HDMI)> at nid 5 on hdaa0 pcm2: <ATI R6xx (HDMI)> at nid 7 on hdaa0 pcm3: <ATI R6xx (HDMI)> at nid 9 on hdaa0 Trying to mount root from zfs:pfSense/ROOT/default []... nda0 at nvme0 bus 0 scbus10 target 0 lun 1 nda0: <WD Blue SN570 250GB 234110WD 22395T804226> nda0: Serial Number 22395T804226 nda0: nvme version 1.4 nda0: 238475MB (488397168 512 byte sectors) uhub1: 6 ports with 6 removable, self powered uhub0: 6 ports with 6 removable, self powered uhub2: 22 ports with 22 removable, self powered Root mount waiting for: usbus0 ugen0.2: <American Megatrends Inc. Virtual Hub> at usbus0 uhub3 on uhub2 uhub3: <7-port Hub> on usbus0 uhub3: 5 ports with 5 removable, self powered Root mount waiting for: usbus0 ugen0.3: <American Megatrends Inc. Virtual Ethernet> at usbus0 Root mount waiting for: usbus0 ugen0.4: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0 ukbd0 on uhub3 ukbd0: <Keyboard Interface> on usbus0 kbd2 at ukbd0 CPU: AMD Ryzen 5 5600G with Radeon Graphics (3900.00-MHz K8-class CPU) Origin="AuthenticAMD" Id=0xa50f00 Family=0x19 Model=0x50 Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x7ef8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX> Structured Extended Features=0x219c97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> Structured Extended Features2=0x40069c<UMIP,PKU,OSPKE,VAES,VPCLMULQDQ,RDPID> Structured Extended Features3=0x10<FSRM> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> AMD Extended Feature Extensions ID EBX=0x191ef657<CLZERO,IRPerf,XSaveErPtr,RDPRU,WBNOINVD,IBPB,IBRS,STIBP,STIBP_ALWAYSON,PREFER_IBRS,SSBD> SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics -
You should disable the on-board audio, though I doubt it's connected to this issue:
hdacc0: <ATI R6xx HDA CODEC> at cad 0 on hdac0 hdaa0: <ATI R6xx Audio Function Group> at nid 1 on hdacc0 pcm0: <ATI R6xx (HDMI)> at nid 3 on hdaa0 pcm1: <ATI R6xx (HDMI)> at nid 5 on hdaa0 pcm2: <ATI R6xx (HDMI)> at nid 7 on hdaa0 pcm3: <ATI R6xx (HDMI)> at nid 9 on hdaa0Ok the actual driver attachment looks to have happened after the bootlog completes but it should be in the main system log.
Yes, you could disable the usbus or root device completely but it would be better to disable just the Ethernet device if you can.
-
Try running:
usbconfig -d ugen0.3 dump_all_config_descIt looks like it has more than one config index, in which case we can force the other one which may make it fail to load.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.