Issues transferring to new hardware

cr

We have a working network using one of our own servers with pfSense installed. We are trying to release this hardware for other uses so we purchased a pfSense xg-7100-1u.
https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/index.html

No matter what we try we can not get the xg-7100 to connect.

The CPE unit from BT provides a 1gbps fibre connection with a static IP.

The current, live firewall, uses pfSense 2.6.0-RELEASE community edition.
The new xg-7100 has pfSense+ 22.05-RELEASE and we updated the firmware.

We believe we have configured the 7100 with identical settings to the live firewall, the only difference being physical port assignments.

On the 7100, the WAN connection has status up with all the IPs etc looking correct but the BT CPE has a red LED light. We even copied over the MAC Address from the live unit with no change.
Image showing the 7100 WAN connection:

Ping to 8.8.8.8 or similar IPs fails with 100% loss.

Firewall rules, gateways, DNS, etc all look to be identical between both units. We've had a number of different people look at this over the last few days.

Could it be the port on the 7100 has a hardware incompatibility with the BT provided fibre module?

We've tried cold restart on all the devices in various orders. No change.
We've swapped out cables with no change.

Do you have any ideas?

stephenw10

I assume it's an ADVA FSP150? And it only has the SFP port enabled so you can't use RJ-45?

Which LED shows red?

Those devices will not link at all on the internal port until they see a valid upstream link.

Steve

stephenw10

For reference I know that it can work with the right modules:
See: https://forum.netgate.com/post/1031802

Steve

cr

@stephenw10
Yes, it is the ADVA FSP150-GE102Pro.
Yes, the RJ-45 is disabled. We did try and connect it but nothing.

The red LED is the one between the SFP port and the JR-45 port. If we swap the cable to the working firewall it instantly goes green.

We are using the BT supplied module; simply moving it between the firewalls.

stephenw10

From what we have seen from other ADVA devices they are very fussy about the link config. I imagine it won't do anything until that LED is green. And there is probably nothing helpful on the console.
What does the other hardware show for the link state when it's working?

What is the actual module?

Steve

cr

Thanks Steve, we'll do some more testing later today.

The current hardware shows the link state running with much less detail than the newer plus version of pfSense:

The module is a Cisco brand supplied by BT. I'll ask our sys admin to take a photo of it this morning. There is some vendor information in the above post, generated by pfSense.

We're going to try today with a factory reset on the new netgate hardware and configure only the WAN and LAN using the wizard. We won't be adding any of our network rules at this point we simply want to get to the green light stage. I'll report back on findings/progress.

Thanks for your help.

Saqqara

It could be an issue for the SPF module, and PFSense not liking it.

cr

Photo of module taken out of the working pfSense firewall.

I'm reticent to try this as it means bringing down our production network for an extended time and it adds risk, but we could also transfer the network card from the working firewall into the netgate hardware. Do you think there is merit in doing this? Could it be this module works with our Intel NIC and not with the netgate hardware?

I'm still hopeful the full reset and wizard rebuild will solve things so I will keep the nic transfer as plan B.

I guess another plan would be to ask BT to enable the RJ45 port on the ADVA unit - assuming they can do this.

cr

The ADVA unit is managed by OpenReach.
I can confirm that BT/OpenReach can not / will not enable the RJ45 port on the ADVA unit. Hopefully that will save someone from repeating the couple of hours we've just "invested" to find an engineer willing to confirm this.

stephenw10

Mmm, that's what we found previously. Which is ridiculous because the RJ-45 port clearly can work. It's only their policy preventing it.

Ok the only significant difference I see there is that it's linked without flow-control on the other device in the ixl NIC. So the first thing I would try is disabling flow-control on ix0 in the 7100:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#flow-control

Though I note in my local test it was linked with flow control and the ADVA was happy.

Steve

stephenw10

I also note all 3 modules I tested were 1000BASE-LX.

cr

We reset the 7100 to factory settings then used the set up wizard to configure the device. We could not get a connection.

@stephenw10 said in Issues transferring to new hardware:

the only significant difference I see there is that it's linked without flow-control on the other device

Thanks Steve, we tried this after the factory reset but were still unable to connect. Will did perform a cold reboot.

I guess the last option for us would be to try the current Intel NIC in the netgate 7100. This will need to be carried out at the weekend as it will mean extended downtime for our network.

stephenw10

Yup, that should work.

Be interesting if it does though, with the same module. Hard to see what's different there.