FTTH (AON): Fritz!Box 5530 works, pfSense not

waldy327

Hey, thank you for your ideas. :-)

First of all, there are some things which I don't understand:

In my region the Genexis does the VLAN tagging. But there were some customers needed the VLAN id on their copper based routers behind those Genexis bridges. e.g. here, someone configured it on a OPNsense successfully. I don't really understand this different behaviour, if the Genexis box would be really only a bridge.
https://forum.opnsense.org/index.php?topic=13172.0
Now, I connected my Linux laptop to the Genexis LAN port and traced the traffic. Ok, not waited 1h to get an IP, but I could see much more incoming traffic than on the pfSense when I do a trace on the WAN side.
Also connected the laptop to the switch as I think the Genexis work. Same problem like connecting the pfSense directly to the fiber. No incoming traffic. (configured the SFP Port on the switch as a tagged port in vlan 362 and the "media conversion" port for the laptop as an untagged access port belonging to the same VLAN 362)
Maybe the network traces from the Fritz!Box 5530 WAN are helping? Maybe it is actual the problem, that I have to accept, both untagged and tagged traffic, because as you can see in the first trace there are incoming HSRP multicast packets related to a vlan 302)

Without the active IP connection:
https://www.dropbox.com/s/030172c2pbg2dm8/fritzbox-vcc0_01.01.70_0111.eth?dl=0

And with the IP connection:
https://www.dropbox.com/s/1n0w6slzs9llwgx/fritzbox-vcc0_31.01.22_0246.eth?dl=0

@keyser
||Are you sure about the VLAN tagged interface?||

Yes. The interface must be definitely tagged on the outgoing side regarding to my ISP and the trace of the Fritz!Box 5530, where the DHCP packets were tagged.

So, I think the first solution would not work for me. Also tried it already in a bit different way. But no luck.

However the secondary suggestion sounds really interesting, but I think I don't understand it completely. :-(
On which interface should I terminate the DHCP traffic? On the vlan interface of the switch? Or on the untagged vlan 1?

@Cool_Corona
Yeah! I could try a media converter for tracing purposes. Maybe that helps to find out the protocol missing on the pfSense. Or does the switch nearly the same thing, doesn't it?
Apart from this I think our fiber ISPs here in Germany are strange, so most of them don't really allow or support own equipment directly connected to the fiber and generally dictate their "bridged" Genexis garbage. ;-)

@stephenw10
No. My ISP doesn't really support it. I would like to centralize the traffic on my pfSense box and to be future ready, when 10G is offered in the consumer market.
The priority bit is obviously not important.

stephenw10

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

The priority bit is obviously not important.

Yes, that seems to be the case here if your laptop can work directly. Some ISPs do require it though.

waldy327

@stephenw10
hmm...I don't think that my laptop works correctly. It looked like the same problem. Also I had to connect the laptop via the switch, because I have no external usb-c
SFP module for it.

Only the Fritz!Box 5530 (why the hell?!) works.

I think, I will buy and try the media converter. Maybe this small box does or does not something magic...

keyser

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

However the secondary suggestion sounds really interesting, but I think I don't understand it completely. :-(
On which interface should I terminate the DHCP traffic? On the vlan interface of the switch? Or on the untagged vlan 1?

Doesn’t matter as the patch between 1 and 362 (untagged) effective makes both VLANs the same L2 domain.

But If you see no DHCP reply frames in a pure tagged vlan362 test and likewise no reply frames in the first test i suggested, then this second test will not work either.

I think your assumption about the link being good is wrong. Once you use the fiber outside the fritzbox, there is something preventing you from recieve frames intirely. We can only assume the problem is the same on the ISP end, and they never see the frames you transmit. Perhaps the fritzbox runs with MacSEC (encrypted L2)?

stephenw10

Mmm. Running through the switch and running a pcap on a mirror port might be the only way to know for sure. Depends how badly you want this I guess.

waldy327

@keyser
Ok, thank your for your explanation.

First of all, this could be true

I think your assumption about the link being good is wrong.
if you mean the layer 2. The physical connection must work in my opinion.

I now checked the VLAN configuration, does not work, too. So, I really wouldn't exclude that there must be something special configured on layer2, because in every configuration - also doing port mirroring on the switch port - I couldn't see any packets incoming (only some own 362 tagged stuff) which looks really weird for me.

But...how will MacSEC work? I never told my ISP the MAC address of the Fritz!Box. It is my own box. Or are the MacSEC keys exchanged dynamically? Then, I could test it on a linux system which seems to support the MacSEC protocol... :-)

stephenw10

How exactly was the mirror setup? There must have been reply packets of some sort if it successfully pulls an IP address.

Steve

waldy327

@stephenw10
hmm...I configured the port of my laptop as the destination port and the SFP port as source port and activated egress and ingress mirroring. Or is that wrong? ;-)

stephenw10

Well what I would try to do is put the switch in between the incoming connection and a device that successfully connects. Then mirror one of the ports to another port and capture on that.

There must be two way traffic so it has to be captured by doing that.

Steve

keyser

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

But...how will MacSEC work? I never told my ISP the MAC address of the Fritz!Box. It is my own box. Or are the MacSEC keys exchanged dynamically? Then, I could test it on a linux system which seems to support the MacSEC protocol... :-)

If it’s your own box (bought it yourself), it’s not MacSEC. To use MacSEC the box needs either a provisioned CA/Key or to be setup for MacSEC via 802.1x port auth. You would know if you had to do either when you bought the box.

waldy327

Hey,
my media converter arrived today! Tried it directly on my ISP's fiber and it worked with my laptop. I could see HSRP packets (from a tagged vlan 302) like on the Fritz!Box's WAN port. :-)

On the one hand that makes me really happy as it means there must be in general no technical problem to connect an own SFP module to the fiber line. So, simple Ethernet...yeah!

But ...where is the difference between the simple media converter and my switch (or the pfSense)? What do I have to configure that both work nearly on the same way? In my opinion every layer3 or layer2 device can be configured as a dumb layer1 device. ;-)

stephenw10

Hmm, so not even a VLAN required on the client at all?

waldy327

@stephenw10
Yes and no. The VLAN id 362 is required for the normal communication and getting an IP via DHCP. But on the underlying interface I can see that multicast traffic from other VLANs.

keyser

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

Hey,
my media converter arrived today! Tried it directly on my ISP's fiber and it worked with my laptop. I could see HSRP packets (from a tagged vlan 302) like on the Fritz!Box's WAN port. :-)

On the one hand that makes me really happy as it means there must be in general no technical problem to connect an own SFP module to the fiber line. So, simple Ethernet...yeah!

But ...where is the difference between the simple media converter and my switch (or the pfSense)? What do I have to configure that both work nearly on the same way? In my opinion every layer3 or layer2 device can be configured as a dumb layer1 device. ;-)

Okay - Then we must be looking at something wrong with the underlying link - just not “link” pr. se, but rather that your SFP does not work (send/recieve) even though it seems to, and shows link. Since you are unable to see any frames recieved on the Switchport or pfSenes packet capture, it must be because the SFP does not work with your switch/pfSense device.

stephenw10

Ok so, to be clear, you were setting VLAN362 on your laptop directly to get a connection?

Can you test pfSense via the media converter?

You might try disabling hardware VLAN offloading on the ix SPF NIC. Though I wouldn't expect an issue with that.

Steve

waldy327

I now connected the pfSense to the media converter and I am totally confused. It works with perfect! Ok, nearly as I think the media converter has maybe little bandwidth problems with single streams, but for testing it is oK. Maybe, it also could be the wrong time for speed testing. ;-)

What I do not understand:
1.) If it would be an issue with the SFP compatibility on my switch/pfSense device, I would expect that the SFP modules would not work on the LAN side, too. But between two switches and between the pfSense box the SFP modules works fine.

2.) The media converter is a TP-Link one, the switch also. So, shouldn't be a problem?

3.) @stephenw10 How can I disable hardware VLAN offloading in the pfSense? Is there a kernel parameter?
Or is it enough to disable
"Hardware TCP Segmentation Offloading"
"Hardware Large Receive Offloading"
?

However, the igb0 adapter looks like this:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT3
	options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>

And the ix0 adapter like I have posted above. Or do I have to disable the VLAN_HWTSO capability explicitly?

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
...
options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>

Insofar I don't see any difference with hardware based VLAN options.

keyser

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

I now connected the pfSense to the media converter and I am totally confused. It works with perfect! Ok, nearly as I think the media converter has maybe little bandwidth problems with single streams, but for testing it is oK. Maybe, it also could be the wrong time for speed testing. ;-)

What I do not understand:
1.) If it would be an issue with the SFP compatibility on my switch/pfSense device, I would expect that the SFP modules would not work on the LAN side, too. But between two switches and between the pfSense box the SFP modules works fine.

2.) The media converter is a TP-Link one, the switch also. So, shouldn't be a problem?

3.) @stephenw10 How can I disable hardware VLAN offloading in the pfSense? Is there a kernel parameter?
Or is it enough to disable
"Hardware TCP Segmentation Offloading"
"Hardware Large Receive Offloading"
?

However, the igb0 adapter looks like this:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT3
	options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
And the ix0 adapter like I have posted above. Or do I have to disable the VLAN_HWTSO capability explicitly?
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
...
options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
Insofar I don't see any difference with hardware based VLAN options.

Good point about the SFP working in the LAN interface…
Perhaps it’s the ix interfaces that are not happy with the SFP, but the IGb is?

waldy327

Perhaps it’s the ix interfaces that are not happy with the SFP, but the IGb is?

No. I don't think so.
My igb* interfaces are all RJ45 based. ;-)
Only the two ix* interfaces are SFP/SFP+ based (Intel X552 onboard NICs).

stephenw10

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

Or is it enough to disable
"Hardware TCP Segmentation Offloading"
"Hardware Large Receive Offloading"

Those should be disabled anyway, they are disabled by default so definitely disabled them if you have set them enabled.

Hardware offloading requires the driver and hardware to work correctly together. Something that works on an igb NIC might work on ix. It might not even work on a different NIC that also uses the igb driver.
They usually do though because those Intels are the best supported. Intel contributes their own driver code to FreeBSD.

To disable that as a test you can run at the command line:

ifconfig ix0 -vlanhwfilter -vlanmtu -vlanhwtag -vlanhwcsum

I had assumed your igb NICs are not SFP?

Steve