FTTH (AON): Fritz!Box 5530 works, pfSense not

waldy327

Hi,

I have a strange problem connecting my pfSense directly to the fiber. I have an active fiber connection, so no passive GPON or something else. The physical connection must be okay, because the network LED is on.

My ISP requires a tagged VLAN-ID (here: 362) on the WAN side when not using the bridge (it is a Genexis one) of my my ISP.

So, in my opinion, it should theoretically work.

But I get no IP connection of my ISP. Waiting 1h after changing the router (which is the DHCP lease time of the provider) or longer doesn't really help. The pfSense sends many DHCPREQUESTS but no DHCPOFFER received. Also I cannot see any other packets incoming on the WAN interface like broadcast traffic (HSRP, Hellos, ...). When I connect Fritz!Box 5530 instead I can see that broadcast traffic.

What am I doing wrong? The SFP module should not be the problem. Fiber connection in the LAN works without problems. I also tried the SFP module delivered with the Fritz!Box 5530. It also works with my Intel NIC (without DOM information ;-)) in LAN but not on the WAN side.

I also tried connecting my L2+ switch directly to the fiber with similar problems. I set the tagged VLAN interface to DHCP client mode, but get no IP. The switch shows physical connection as up, but obviously gets also no DHCPOFFER.

Do you have any ideas what I can try? Is there maybe a special protocol needed when I directly connect to the fiber? Or are there any other differences between AON and ethernet, so that setting the VLAN-ID to 362 is not enough?

Thank you for your help!

stephenw10

What does ifconfig -vvvm show for the interface with the module in it?

You could try spoofing the MAC address to match the Fritzbox.

Steve

waldy327

@stephenw10

Thank you for your reply.

Spoofing the MAC address of the Fritz!Box and the Genexis Fibertwist didn't help. Already tried that. :-(

Output of ifconfig seems to be ok. (but currently there is no cable plugged, so the RX is at -40 dBm now)

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT12
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether *******************
	inet6 *******************%ix0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (LC)
	vendor: FS PN: SFP-GE-BX SN: ******************* DATE: 2022-01-06
	module temperature: 50.81 C Voltage: 3.25 Volts
	RX: 0.00 mW (-40.00 dBm) TX: 0.22 mW (-6.42 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 00 00 00 02 00 00 00 00 01 0D 00 0A 64 
	00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 1B 21 53 46 50 2D 47 45 2D 42 
	58 20 20 20 20 20 20 20 20 20 20 20 05 1E 00 0D 
	00 1A 00 00 43 32 31 31 32 33 39 31 32 37 39 20 
	20 20 20 20 32 32 30 31 30 36 20 20 68 90 01 66 
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20

And the output of the related vlan interface:

ix0.362: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WANFIBER
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether *******************
	inet6 *******************%ix0.362 prefixlen 64 scopeid 0x12
	groups: vlan
	vlan: 362 vlanpcp: 0 parent interface: ix0
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

currently there is no cable plugged, so the RX is at -40 dBm

Ok, can you get that output with the cable connected?

Does it show as linked? What sort of link?

It's a 10G NIC but I assume a 1G link? You might need to force it to 1G.

Steve

waldy327

@stephenw10

Following the output with a cable connected. As you can see the link is shown as up, but the vlan interface gets no ip (ok, I waited not for an hour now, but that is the typical behaviour on the WAN side ;-)).

The NIC is a 10G one, right. Forcing to 1G is not possible, because there is own the autoselect option choosable. Or is there another way to force the port down to 1G? On the other hand, that probably does not matter, because connecting the same module on the LAN side to another tranceiver it works fine. So, I think there is no physical problem. For me it looks like an ip or ethernet based problem. But where?

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT12
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether *******************
	inet6 *******************%ix0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (LC)
	vendor: FS PN: SFP-GE-BX SN: ******************* DATE: 2022-01-06
	module temperature: 51.52 C Voltage: 3.24 Volts
	RX: 0.19 mW (-7.11 dBm) TX: 0.23 mW (-6.35 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 00 00 00 02 00 00 00 00 01 0D 00 0A 64 
	00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 1B 21 53 46 50 2D 47 45 2D 42 
	58 20 20 20 20 20 20 20 20 20 20 20 05 1E 00 0D 
	00 1A 00 00 43 32 31 31 32 33 39 31 32 37 39 20 
	20 20 20 20 32 32 30 31 30 36 20 20 68 90 01 66 
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20

ix0.362: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WANFIBER
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether *******************
	inet6 *******************%ix0.362 prefixlen 64 scopeid 0x12
	inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
	groups: vlan
	vlan: 362 vlanpcp: 0 parent interface: ix0
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

keyser

Are you sure about the VLAN tagged interface? I assume you have tested the base untagged interface as well?

From experience I have once tried a very very peculiar setup that required the interface to accept both tagged and untagged frames into the same VLAN for DHCP to complete. Unfortunately that rendered pfSense unusable as a direct connect box because it cannot be configured as such. A switch in between was needed.

But here’s how you can test it: (two different tests)

Try terminating the Fiber(SFP) in your L2+ switch with the port untagged/native in VLAN 362 while the port is still in trunk mode (if running cisco). That should allow your switch to accept both tagged and untagged vlan 362 frames into the same vlan on the interface.
If this works your setup requires to accept vlan362 tagged frames, but must transmit untagged in the same vlan.

The other test, where you transmit with tagged 362 frames and accept both tagged and untagged 362 frames is a lot more difficult to test. One “freaky” way of doing it is:
Switchport with ISP SFP module: untagged vlan 1, tagged 362
You then setup two other switchports, one untagged vlan 1, one untagged vlan362, and patch them directly together.
If this works you must transmit with tagged 362 frames but also accept untagged frames to the same vlan.

Hope my test explanation makes sense. The latter test makes a short duration transmit “double” (both tagged and untagged on 362) until you switch has learned what lives on each port. (So It is a very vey bad setup). But for testing purposes, it should serve to identify the issue.

Cool_Corona

I had the same problem with SAK here in Switzerland.

I changed the Fritzbox and connected an ASUS router and no luck.

When inserting an fiber to RJ45 converter, it all worked and not running tagged VLAN's at all.

So you need a fiber converter....

stephenw10

Does this ISP expect you to be able to use this type of setup or is it just something you're trying?

Do they require priority tagging also?

Steve

waldy327

Hey, thank you for your ideas. :-)

First of all, there are some things which I don't understand:

In my region the Genexis does the VLAN tagging. But there were some customers needed the VLAN id on their copper based routers behind those Genexis bridges. e.g. here, someone configured it on a OPNsense successfully. I don't really understand this different behaviour, if the Genexis box would be really only a bridge.
https://forum.opnsense.org/index.php?topic=13172.0
Now, I connected my Linux laptop to the Genexis LAN port and traced the traffic. Ok, not waited 1h to get an IP, but I could see much more incoming traffic than on the pfSense when I do a trace on the WAN side.
Also connected the laptop to the switch as I think the Genexis work. Same problem like connecting the pfSense directly to the fiber. No incoming traffic. (configured the SFP Port on the switch as a tagged port in vlan 362 and the "media conversion" port for the laptop as an untagged access port belonging to the same VLAN 362)
Maybe the network traces from the Fritz!Box 5530 WAN are helping? Maybe it is actual the problem, that I have to accept, both untagged and tagged traffic, because as you can see in the first trace there are incoming HSRP multicast packets related to a vlan 302)

Without the active IP connection:
https://www.dropbox.com/s/030172c2pbg2dm8/fritzbox-vcc0_01.01.70_0111.eth?dl=0

And with the IP connection:
https://www.dropbox.com/s/1n0w6slzs9llwgx/fritzbox-vcc0_31.01.22_0246.eth?dl=0

@keyser
||Are you sure about the VLAN tagged interface?||

Yes. The interface must be definitely tagged on the outgoing side regarding to my ISP and the trace of the Fritz!Box 5530, where the DHCP packets were tagged.

So, I think the first solution would not work for me. Also tried it already in a bit different way. But no luck.

However the secondary suggestion sounds really interesting, but I think I don't understand it completely. :-(
On which interface should I terminate the DHCP traffic? On the vlan interface of the switch? Or on the untagged vlan 1?

@Cool_Corona
Yeah! I could try a media converter for tracing purposes. Maybe that helps to find out the protocol missing on the pfSense. Or does the switch nearly the same thing, doesn't it?
Apart from this I think our fiber ISPs here in Germany are strange, so most of them don't really allow or support own equipment directly connected to the fiber and generally dictate their "bridged" Genexis garbage. ;-)

@stephenw10
No. My ISP doesn't really support it. I would like to centralize the traffic on my pfSense box and to be future ready, when 10G is offered in the consumer market.
The priority bit is obviously not important.

stephenw10

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

The priority bit is obviously not important.

Yes, that seems to be the case here if your laptop can work directly. Some ISPs do require it though.

waldy327

@stephenw10
hmm...I don't think that my laptop works correctly. It looked like the same problem. Also I had to connect the laptop via the switch, because I have no external usb-c
SFP module for it.

Only the Fritz!Box 5530 (why the hell?!) works.

I think, I will buy and try the media converter. Maybe this small box does or does not something magic...

keyser

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

However the secondary suggestion sounds really interesting, but I think I don't understand it completely. :-(
On which interface should I terminate the DHCP traffic? On the vlan interface of the switch? Or on the untagged vlan 1?

Doesn’t matter as the patch between 1 and 362 (untagged) effective makes both VLANs the same L2 domain.

But If you see no DHCP reply frames in a pure tagged vlan362 test and likewise no reply frames in the first test i suggested, then this second test will not work either.

I think your assumption about the link being good is wrong. Once you use the fiber outside the fritzbox, there is something preventing you from recieve frames intirely. We can only assume the problem is the same on the ISP end, and they never see the frames you transmit. Perhaps the fritzbox runs with MacSEC (encrypted L2)?

stephenw10

Mmm. Running through the switch and running a pcap on a mirror port might be the only way to know for sure. Depends how badly you want this I guess.

waldy327

@keyser
Ok, thank your for your explanation.

First of all, this could be true

I think your assumption about the link being good is wrong.
if you mean the layer 2. The physical connection must work in my opinion.

I now checked the VLAN configuration, does not work, too. So, I really wouldn't exclude that there must be something special configured on layer2, because in every configuration - also doing port mirroring on the switch port - I couldn't see any packets incoming (only some own 362 tagged stuff) which looks really weird for me.

But...how will MacSEC work? I never told my ISP the MAC address of the Fritz!Box. It is my own box. Or are the MacSEC keys exchanged dynamically? Then, I could test it on a linux system which seems to support the MacSEC protocol... :-)

stephenw10

How exactly was the mirror setup? There must have been reply packets of some sort if it successfully pulls an IP address.

Steve

waldy327

@stephenw10
hmm...I configured the port of my laptop as the destination port and the SFP port as source port and activated egress and ingress mirroring. Or is that wrong? ;-)

stephenw10

Well what I would try to do is put the switch in between the incoming connection and a device that successfully connects. Then mirror one of the ports to another port and capture on that.

There must be two way traffic so it has to be captured by doing that.

Steve

keyser

@waldy327 said in FTTH (AON): Fritz!Box 5530 works, pfSense not:

But...how will MacSEC work? I never told my ISP the MAC address of the Fritz!Box. It is my own box. Or are the MacSEC keys exchanged dynamically? Then, I could test it on a linux system which seems to support the MacSEC protocol... :-)

If it’s your own box (bought it yourself), it’s not MacSEC. To use MacSEC the box needs either a provisioned CA/Key or to be setup for MacSEC via 802.1x port auth. You would know if you had to do either when you bought the box.

waldy327

Hey,
my media converter arrived today! Tried it directly on my ISP's fiber and it worked with my laptop. I could see HSRP packets (from a tagged vlan 302) like on the Fritz!Box's WAN port. :-)

On the one hand that makes me really happy as it means there must be in general no technical problem to connect an own SFP module to the fiber line. So, simple Ethernet...yeah!

But ...where is the difference between the simple media converter and my switch (or the pfSense)? What do I have to configure that both work nearly on the same way? In my opinion every layer3 or layer2 device can be configured as a dumb layer1 device. ;-)

stephenw10

Hmm, so not even a VLAN required on the client at all?