New Fiber install, fresh Pfsense install, only getting 20Mbps up/down

jddoxtator

Just tried putting a switch between the converter and the WAN port. No change.

Also tried Disable hardware checksum offload, that just caused the router to lose all connectivity and forced me to reload defaults.

I'm back to the ISP router for now paired with the breakout switch until I can get some input on this. I've tried all the guide troubleshooting tips and nothing seems to work.

Patch

@jddoxtator said in New Fiber install, fresh Pfsense install, only getting 20Mbps up/down:

The fiber is plugged directly to WAN port via optical/copper converter.

Is this converter also used with the ISP router and what converter are you using?
How does the ISP router configure the WAN.

jddoxtator

@patch The ISP configures it with DHCP, MAC address and an IP alias. The gateway is picked up automatically.

Converter is UMC-GA1F1T some FS-unlimited brand. The router is just a bog standard consumer router.

stephenw10

So is that, yes, the converter is also used with the ISP router?

How are you testing?

jddoxtator

@stephenw10 Yes, they came as a kit. I am looking into a possible FSP adapter change. The FSP adapter they use is only one sided and is unrecognizable by my FSP intelX520 ports.

Oh, and testing on the PC connected to LAN.

stephenw10

Looks like it has some DIP switches including one for FX speed. Have you tried setting those?

Though id the ISP router can get full rate through it that should prove it correctly configured.

jddoxtator

@stephenw10 Yeah, I looked into the dip switches and they are mostly about connection keep alive states. I think there was one that allowed it to down step connection speed based on poor connection, but i'll have to look again.

It would be odd if that were the case because it is an odd speed. i would expect either 10/100/1000 based on the level of connection, not 20.

jddoxtator

@stephenw10
here is the page: https://www.manualslib.com/manual/2199464/Fs-Umc-Ga1f1t.html?page=11#manual

The only one to do with speed only steps it down to 100Mbps, which should still be netting me more than 20Mbps.

I will say when speed testing it does spike to over 100Mbps then very quickly drops to the 20Mbps and ISP router just hums along around 97Mbps

stephenw10

A speed like 20Mbps usually indicates a speed/duplex mismatch somewhere. But if that were the case it would apply to the ISP router too. Also I wouldn't expect to see the spike to 100.

Is it possible the ISP has registered the MAC of the old router and it shaping your traffic with the new MAC? Try spoofing the pfSense WAN MAC to match the other router.

Steve

jddoxtator

@stephenw10 One of the first things I did. It won't even connect to the gateway without the spoof. I am starting to highly suspect that there is some conversion algorithm happening in that converter that Pfsense doesn't like.

My only solution would be to buy a compatable FSP for my intel nic and connect the fiber directly.

JKnott

@jddoxtator

I have a 500/20 package from my ISP and have seen over 900 down. The problem is not with pfSense.

jddoxtator

@jknott Well, yeah. Getting over 400Mbps isn't a problem. If you were getting 400Mbps less then I think you would agree that it is a slight issue.

I'm not going to play the "My software is perfect" game here. We do need to explore the reality of the situation. There are fringe case devices that do not play well together regardless of how perfect you think your software is.

Like I was suggesting, I am highly suspect this offbrand SFP device that came with the converter.

stephenw10

I wouldn't expect anything the converter is doing to affect the connection. As long as it's linked correctly on both sides. But clearly something is happening.

You might try running a packet capture just to see if there is anything obvious being sent.

Steve

jddoxtator

@stephenw10

Ok, I found something. Turns out the ISP router is getting a different remote gateway then the Pfsense router. I'm going to try directing it to that gateway and see what happens.

jddoxtator

Interesting....

the ISP router is getting a gateway IP of 192.24.57.1 and the Pfsense box is getting a gateway IP of 172.31.16.1

If I try to point it towards the Gateway of the ISP router, Pfsense tells me that The gateway address 192.24.57.1 does not lie within one of the chosen interface's subnets.

jddoxtator

Damn no dice.

I found the setting to use out of range IP's in advanced, but even setting it to default gateway it still uses 172.31.17.1

jddoxtator

Noticed another discrepancy. ISP router does not connect IPv6. So I turned it off in Pfsense. No effect, still 20Mbps.

jddoxtator

Really at a loss of what to do now.

Here is the entire connectivity screen from the ISP router. Maybe one of you can see something I am not.

INTERNET

Internet Status
Internet Status reflects the status of the ISP connection.

Connection Status
IPv4 Connection Connected
IPv6 Connection Disconnected

Internet Settings
The table below displays the current state of the Internet connection and settings.

Internet Setting Status
IPv4 WAN Protocol dhcp
IPv6 WAN Protocol dhcpv6
MTU Size 1500
MSS Size 1460
TCP Connection 109
RWIN Size 163840
Packets Sent 4568070
Packets Received 11488214

IPv4 Addressing
The table below displays currently assigned Internet connectivity settings for the device.

Parameter Status
Device IPv4 Address 192.24.57.117
Device IPv4 Subnet Mask 255.255.255.0
DNS Address #1 8.8.8.8
DNS Address #2 64.235.98.226
Remote Gateway Address 192.24.57.1
Link Uptime 0D 0H 0M 44S

keyser

@jddoxtator said in New Fiber install, fresh Pfsense install, only getting 20Mbps up/down:

Interesting....

the ISP router is getting a gateway IP of 192.24.57.1 and the Pfsense box is getting a gateway IP of 172.31.16.1

If I try to point it towards the Gateway of the ISP router, Pfsense tells me that The gateway address 192.24.57.1 does not lie within one of the chosen interface's subnets.

Your ISP router is likely using a certain tagged VLAN on its interface since it lands in a different subnet and gets a different Gateway.
Your pfsense by default uses the untagged native VLAN of the interface.

Try using the ISP router, quickly switch to the pfSense and do a packet capture in promiscous mode (diagnostics -> packet capture).
See if you capture any frames with a VLAN tag that indicates what VLAN you should be usinng

jddoxtator

@keyser

Ok, I captured packets from WAN with nothing attached to make sure there was no activity, then started a new capture and unplugged the WAN from the ISP router and directly plugged it into the WAN on the Pfsense router

This is what I got after 30 seconds of capture:

02:28:01.732611 DTPv1, length 38
02:28:02.699840 ARP, Request who-has 192.24.57.1 tell 192.24.57.117, length 28
02:28:02.731372 DTPv1, length 38
02:28:03.429465 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:03.733838 DTPv1, length 38
02:28:04.264595 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:04.796229 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:05.302364 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:05.773757 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:06.372418 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:06.683297 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:06.816486 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:08.433281 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:08.473367 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:08.822134 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:10.838025 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:11.155034 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:11.194577 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:12.003469 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:12.854932 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:14.882978 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:16.901047 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:18.975871 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:19.022785 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:20.998571 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:22.254055 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:23.002261 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:24.361084 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:25.030319 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:27.049727 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42

It looks like a bunch of spam of IP 0.0.0.68 complaining about topology change. What is interesting is the bridge ID. Is that Pfsense or the ISP gateway?