New Fiber install, fresh Pfsense install, only getting 20Mbps up/down
-
My mistake, the ARP request came from some Calix device.
Here is the expanded log:
1 0.000000 Cisco_89:a0:f6 CDP/VTP/DTP/PAgP/UDLD DTP 60 Dynamic Trunk Protocol
2 0.967229 Calix_6b:e8:f7 Broadcast ARP 42 Who has 192.24.57.1? Tell 192.24.57.117
3 0.998761 Cisco_89:a0:f6 CDP/VTP/DTP/PAgP/UDLD DTP 60 Dynamic Trunk Protocol
4 1.696854 0.0.0.0 255.255.255.255 DHCP 342 DHCP Request - Transaction ID 0xa4d00549
5 2.001227 Cisco_89:a0:f6 CDP/VTP/DTP/PAgP/UDLD DTP 60 Dynamic Trunk Protocol
6 2.531984 0.0.0.0 255.255.255.255 DHCP 342 DHCP Request - Transaction ID 0xa4d00549
7 3.063618 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
8 3.569753 0.0.0.0 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x1c1ffc0e
9 4.041146 fe80::3eec:efff:fe70:1cf5 ff02::1:2 DHCPv6 98 Information-request XID: 0x0163ec CID: 0001000129f61dd33cecef701cf5
10 4.639807 0.0.0.0 255.255.255.255 DHCP 342 DHCP Request - Transaction ID 0xa4d00549
11 4.950686 fe80::3eec:efff:fe70:1cf5 ff02::1:2 DHCPv6 98 Information-request XID: 0x0163ec CID: 0001000129f61dd33cecef701cf5
12 5.083875 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
13 6.700670 0.0.0.0 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x1c1ffc0e
14 6.740756 fe80::3eec:efff:fe70:1cf5 ff02::1:2 DHCPv6 98 Information-request XID: 0x0163ec CID: 0001000129f61dd33cecef701cf5
15 7.089523 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
16 9.105414 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
17 9.422423 0.0.0.0 255.255.255.255 DHCP 342 DHCP Request - Transaction ID 0xa4d00549
18 9.461966 0.0.0.0 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x1c1ffc0e
19 10.270858 fe80::3eec:efff:fe70:1cf5 ff02::1:2 DHCPv6 98 Information-request XID: 0x0163ec CID: 0001000129f61dd33cecef701cf5
20 11.122321 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
21 13.150367 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
22 15.168436 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
23 17.243260 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
24 17.290174 fe80::3eec:efff:fe70:1cf5 ff02::1:2 DHCPv6 98 Information-request XID: 0x0163ec CID: 0001000129f61dd33cecef701cf5
25 19.265960 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
26 20.521444 0.0.0.0 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xa45704e7
27 21.269650 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
28 22.628473 0.0.0.0 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xa45704e7
29 23.297708 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814f
30 25.317116 Cisco_89:a0:f6 PVST+ STP 64 Conf. TC + Root = 24576/85/7c:69:f6:f2:da:40 Cost = 2 Port = 0x814fNot a network engineer, so not exactly sure what I am looking at, but I don't see anything that specifically references VLAN. Unless PVST+ is some kind of VLAN like protocol.
Editr: Search is your friend. It appears that PVST+ is a cisco brand Per VLAN Spanning Tree Plus. Though I don't see anything but a MAC address and ports, no IP to configure a VLAN from.
-
@keyser said in New Fiber install, fresh Pfsense install, only getting 20Mbps up/down:
@jddoxtator said in New Fiber install, fresh Pfsense install, only getting 20Mbps up/down:
Ok, I captured packets from WAN with nothing attached to make sure there was no activity, then started a new capture and unplugged the WAN from the ISP router and directly plugged it into the WAN on the Pfsense router
This is what I got after 30 seconds of capture:
02:28:01.732611 DTPv1, length 38
02:28:02.699840 ARP, Request who-has 192.24.57.1 tell 192.24.57.117, length 28
02:28:02.731372 DTPv1, length 38
02:28:03.429465 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:03.733838 DTPv1, length 38
02:28:04.264595 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:04.796229 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:05.302364 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:05.773757 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:06.372418 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:06.683297 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:06.816486 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:08.433281 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:08.473367 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:08.822134 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:10.838025 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:11.155034 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:11.194577 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:12.003469 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:12.854932 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:14.882978 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:16.901047 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:18.975871 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:19.022785 IP6 fe80::3eec:efff:fe70:1cf5.546 > ff02::1:2.547: UDP, length 36
02:28:20.998571 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:22.254055 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:23.002261 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:24.361084 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
02:28:25.030319 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42
02:28:27.049727 STP 802.1d, Config, Flags [Topology change], bridge-id 8055.e0:2f:6d:a5:16:80.814f, length 42It looks like a bunch of spam of IP 0.0.0.68 complaining about topology change. What is interesting is the bridge ID. Is that Pfsense or the ISP gateway?
Well we can’t decode everything from this as that is only a summary “overview” of the capture. You need to open it in Wireshark or another pcap decoder application.
However, a few things is obvious. Your ISP is not your average setup since they run Spanning Tree to the client edge - that’s a new for me - never seen that before :-)
But there is also Cisco dynamic trunking protocol frames on the wire, so it seems your ISP is running some VLANs on the wire.The funny thing though… all the 0.0.0.0:68 frames is your pfSense trying to aqquire a IP address via DHCP - it doesn’t get any. So there is no Internet available to it - how on earth are you testing with success albeit very slow speed?
Forgot to address the connection with no IP. It does get one, but it seems it is the wrong gateway. They are currently still building out the network in my area, so there may be some insecure patch devices in the line for workers to access? That's my only thought...
-
Mmm, this does seem like either a VLAN is required or maybe priority tagging. Or possibly some DHCP client options.
A pcap of the ISP router connecting would show it either way.Steve
-
So if I am correct in my understanding.... It sounds like I just need to make a VLAN based around the IP address in that ARP request.
I have two IP's
Sender IP address: 192.24.57.117
Target IP address: 192.24.57.1
Target has to be the gatway VLAN and I have to apply this to WAN device?
Oh, and the Calix device is the ISP router, so this was a captured broadcast from the ISP router. I'm guessing I caught the echo off the gateway because it wasn't plugged into any switch. It was a really fast port swap and I had the recorder going when I did it.
Did some more research, and the VLAN tag should be 57 based on the IP addresses I think.
-
Easy enough to test. Create a VLAN interface with ID 57 on the current WAN interface (ix3?). Then reassign WAN to be that new VLAN (ix3.57).
Steve
-
@stephenw10
Tried that, but I think I am extrapolating the Tag ID wrong. as 57 did not work.I think it is the 802.1Q number I am after. Which is 0xa5 or a5 hex / 165 decimal
-
Try that then. Where are you reading that from?
-
That was from the sub menus of DTP. It did not work unfortunately.
I also found Originating VLAN: 85 in PVST+ , but that did not work as well.
I am about to try PID: PVSTP+ (0x010b) or 267 dec.
-
I had the same issues on FTTH here in Switzerland.
It was the SFP+ when mounted in a switch. In a converter it didnt get an address and exposed th routers MAC to the ISP and everything worked perfectly.
-
From what I understand by reading a description of Cisco's implementation of PVST+, DTP is part of the trunk that the routers use in their network. So we can safely ignore that.
STP seems to be the client side of the VLAN. This being the most important information I can find in STP protocol:
Originating VLAN (PVID): 85
Type: Originating VLAN (0x0000)
Length: 2
Originating VLAN: 85By my understanding that should make the VLAN 85, but that doesn't work. So there is still something missing.
-
what brand are the ISP router?
-
@cool_corona Calix
-
Have you told the support that you want to use your own router?
So they will release the MAC and let you do that?
-
@cool_corona Yes, they wont allow it.
-
@jddoxtator Have you tried to spoof the mac of the org router?
-
@cool_corona Yes, the spoof has been enabled since the start.
-
Are there any dip switches in the converter?
-
Ok, testing this locally I expect to be able to see the tagged traffic in the GUI packet capture if the view detail is set to full however there is some oddness there. I'm digging into that but it will show there if you do not filter like:
19:36:07.585799 90:ec:77:1f:8a:5f > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 229, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.229.5.10 tell 10.229.5.1, length 28
There is no question of which VLAN is in use there.
You can also run at the CLI something like:
tcpdump -nvve -i ix0
And you will see all the traffic on the interface including vlan tags.
Steve
-
@stephenw10 Alright tried the console code and got a different VLAN again
15:21:32.364086 3c:ec:ef:70:19:a6 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4, (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 3c:ec:ef:70:19:a6, length 300, xid 0xa6981c02, Flags [none] (0x0000) Client-Ethernet-Address 3c:ec:ef:70:19:a6 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Discover Client-ID Option 61, length 7: ether 3c:ec:ef:70:19:a6 Hostname Option 12, length 7: "pfSense" Parameter-Request Option 55, length 10: Subnet-Mask, BR, Time-Zone, Classless-Static-Route Default-Gateway, Domain-Name, Domain-Name-Server, Hostname Option 119, MTU 15:21:32.865804 10:f9:20:89:a0:f6 > 01:00:0c:cc:cc:cc, 802.3, length 40: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid DTP (0x2004), length 38: DTPv1, length 38 Domain TLV (0x0001) TLV, length 11, Packet Status TLV (0x0002) TLV, length 5, 0x81 DTP type TLV (0x0003) TLV, length 5, 0xa5 Neighbor TLV (0x0004) TLV, length 10, 10:f9:20:89:a0:f6 15:21:33.395704 3c:ec:ef:70:19:a6 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4, (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 3c:ec:ef:70:19:a6, length 300, xid 0xa6981c02, secs 1, Flags [none] (0x0000) Client-Ethernet-Address 3c:ec:ef:70:19:a6 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Discover Client-ID Option 61, length 7: ether 3c:ec:ef:70:19:a6 Hostname Option 12, length 7: "pfSense" Parameter-Request Option 55, length 10: Subnet-Mask, BR, Time-Zone, Classless-Static-Route Default-Gateway, Domain-Name, Domain-Name-Server, Hostname Option 119, MTU 15:21:33.865863 10:f9:20:89:a0:f6 > 01:00:0c:cc:cc:cc, 802.3, length 40: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid DTP (0x2004), length 38: DTPv1, length 38 Domain TLV (0x0001) TLV, length 11, Packet Status TLV (0x0002) TLV, length 5, 0x81 DTP type TLV (0x0003) TLV, length 5, 0xa5 Neighbor TLV (0x0004) TLV, length 10, 10:f9:20:89:a0:f6 15:21:34.410039 3c:ec:ef:70:19:a6 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4, (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 3c:ec:ef:70:19:a6, length 300, xid 0xa6981c02, secs 2, Flags [none] (0x0000) Client-Ethernet-Address 3c:ec:ef:70:19:a6 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Discover Client-ID Option 61, length 7: ether 3c:ec:ef:70:19:a6 Hostname Option 12, length 7: "pfSense" Parameter-Request Option 55, length 10: Subnet-Mask, BR, Time-Zone, Classless-Static-Route Default-Gateway, Domain-Name, Domain-Name-Server, Hostname Option 119, MTU 15:21:35.057589 3c:ec:ef:70:1c:f5 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 3c:ec:ef:70:1c:f5, length 300, xid 0xc9c42930, Flags [none] (0x0000) Client-Ethernet-Address 3c:ec:ef:70:1c:f5 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Discover Client-ID Option 61, length 7: ether 3c:ec:ef:70:1c:f5 MSZ Option 57, length 2: 576 Parameter-Request Option 55, length 7: Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname Domain-Name, BR, NTP Vendor-Class Option 60, length 12: "udhcp 1.23.1" 15:21:35.108688 10:f9:20:89:a0:f6 > 01:00:0c:cc:cc:cc, ethertype 802.1Q (0x8100), length 560: vlan 1, p 7, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid CDP (0x2000), length 534: CDPv2, ttl: 180s, checksum: 0x72f9 (unverified), length 534 Device-ID (0x01), value length: 32 bytes: 'MtBrydges-4507-2.nftctelecom.com' Version String (0x05), value length: 285 bytes: Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500es8-UNIVERSALK9-M), Version 03.09.00.E RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2016 by Cisco Systems, Inc. Compiled Tue 19-Jul-16 12:34 by prod_rel_team Platform (0x06), value length: 17 bytes: 'cisco WS-C4507R+E' Address (0x02), value length: 13 bytes: IPv4 (1) 172.31.16.2 Port-ID (0x03), value length: 19 bytes: 'GigabitEthernet6/15' Capability (0x04), value length: 4 bytes: (0x00000029): Router, L2 Switch, IGMP snooping Prefixes (0x07), value length: 10 bytes: IPv4 Prefixes (2): 172.31.16.0/22 192.168.3.0/24 VTP Management Domain (0x09), value length: 6 bytes: 'Packet' Native VLAN ID (0x0a), value length: 2 bytes: 85 Duplex (0x0b), value length: 1 byte: full AVVID trust bitmap (0x12), value length: 1 byte: 0x00 AVVID untrusted ports CoS (0x13), value length: 1 byte: 0x00 Management Addresses (0x16), value length: 13 bytes: IPv4 (1) 172.31.16.2 unknown field type (0x1a), value length: 12 bytes: 0x0000: 0000 0001 0000 0000 ffff ffff unknown field type (0x1b), value length: 1 byte: 0x0000: 00 unknown field type (0x1f), value length: 1 byte: 0x0000: 00 unknown field type (0x1005), value length: 20 bytes: 0x0000: 5753 2d58 3435 2d53 5550 382d 4500 2830 0x0010: 2972 3f7c unknown field type (0x1004), value length: 15 bytes: 0x0000: 6530 3266 2e36 6461 352e 3136 3830 00 unknown field type (0x1003), value length: 1 byte: 0x0000: 31
I copied everything from connection until response from a cisco router. I see VLAN 1 but I tried that and it gives me no IP. Same as any other VLAN I have tried.
-
That's after setting VLAN1? It looks like dhcp requests from pfSense tagged as that.
You might try switching the ISP router in and back out before the pcap to try to get some tagged traffic from the ISP as you did before with the ARP packet.
Ultimately the only way to know for sure is to setup a switch with a mirror port so you can capture exactly what the ISP router is doing.
The other thing is that you are almost certainly not the first person trying this. Someone else may have documented what's required for that ISP. Somewhere.
Steve