Can't get an IP from ISP behind switch

JKnott

When you disable WAN, that NIC still has the same MAC. Are you sure the NIC has been disabled? Try unplugging the disabled one and see if your problem clears.

johnpoz

@JKnott I think he did that already when he says

"Pfsense box to the same switch port as the primary"

So does this 2nd pfsense box get an IP when you use its own mac.. And just disconnect the first one and then power cycle the cable modem?

Any cable modem or setup I have seen, when you change the mac of the device connected you need to power cycle the modem to attach to the new mac.

As to your switch config - why are you tagging? Does your isp require to see a tag. I run my cable modem through my switch to pfsense wan.. But these ports are access with a specific vlan set on the switch..

cable modem -- access port vlan 99 on switch --- switch -- access port port vlan 99 -- (wan) pfsense.

You might tag the vlan if you have setup pfsense wan as a vlan on pfsense? But that tag would only be on the port pfsense is connected to not the modem port.

I use to do the same sort of thing with VMs - if I wanted to fire up a different version of pfsense as vm, or another router distro I wanted to play with - but didn't want to have to cycle my modem.. I would set the mac address of the different vms wan interface to all be the same.. Then would turn off one vm, and boot the other and it would grab the same IP from isp dhcp.

But when I went with hardware for my pfsense all went away - reason I run it through a switch before pfsense is so if need be I can sniff this traffic via span port on the switch..

You might want to do that to watch your dhcp traffic, etc.

If works with remove 1st pfsense, power cycle cable modem, boot 2nd pfsense with its own mac this would validate your switch config is working etc..

If that works.. Then get your 1st pfsense working via dhcp. then with the cloned mac on the 2nd just unplug the 1st pfsense and fire up the 2nd pfsense setting static info that your first pfsense had.. Does this work? If so something wrong with the dhcp.. While the isp dhcp might not renew a lease if not enough time as has elapsed - but when 2nd pfsense sends out a discover you should get another lease.. But could be something on the isp not doing that.. Sniffing the dhcp traffic would tell you if they are sending you a nak or something..

sef1414

@johnpoz

I will have to try using the default MAC on the secondary pfsense box via the switch. I've only tried it without the switch involved (works fine).

I am tagging the WAN so I can have multiple WAN connections in one switch.

So my set up is like yours, except it has a trunk port:

Modem -- Access Port VLAN 101 on switch -- Switch -- Pfsense trunk port -- Pfsense WAN

"If works with remove 1st pfsense, power cycle cable modem, boot 2nd pfsense with its own mac this would validate your switch config is working etc.." - I'm going to give this a go as soon as I can bring the network down.

"Then get your 1st pfsense working via dhcp. then with the cloned mac on the 2nd just unplug the 1st pfsense and fire up the 2nd pfsense setting static info that your first pfsense had.. Does this work?" - This does not work. This is what I did yesterday. My secondary pfsense has the same WAN config as my primary. I was expecting that since I cloned the MAC from primary on that interface, it should simply grab the same IP lease from the modem. But it actually doesn't get any IP at all, even with a power cycle (which I am not expecting to be necessary).

johnpoz

@sef1414 said in Can't get an IP from ISP behind switch:

Modem -- Access Port VLAN 101 on switch -- Switch -- Pfsense trunk port -- Pfsense WAN

Yeah that makes sense if your going to bring in multiple wan connections into pfsense on same physical via vlan tags.

So your saying it doesn't work with same mac and set static IP info to match the dhcp info the 1st pfsense got?

Did you plug this into the same switch port as 1st pfsense? Using the same tag and such on the 2nd pfsense?

That should of worked - if connectivity was there, and mac was the same.. Setting the IP static to what 1st pfsense got via dhcp should work..

JKnott

@johnpoz said in Can't get an IP from ISP behind switch:

Any cable modem or setup I have seen, when you change the mac of the device connected you need to power cycle the modem to attach to the new mac.

With my cable modem, I can have 2 devices connected and they both work. I suspect his problem is he's trying to have 2 devices with the same MAC connected at the same time. With Cisco, when you set up a fallback system, they have the same virtual MAC, not physical, which means the idle one won't be showing the same MAC as the online.

JKnott

@johnpoz said in Can't get an IP from ISP behind switch:

@JKnott I think he did that already when he says

"Pfsense box to the same switch port as the primary"

Must have missed that. I haven't had my morning beer yet.

johnpoz

@JKnott yes its quite possible that your isp might give you more than 1 IP, etc.

But I doubt he is trying to do it at the same time, because he clearly stated

"When I disable WAN on the primary box, and enable it on the secondary"

JKnott

@johnpoz said in Can't get an IP from ISP behind switch:

LAYER 8
GLOBAL MODERATOR
@JKnott
4 minutes ago

@JKnott yes its quite possible that your isp might give you more than 1 IP, etc.

But I doubt he is trying to do it at the same time, because he clearly stated

"When I disable WAN on the primary box, and enable it on the secondary"

As you mentioned, he should do some packet captures to see what's actually happening. It's a point I've often tried to make.

BTW, here's what my ISP has to say about connecting multiple devices on their 8 Gb fibre service. So, they don't just give out a single IPv4 address. They even toss in a switch to help you share the connection.

johnpoz

@JKnott they give you an 8 port 10ge switch for "free" with the service - that is pretty freaking nice of them for sure.

And they hand out multiple public public IPv4 - without added charges, if the rates are reasonable those seem like great bonus to go with them over some other isp that doesn't do that..

JKnott

@johnpoz

I believe it's $300 (Cdn) per month for 8 Gb. And no, there's no extra charge for the IPv4 addresses. Just plug in whatever you want and it works. I don't know if there's a maximum, for example if you had a 48 port switch. But I have had 2 IPv4 addresses for years.

sef1414

@johnpoz

Hmm, yeah. Sorry. I missed that in your first post, thought you were just meaning 2nd pfsense should have the same settings. I did not try setting it to a static IP on secondary, I had it set as DHCP, as the IP is dynamic. I will try doing that though.

sef1414

@JKnott

I did try some packet captures, though I'm a bit inexperienced at that level of detail. All I could see was IP 0.0.0.0 sending requests to a subnet mask like 255.255.255.x and not getting any response. I will re-create and actually copy that info.

JKnott

@sef1414

At the start of the DHCP sequence, before it has received any info, the client will broadcast the request and use 0.0.0.0 as the source address.

sef1414

@johnpoz

Alright, I gave it a go. Changed secondary WAN config from DHCP --> Static, and input the IP fetched from DHCP on the primary box. No dice. Can't get any connectivity.

It showed up on assignments page, but no connectivity and gateway status page showed 100% packet loss.

I'm really stumped here, only way I can get an active connection is by removing the cloned MAC spoof, and then power cycling the modem.

sef1414

@JKnott Yep, this is all I saw on the packet capture

johnpoz

@sef1414 so you sure your clone mac is the same? If you say it works when you use the native mac and just power cycle the modem it points to your clone mac not being correct..

In your packet capture - you see the correct "cloned" mac?

sef1414

@johnpoz Pretty sure. It was copied from the MAC address field of the interfaces status page on the primary box. Just verified that it is correct. Perhaps I'm missing something.

Guess I will need to do a more verbose packet capture?

All I saw was these lines on repeat:

13:21:56.318459 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300

johnpoz

@sef1414 or just download it and look in say wireshark

sef1414

@johnpoz

Alright so I was able to do some more testing, and it is pretty bizarre compared to what I believe should be expected behavior.

I've checked a countless times at this point to ensure the MAC from my primary box is set on the WAN interface of the secondary box. I would expect that my modem can not tell the difference between the two boxes, and both pfsense boxes should be able to grab the same IP from my modem if I disable the WAN interface on one of them, or even perhaps both simultaneously, though I would expect issues in that scenario.

Here are the steps I took with the results:

On pfsense primary, put down WAN interface via ifconfig in SSH session
Physically plug ethernet cable from pfsense secondary into trunk port 5 on switch
Pfsense secondary shows WAN connection as up, but doesn't receive an IP address
Unplug pfsense secondary from trunk port 5, and plug it into trunk port 1 (where pfsense primary was previously plugged in)
Pfsense secondary obtains an IP quickly (without modem reboot) and has connectivity - With a different IP address!
Unplug pfsense secondary from trunk port 1, replace pfsense primary into trunk port 1. - Pfsense primary grabs the previous IP.

I could see it possibly being a switch configuration issue, though I've been over that a dozen times as well. That would not explain pfsense secondary getting a different IP though. My best guess is MAC spoofing is not working somewhere between the interface and modem.