Vanilla install not working interface cannot be accessed

pftdm007

Hey guys

So I'm trying to move away from a virtualized pfSense install to a stand-alone I and I am having problems with the interfaces assignment.

Platform is running a supermicro X9SCL motherboard with two Intel network adapters. Nothing complicated and very vanilla. Somehow I cannot access the LAN IP address nor can I ping it.

At first I didn't had realized that pfSense somehow had assigned em0 to the second interface (Mac ending with :a1) and em1 to the first interface (Mac ending with:a0). I had connected the wan cable to the first interface and the Lan cable to the second interface but obviously the network assignment in pfSense was inverted so that didn't work so I fixed it.

Now
Mac :a0 = em1 = wan
Mac :a1 = em0 = LAN

Now to the actual problem: I can ping other network clients and computers I can access their web interfaces and firmware such as IPMI but cannot access pfSense webUI or ping it. I get network destination host unreachable when I ping from a Linux computer.

Right now pfsense shell interface says

WAN (wan) -> em1 ->
LAN (LAN) -> em0 -> v4: 192.168.0.1/24

Second thing that I noticed: wan doesn't get an IP from my modem (ISP). Shouldn't I see an IP for wan above???

Sorry if something is not clear I am typing this on my phone and it's not easy ;)

Thanks to whoever can shed some light on this situation…

pftdm007

Not sure if this will help but dropping in a shell and running ifconfig em0 I see (only some info here since I can't copy paste the command output)

Ether xxxxxxxx:a1
Inet 192.168.0.1
Media: Ethernet autoselect
Status: no carrier

For em1

Ether xxxxxxxx:a0
Media: Ethernet autoselect
Status: no carrier

phil.davis

"Status: no carrier" means the system has not seen the network cable(s) connected properly to anything. Check all your network cables/switches/connections.

pftdm007

Hmmmm

Ok I managed to find out why WAN did not get an IP at system bootup. Seems, for whatever reason, that the cable modem (thomson DCM-470) had to be hard reset. Now that this happened, I am recalling a few occurences where I did changes on the pfsense side and had to either unplug the modem from power completely, or do a reset after the modifications on pfsense otherwise they would never sync (pfsense would say that it got an IP from my ISP which was true, but could not communicate with the outside world).

Weird, maybe this is normal with cheap consumer grade modems?? Somebody with an enterprise modem can chime in to confirm this ?

For the LAN not being accessible, I am not 100& sure, but I believe the ethernet port on this mobo is defective. If I use the Intel Quad NIC adapter solely, I can get access to LAN. The ethernet port LED's are lit up when I connect a cable, but somehow its like traffic is not getting thru..

Is there a way to test the port per-se and see if it is defective? I've never had a ethernet port go bad on me let alone on a supermicro product.

pftdm007

OK quick question regarding VLAN and managed switch. I have successfully got pfsense running fine, with a different motherboard. So I assume the ethernet port is defective on the X9SCL. Will return it.

For now, em1 is assigned to LAN, and connected to a Procurve 24 port managed switch. All is fine. I would like em2 (the first port of the quad NIC adapter) to be OPT1 for DMZ clients. Since only one port is not sufficient, I thought about bridging all 4 ports of the quad NIC adapter into a single bridge and use it as if it were a switch. That did not work well.

Now plan B looks like taking advantage of my Procurve managed switch, and using VLAN's. I am thinking about creating a VLAN1 that would have ports 0 to 11 (the 12 first ports), then another VLAN with the remaining ports. It would be like splitting the physical switch into 2 independant switch. Before I try this (no experience so I am a bit worried about breaking everything), so I'd ike to confirm:

-VLAN's are the right feature for what I want to do
-I do NOT have to do anything on pfsense at ALL. (only to config the switch as 2 VLAN's)
-The switch webUI will still be accessible via LAN (VLAN1) since it will reside in the same subnet (192.168.0.X). VLAN2 will have subnet 192.168.1.X

Thanks guys!!

divsys

-VLAN's are the right feature for what I want to do
-I do NOT have to do anything on pfsense at ALL. (only to config the switch as 2 VLAN's)
-The switch webUI will still be accessible via LAN (VLAN1) since it will reside in the same subnet (192.168.0.X). VLAN2 will have subnet 192.168.1.X

You want two different subnets to be routed through the same switch but keep them independent of one another (traffic between them handled by pfSense)
VLANs definitely cover of this aspect. You save buying a new switch for the second LAN subnet.
You can leave pfSense alone, it means setting up the switch correctly to tag incoming traffic on the LAN and OPT1 switch ports with the correct VLAN ID's. While that should work, it's not really hard to add VLANs directly into your pfSense config. Either way can be made to work.
If you can talk to every thing else in the LAN subnet, your switch will reside there as well (if you configure it that way) and is just another device.

Small aside, you might want to take this opportunity to move your LAN and OPT1 subnets away from 192.168.0.x and 192.168.1.x. Those two defaults are used by almost EVERY off the shelf device out there. While they probably work fine for now, you're almost guaranteed to get a collision from some other net when you try remote (Coffee shop, Airport, etc.) connectivity. Not a life threatening change, but can save you a wack of hassles in the future.

Just my $.02

pftdm007

Thanks divsys for your input.

Im glad to see that VLAN's will help me achieve the desired setup. YOu understood perfectly what I want to achieve.

I went ahead with setting up VLAN's on the switch (see attached screenshots). VLAN1 has ports 1-16 assigned to it, and ports 17-24 assigned to VLAN2. pfsense only has the interfaces configured, no VLAN related stuff at all. LAN (em1) is physically connected to the port group belonging to VLAN1 in the switch config. Same for OPT1 (em2) on pfsense.

For now it doesnt work. The switch behaves as if it were a single switch, and I cannot access clients on VLAN2.

Then I decided to activate tagging on the ports. I tagged all 24 ports. This is when I lost all network connectivity and had to factory reset the switch…

I am doing something stupid.

it means setting up the switch correctly to tag incoming traffic on the LAN and OPT1 switch ports with the correct VLAN ID'

How do you do this?

While that should work, it's not really hard to add VLANs directly into your pfSense config.

Do you mean that I should do this from within pfsense instead of at the switch level?

Thanks a bunch! Some day I will be smart enough to handle this stuff… ;)

111.png_thumb

222.png_thumb

333.png_thumb

phil.davis

Note first: using VLAN1 for real stuff in many smart switches can have unexpected surprises. I would try with VLAN numbers higher than 1 - e.g. 2 and 3.

What you did initially should work in principle:

Make a bunch of untagged ports in VLAN2 on the switch
Make a different bunch of untagged ports in VLAN3 on the switch
Now the switch is operating like 2 separate hardware switches.
Connect pfSense physical LAN port to one of the untagged VLAN2 ports
Connect pfSense physical OPT1 port to one of the untagged VLAN3 ports

Do nothing special with pfSense - it does not need to know that there are VLANs on the switch, because you have enough physical ports on the pfSense hardware to use 1 physical port per network segment.

(If you wanted LAN and OPT1 to share a single physical port on pfSense, then you would have to make a trunk port on the switch, tagged for both VLAN2 and VLAN3, then create the VLAN interface on pfSense and assign them…)

pftdm007

Hmmm kay.. I took your word of advice and avoided to use VLAN1 for actual work and created 2 & 3… So now I got 3 VLAN's.

VLAN1 - 24 ports assigned (left identical from factory settings)
VLAN2 - ports 1-16 assigned
VLAN3 - ports 17-24 assigned

In the Port Configuration page (screenshot 333 of my previous post):

-All ports are untagged
-All ports are "VLAN aware Enabled" checked (SEE EDIT BELOW)
-All ports "Packet Type" is set to "ALL".
-All ports have "PVID" set to 1.

In the main page of the firmware, the Management VLAN is set to 1.

I still dont understand how VLAN1 can have all ports assigned to it, while VLAN2 and 3 have only specific ports. Doesn't having VLAN1 with 24 ports defeat the purpose of creating 2 VLAN's in the first place? If so, if I uncheck the ports in VLAN1, how will be able to manage the switch?

Nevertheless, I still cant access clients on OPT1 (connected to VLAN3) from LAN (VLAN2).

If my switch configuration is correct, then I suspect the issue is with pfsense itself.

EDIT: Shouldn't I uncheck "VLAN aware Enabled"?? If I understand HP's terminology, VLAN aware means that the ports are configured to work with multiple VLAN's simultaneously, and therefore can interpret tagged packets. This is NOT my case is it? Should I uncheck this setting?

I know I could just try it, but every time I try something, all network crashes and I have to factory reset the switch.. So I'd rather cautiously try stuff instead of just "twitching" around..

divsys

Definitely good advice about staying away from VLAN ID#1 (note that's not necessarily the same as VLAN1 in your switch's parlance!).

As a general suggestion I would suggest creating your two different PVID's in a little higher range, say PVID 10 and PVID 20.
The other thing you can do to keep from getting locked out is only use 6 of your ports while VLAN testing.
You could assign 15-18 to PVID 10, and 20-22 to PVID 20, leaving all the others at factory.
That way if something is majorly wrong, you can still connect back on 1-14 and changes what needs changing.

Keep at it, you'll end up resolving your problem and learning a bunch in the process.

pftdm007

Finally got everything to work as intended!

After a defective motherboard, a defective patch cable, a misconfiguration and confusion with the quad NIC adapter, and the learning curve of VLAN's, its working well I think.

Ports 1-16 are assigned to VLAN1, ports 17-24 are assigned to VLAN2. pfsense LAN is connected to VLAN1, OPT1 to VLAN2. Clients on both VLAN's can get IP's in their respective subnets from pfsense, and after a bit of tweaking on the FW rules, traffic is being routed between subnets so as to the outside (to WAN).

Im still puzzled about the management VLAN (1) which I left intact from factory defaults and has all 24 ports assigned to it. I didnt want to touch this after I saw that my setup worked for fear of breaking something once again.

If someone can shed some light into this for information and improvement purposes, that'd be great!

Special thanks to divsys and phil davis for the help !!!!

divsys

VLAN ID-1 is a special case or factory default VLAN set to all ports out of the box.
The general idea is that VLAN ID is used if you don't want/need specific VLAN's on your switch.

In your scenario you can remove it from most (all) of your live ports.
There should be a reasonably easy way to test this by removing it from the port assigned to one device you can live without.
That way you can test without messing everything else up.

Another way to save your self in case of trouble, is to NOT write your changes to configuration until you're sure they're all good.
That way if something goes ugly, you simply power down the switch and when you bring it back up it will revert to the last written configuration.

phil.davis

I don;t have a smart switch handy to play with. But with ones I have done in the past I just reserved a port for that VLAN1 - e.g. leave just port 1 on VLAN1, then put ports 2-8 on VLAN2 and 9-16 on VLAN3, that sort of thing. Then I always had physical port 1 to plug into if needed.