Can't access XG-7100
Hi, I've just configured an XG-7100 and can't access it. I went through the basic setup in our office before then installing it in a rack at the data centre. Here is what I have done...
- Set up two VLAN's on my switch. Ports 1-3 (outside) Ports 4-48 (inside)
- Set up ETH1 on the 7100 to x.x.83.20/29 with gw x.x.83.17
- Set up ETH2 on the 7100 to x.x.86.1/25 (x.x.86.0/25 has been static routed to x.x.83.20)
- Connected uplink 1 in the rack to port 1 on switch and uplink 2 to port 2
- Connected 7100 ETH1 to port 3 on switch and ETH2 to port 4 (inside VLAN) and connected a couple of servers to ports 5 and 6
I can't connect to the 7100 or anything else. Can't ping anything.
The networks guy at the DC says that he can't ping them but he's getting an ARP response...
I am seeing an ARP reply though:
x.x.83.20 00:32:04 MACaddr removed Dynamic ARPA GigabitEthernet0/1/0/10
But, I do however also seem to be seeing ARP entries for your routed subnet as well, which I'd not generally expect to see.
x.x.86.1 00:02:58 MACaddr removed (identical) Dynamic ARPA GigabitEthernet0/1/0/10
x.x.86.121 01:47:54 removed Dynamic ARPA GigabitEthernet0/1/0/10
x.x.86.122 00:02:37 removed Dynamic ARPA GigabitEthernet0/1/0/10
(Of note is that both .20 and the .1 are both MACaddr removed- is that the MAC address you'd expect for eth1 or eth2?)
So for some reason both interfaces on the 7100 are returning the same mac address.
Any ideas why? Would appreciate some pointers on how to go about troubleshooting this. It's a little outside my comfort zone.
The XG-7100 has one 8-Port Switch, this is why you see the same MAC for all 8 Ports.
By default, ETH1 on the the switch is configured as a WAN interface (VLAN 4090) and ETH2-8 are configured as the LAN interface (VLAN 4091).
Did you touch anything there?
Thanks for the reply. I went through the basic set up wizard via the official documentation and configured both interfaces that way. Even if I made a mess of ETH2 I should still be able to connect to the device on the WAN port. I'm out of ideas.
To be honest, I don't understand what your are building up there. So you have a public network connected to your pfSense WAN and another public one to your pfSense LAN? You won't hide RFC1918 networks when posting, right?
What is this static route for?
What exactly can't be accessed? pfSense WebGUI from either WAN and LAN?
@Rico It's configured so that the device doesn't have to be used in "transparent" mode. The firewall is protecting servers that are on public IP's.
The premise as it was explained to me...
we should uplink your router to both of our routers (there will be no additional cost for the extra uplink) using HSRP over a /29 linknet. They will then statically route the /25 to your router for you to use inside of it (this then avoids to need to use transparent mode.
HSRP would mean that you would need to connect our 2 routers and your router into an outside VLAN on your switch.
So I've configured it the way they asked me to. I set up two VLANS on the layer 2 switch (cisco 2960X), configured the interfaces on the pfsense as advised and connected it all up. It's not working unfortunately. I can't see the webgui, or ping the pfsense, or access the two servers I've added.
My level of expertise is not in networking so this is slightly outside my comfort zone.
Did you check and follow https://docs.netgate.com/pfsense/en/latest/book/routing/routing-public-ip-addresses.html ?
@Rico Ahh no I didn't see that! I just assumed the interfaces would be configured separately, although I'm not sure it would make any difference on the WAN port - I should still be able to access that shouldn't I?
This is more complicated than I was expecting. I really appreciate you taking the time to reply. Thank you.
The big question now is...
How do I get back into the 7100 to start again? Is there a "reset to default" sequence on the hardware?
You can factory reset via the Serial console.
Well for the WAN port any traffic entering the interface is blocked by the default Firewall Rules.
I would leave the default LAN as your management port, then you can't lock yourself out as happened now. Create a separate OPT Interface for your internal server network.
@Rico Ok thanks for the advice. This may sound like a dumb question but how do I get the second network on the switch? does it all go through ETH1 with this "OPT" interface? So I only need to connect ETH1 to the switch? what about my VLANs? Do I now need to configure the switch differently?
The switch config is very good explained by @jimp in this hangout: https://www.netgate.com/resources/videos/configuring-netgate-appliance-integrated-switches-on-pfsense-244.html
For example, I have all 8 switch ports as complete discrete ports for my XG-7100. But you can build any switch config you want. The VLANs are only used pfSense internally in this example.
@Rico Sorry I was referring to my network switch. I set up two VLAN's to accommodate this config but I'm not sure if that is relevant now you have advised about the OPT interface?
Assuming we talk about untagged VLANs (Port based) you don't need to change anything there.
If you work with tagged VLANs on this Cisco Switch...well maybe we found your culprit.
@Rico They are just normal VLAN's
Can you explain what the OPT interface is for? I'm a little puzzled how I would connect the pfsense to my network switch if I'm not using ETH2?
Currently I have ETH1 connected to the first VLAN on my switch for outside - and ETH2 connected to the second VLAN on my switch for inside. If you are saying I should not use ETH2 and configure some sort of "virtual port" OPT1" - how can I connect my servers to it?
It's possible to trunk all the VLANs to pfSense across one connection if you need to but that doesn't sound like what you're doing.
There looks to be some confusion here between ports and interfaces. Eth1-8 are ports on the internal switch.
By default the XG-7100 has two interfaces defined:
WAN is connected (by internal VLANs) to port Eth1.
LAN is connected to ports Eth2-8.
The way you describe connecting it in your first post seems correct for those defaults. But the fact the provider is seeing ARP replies for the routed subnet implies your Cisco switch is not correctly segregating those two sets of ports. I would check that first.
Because the XG-7100 has an internal switch you could just connect everything to it directly and not use VLANs on the Cisco (or eve at all until you need more ports).
You would have to reconfigure the 7100 switch ports so that WAN was then Eth1+2 to provide connections to both uplinks and LAN was Eth3-8 for connections on the routed subnet.
Let me know if you want to go that way and I'll set something up.
In your Switch assignment for VLANS if you are tagging traffic always ensure that you include ports 9T and 10T (uplink ports) in the port member list otherwise nothing will get through. I had this problem and although Jim Pingle's presentation is clear, this part is not...well it wasn't to me anyways. The XG7100 is different from other appliances because of the built-in switching which is complicated to understand at first.
@stephenw10 Thank you for your reply Stephen, I really appreciate it.
I have more than 40 servers to connect, so I need the cisco switch.
It's really hard work standing in the hot aisle trying to configure these on the fly, so I will bring back both the switch and the pfsense to the office and try again. I was sure I had set up the VLAN's correctly.
@claferriere Thanks for responding too. I don't believe I'm using VLAN tagging on the Cisco switch - I haven't set anything up to do that.
I believe the networks guy at the DC recommended I use VLANs in order to prevent running the pfsense in "transparent" mode - so while I'm dealing with two public networks, one is inside and one is outside as far as the firewall is concerned. If this isn't the best way to achieve it, I'm very open to suggestions, especially a configuration that is more simple. I can reset both these devices to factory defaults and start again. (not sure how to do that on the pfsense).
I can ask them to change the network to a single uplink for just the /25 I need (which is what I'm accustomed to).
@Chris-187 Not sure I understand the intricacies of your network design, I was just mentioning that on the XG7100 the Switch tab in the interface assignment is new and you need to pay attention to how the VLANs are assigned and how you identify members of that assignment. For me it was simply a question of adding ports 9 and 10 as they are part of the connection to the Denverton Soc. (From the manual: https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/switch-overview.html) Without these members added to the switch config I could NOT access the internet.
@claferriere Ok thanks. The VLANs I created are on a separate switch, not the 7100 itself but I appreciate you taking the time to respond and for the info concerning the 7100 integrated switch.
Two uplinks for failover is a good idea, especially if they're not charging you for it.
It's the routed subnet that allows you to avoid transparent mode and that is also a very good idea.
You are going into the Cisco switch and back out into the XG-7100 switch unnecessarily though. Most firewalls do not have a built in switch so you would need to which is why you were instructed to do so but here you can just re-assign one of the Eth ports to WAN and connect both uplinks to it. Then use any of the other ports to connect to the Cisco switch which no longer needs to have any VLAN config on it.
To do that, change the PVID of port Eth2 to 4090 (the WAN internal VLAN) on the ports tab:
Then on the VLANs tab remove port 2 from LAN vlan group and add it to the WAN:
Now you can connect the two uplinks to Eth1 and Eth2 and everything else to the remaining ports.
@stephenw10 Thanks that's really helpful. My current predicament is that I can't connect to the 7100 at all. I need to be able to either connect to the web interface or reset it somehow. I think I can achieve this simply by connecting the uplink directly to the wan port ETH1, then it should spring into action hopefully.
By default there are no firewall rules to allow access on the WAN, you can only access it by connecting from the LAN side.
If you are unable to connect from either you can always connect to the serial console and add rules to allow you to connect to the GUI or reset the config and start over.
@stephenw10 I've connected one of the uplinks to ETH1 and my laptop to ETH2 by putting it on the same subnet. I can now access the pfsense GUI.
I've set up the WAN interface exactly as described. x.x.83.20/29 with gw x.x.83.17
I've set up the LAN interface to x.x.86.1/25 with a gateway of x.x.86.0
I still can't access x.x.86.1 from outside.
I wonder if I can start this thread again....
I have an XG-7100 rack mount. It is now in my office to be pre-configured to be fitted in a data center.
My requirements are that I need to use the device to protect a bunch of servers that are on public IP's
I have two 1GB uplinks and my WAN is as follows...
x.x.83.20/29 with a gw x.x.83.17
I set up the WAN interface to a single uplink on ETH1as above for testing
My servers will need to be on the following network
x.x.86.1/25 with a gw of x.x.86.0 which has been statically routed to x.x.83.20
I set up the LAN interface as above on ETH2 but have been unable see the servers that are connected via a layer 2 switch to ETH2 and I can't connect to x.x.86.1 for the pfsense gui.
I couldn't get this to work after two attempts, so I have now removed the device from the data centre and brought it back to the office.
Any help greatly appreciated.
I still can't access x.x.86.1 from outside.
Have you added rules to allow that?
By default all inbound connections on WAN are blocked. Check the firewall logs Status > System Logs > Firewall
You have a laptop connected to Eth2. I assume that is still in the LAN at this point? You have not moved it to be the other WAN uplink yet?
Ah! Yes that's wrong.
If the pfSense LAN IP is x.x.86.1 then servers in the LAN subnet should be using x.x.86.1 as their gateway.
The LAN interface itself should not have a gateway defined on it.
Ok, there was no gateway defined - it was configured how you say in your most recent response.
You asked if I have set up firewall rules? No.
I couldn't see any rules defined so I just assumed I was starting with a blank slate. I hadn't got around to looking into the firewall yet - I just wanted to get this set up so I can start to configure it.
The documentation suggests that the deployment starts with setting up the WAN and LAN interfaces, then connecting to the GUI to continue configuring. I've done that but I can't connect to it. It doesn't say anything about the firewall. I guess it assumes you are always going to be on the LAN so you can connect to the LAN port at any time. I need to connect from outside so I guess it is different.
What do I need to do so I can connect to the public IP x.x.86.1 to login to the GUI?
From the LAN side you should be able to connect already there are default rules allowing that.
From the WAN side you need to add rules for any connection you want so to hit the pfSense webgui from WAN on the LAN IP (something you can do because it's a routable IP) add a new firewall rule on WAN like so:
Thanks, I'm a bit confused. For clarity, there is no actual LAN here - so unless I connect directly to the XG-7100 I have no access to this device.
Do I need to set up individual rules for individual ports on the WAN interface for access to the LAN interface?
I assume I've not been able to connect to the "LAN" IP's from outside, simply because there are no rules set up to allow that via the WAN firewall rules.
The point of this is to protect the so called "LAN" IP's and all traffic inwards has to come through the WAN first, so if for example, I want to connect to a server in the LAN network on port 53, Do I have to set the rule both on the WAN interface AND on the LAN interface? Or just on the WAN, with destination of "Lan Address".
This interface is not what I'm used to. In other software it's possible to provide a comma separated list of ingress and egress ports, rather than setting a rule for each one. Is that possible here?
WAN and LAN are just the default names there, you could use External and Internal equally. Because the subnet you are using is public that make less sense but it's just two interfaces.
You can just add an allow all rule and you could access everything but that's probably not what you want.
You can use Aliases in the firewall rules to simplify them. The Aliases can be lists of ports or IPs so you can have a rule that says 'pass traffic for destination "Server_Group1" to ports "Allowed_Ports3"` for example.
Those are configured in Firewall > Aliases.
You will probably want to disable outbound NAT in this configuration (in Firewall > NAT > Outbound) so that servers behind the firewall use their own IP for outbound traffic.
Still no joy with this at all.
I've disabled packet filtering altogether and I still can't connect to anything from outside. Doesn't make any sense.
@Chris-187 I think you may need to look at your interface assignments, specifically the VLAN and Switch settings. I had similar problems and when creating VLANS, you MUST ensure that you include members 9T and 10T. These are the Denverton Soc. Check the manual as the Switching implemented on this unit has a Switch LAGG where ix2 and ix3 (switch uplink ports 9 and 10), are configured as a load-balanced LAGG. This provides an aggregate uplinkcapable of 5Gbps for ethernet switchports ETH1-8. So you cannot disregard them as members of your VLAN configuration.
Under the Switch tab for VLANs in the Interface assignments, you must list the members of a VLAN. IF you omit to include 9T and 10T, it will NOT pass anything through.
Manual Page 6 and 7:
Check out an image of what I had to do (for a simpler setup) You see 9t and 10T in ALL assignments. If you miss this, it will NOT work...
@claferriere That's definitely not the problem. The VLAn's are set up exactly as @stephenw10 showed in his screenshots. All I did was remove eth2 from the LAN and add it to the WAN when setting it up for both uplinks.
I don't believe I have any settings wrong. I have configured the interfaces correctly and I've disabled the firewall but I can't access the LAN IP from outside. I wonder if the static routing hasn't been configured by the network team here. How can I test that?
Now that I've disabled the firewall shouldn't I at least be able to ping the WAN IP assigned to the pfsense?
Yes you should. Can you ping out from pfSense?
You can try running a packet capture on the WAN in pfSense whilst you ping it. You should see those pings arriving even if pfSense is unable to respond.
You might see ARP requests from the ISP if pfSense cannot respond to ARP.
Try running a traceroute to the LAN side subnet from somewhere external. You should see that go through your WAN subnet.
@stephenw10 I've had to leave it installed at the data centre this afternoon so I can't do anything directly on the pfsense now. Would be good to know what questions to ask the networks guy.
I don't get a response from ping on the main IP or the LAN IP and tracert starts timing out after about 9 hops.
If I tracert to the uplink gateway IP, it gets there in 5 hops If I try the main IP of the pfsense, there are a further 4 hops before it starts timing out. Should I expect to see the gateway IP in one of the hops, or doesn't it work like that?
Yeah I would certainly expect the next hop after the gateway to be the pfSense WAN IP.
That requires the gateway to be able to contact the WAN IP though. Did the gateway status show as up in pfSense? That would indicate it can ping the gateway.
What actually is the next hop? There could be some loop, you sometimes see that if something redirects.
Sorry - what I'm saying is that I can't see the 83.17 gateway in the tracert. Should I see that?
Ah, OK then that does seem like a routing problem. I'd expect to see that there even if pfSense is not connected to it.