Configuring OPT3

m3tatr0n · Sep 22, 2016, 2:51 PM

This should be dead simple but for some reason I can not seem to get any packets in and out of OPT3.
I have 2 Netgate SG-4860 (CARP). I already have WAN, LAN and OPT1 working. We added another
network segment so I used OPT3 on the primary and gave it IP address 192.168.30.252
and OPT3 on the secondary with IP address 192.168.30.253. Eventually, 192.168.30.254 will be the
VIP CARP. As an initial test, I tried to ping both 192.168.30.252 and 253, none responded to ping.
Interface OPT3 on the PFsense Dashboard is up.
I can ping, ssh, https the ip address of both LAN and OPT1 interfaces but not OPT3.
I ssh'ed to the pfsense box via one of the OPT interfaces and did a ping to and ssh to one of the
servers on the 192.168.30.0 network and can not connect.

Thanks

KOM · Sep 22, 2016, 3:23 PM

You didn't explicitly say if you added the required firewall rule for OPT3. Only LAN gets one, and all other interfaces need one to be able to talk.

johnpoz · Sep 22, 2016, 3:26 PM

While I agree your going to need rules, that should not stop pfsense from pinging something in that network, be there are rules or not on opt3

How exactly are your devices connected to pfsense that are in opt3.. Does pfsense see the mac of what your trying to ping in that network? If not then you have a layer1/2 connectivity issue.

KOM · Sep 22, 2016, 3:33 PM

Without seeing his firewall rules, it's hard to know what exactly is going on.

johnpoz · Sep 22, 2016, 3:43 PM

very true. but the firewall rules for opt3 wouldn't stop pfsense from pinging something on opt3 network. If he had something in floating blocking on opt3 then sure that could be the problem.

While I agree no firewalls rules on opt3 could stop client in opt3 network from pinging pfsense IP in that network. I doesn't stop pfsense from pining into the network from its interface in opt3. If he can not ping, then either clients blocking it not answering. He has a mask issue on this network between clients and pfsense. Or he has some sort of connectivity issue be it at layer 1 or 2.

If he can arp then points to firewall issue on the client he is trying to ping from pfsense. If can not arp then points to connectivity issue at layer1/2

m3tatr0n · Sep 22, 2016, 3:44 PM

@KOM:

You didn't explicitly say if you added the required firewall rule for OPT3. Only LAN gets one, and all other interfaces need one to be able to talk.

Only firewall rule I added is to allow interface OPT2 using any protocol with the source OPT2 net to access any destination.

Thanks

m3tatr0n · Sep 22, 2016, 3:58 PM

@johnpoz:

very true. but the firewall rules for opt3 wouldn't stop pfsense from pinging something on opt3 network. If he had something in floating blocking on opt3 then sure that could be the problem.

While I agree no firewalls rules on opt3 could stop client in opt3 network from pinging pfsense IP in that network. I doesn't stop pfsense from pining into the network from its interface in opt3. If he can not ping, then either clients blocking it not answering. He has a mask issue on this network between clients and pfsense. Or he has some sort of connectivity issue be it at layer 1 or 2.

If he can arp then points to firewall issue on the client he is trying to ping from pfsense. If can not arp then points to connectivity issue at layer1/2

I have two servers on the network segment that can ping each other. I tried both opt2 and opt3 on pfsense. I tried connecting all servers and pfsense in a different switches. Only arp entry I see on the pfsense is it's own mac address for opt3.

P.S.
in previous post I said OPT3 and other post OPT2… sorry for the confusion... it's because I have OPT2 and OPT3 that free that I am working on.
Nonetheless I can't get either to work.

johnpoz · Sep 22, 2016, 3:56 PM

Well if your going to want opt3 devices to do anything, ping pfsense - go to opt2 or lan or internet then you would need to put those rules there. But again that does not stop pfsense from pinging into opt3 network. If you can not ping stuff in opt3 from pfsense then you have something else going on other than just lack of rules on opt3

m3tatr0n · Sep 22, 2016, 4:04 PM

@johnpoz:

Well if your going to want opt3 devices to do anything, ping pfsense - go to opt2 or lan or internet then you would need to put those rules there. But again that does not stop pfsense from pinging into opt3 network. If you can not ping stuff in opt3 from pfsense then you have something else going on other than just lack of rules on opt3

There will be rules added to this interface but for now I just want servers on that network to be able to at least go out through the WAN.
This network will be totally separated from our LAN and the network on OPT1. Only remote client will access this network via OpenVPN and HTTP/S.

johnpoz · Sep 22, 2016, 4:21 PM

well how are you connecting these interfaces to your network? What vlan are they in, are they just dumb switches that you have isolated for each network?

m3tatr0n · Sep 22, 2016, 4:39 PM

@johnpoz:

well how are you connecting these interfaces to your network? What vlan are they in, are they just dumb switches that you have isolated for each network?

Yes, dumb switches. Just wanted it totally separated from the other segments.

johnpoz · Sep 22, 2016, 4:56 PM

that is fine and rules out issues with vlan config on your switches.

So your saying you have this

pfsense opt3 - dumbswitch - pc
psense lan - different dumbswitch - different pc

Both with network 192.168.30/24, lets forget the whole carp setup for a bit. You configure pfsense opt3 with 192.168.30.1/24 and your pc with 192.168.30.100/24

If pfsense can not see mac in his arp table when you try and ping 192.168.30.100 then you got bad cable, bad switch, interface not actually working, etc.

PC firewall might block you from getting answer, but its not going to block the arp for the mac. Doing the same thing from pc to 192.168.30.1 again if you don't have rules on opt3 to allow ping you wont get an answer but you should see the mac of pfsense opt3 interface.

your pfsense is on a physical device right, its not VM running in some vm host??

m3tatr0n · Sep 22, 2016, 5:33 PM

@johnpoz:

that is fine and rules out issues with vlan config on your switches.

So your saying you have this

pfsense opt3 - dumbswitch - pc
psense lan - different dumbswitch - different pc

Both with network 192.168.30/24, lets forget the whole carp setup for a bit. You configure pfsense opt3 with 192.168.30.1/24 and your pc with 192.168.30.100/24

If pfsense can not see mac in his arp table when you try and ping 192.168.30.100 then you got bad cable, bad switch, interface not actually working, etc.

PC firewall might block you from getting answer, but its not going to block the arp for the mac. Doing the same thing from pc to 192.168.30.1 again if you don't have rules on opt3 to allow ping you wont get an answer but you should see the mac of pfsense opt3 interface.

your pfsense is on a physical device right, its not VM running in some vm host??

PFSense is a physical device Netgate SG-4860
Before configuring OPT3, the WAN, LAN and OPT1 are already working, all are configured for CARP.

OPT3 ip address is 192.168.30.252
Server1 ip address is 192.168.30.10
Server2 ip address is 192.168.30.11

Only firewall rule is to allow OPT3 interface, any protocol, with source OPT3 net to allow connection to any.
server1 can ping server2 and vice versa.

Steps taken so far:

Use another dumb switch
Use OPT2 instead of OPT3
Use different network cables.

johnpoz · Sep 22, 2016, 7:49 PM

You sure you have the mask right, pfsense defaults to /32 when you create a new IP.. Sure you didn't put /32 vs /24??

m3tatr0n · Sep 22, 2016, 8:04 PM

@johnpoz:

You sure you have the mask right, pfsense defaults to /32 when you create a new IP.. Sure you didn't put /32 vs /24??

Holy crap batman! That was it, the mask was set to 32 instead of 24.

Thanks a lot

johnpoz · Sep 22, 2016, 8:18 PM

Dude I brought that up much earlier in the thread.. ;)

" If he can not ping, then either clients blocking it not answering. He has a mask issue on this network between clients and pfsense. Or he has some sort of connectivity issue be it at layer 1 or 2."

Glad you got it sorted..

m3tatr0n · Sep 22, 2016, 10:00 PM

@johnpoz:

Dude I brought that up much earlier in the thread.. ;)

" If he can not ping, then either clients blocking it not answering. He has a mask issue on this network between clients and pfsense. Or he has some sort of connectivity issue be it at layer 1 or 2."

Glad you got it sorted..

I admit I am kinda overwhelmed with other stuff here, wearing too many hats ;)

Thanks so much for helping out.