Blocked my internal lan but I dont see where

[poshpaws]

Howdy Firewall noob and i seem to have filterd most of my internal lan but cant see where
Logs are showing me I have a block on most internal traffic , except i cant seem to find where I set it .

Oct 25 12:26:35 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:36 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:37 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:38 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:39 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:40 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:41 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:43 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:46 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:50 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:26:50 redgate filterlog: Oct 25 12:26:54 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale
Oct 25 12:27:02 redgate filterlog: 5,16777216,,1000000103,igb0_vlan1,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,192.168.1.230,192.168.1.109,443,54584,0,SA,526202898,1087202650,28960,,mss;sackOK;TS;nop;wscale

some topology :

WAN - DHCP    igb2 int
LAN - vlan tagged 1    igb0 int    192.168.1/24
LABNET - vlan tagged 4 igb0 int  192.168.4/24

What info do you fellows need to help me out?
labnet rules and lan rules attached


[KOM]

You have Allow All rules on both LAN and LABNET, so nothing should be getting blocked.  It is likely just out of state traffic.

https://doc.pfsense.org/index.php/Why_do_my_logs_show_%22blocked%22_for_traffic_from_a_legitimate_connection

Are you having any connectivity problems, or you just saw the log data and wondered what was going on?

[poshpaws]

Yes Traffic is getting blocked , ssh , web etc all seem to be blocked by the default deny rule (1000000103)
but as far as i can tell should not be.

/tmp/rules.debug follows , maybe someone spots something daft …

set limit table-entries 2000000
set optimization normal
set limit states 404000
set limit src-nodes 404000

#System aliases

loopback = "{ lo0 }"
WAN = "{ igb2 }"
LAN = "{ igb0_vlan1 }"
LABNET = "{ igb0_vlan4 }"
WLAN = "{ ath0_wlan0 }"
OpenVPN = "{ openvpn }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
table <vpn_networks>{ 10.8.0.0/24 }
table <negate_networks>{ 10.8.0.0/24 }

User Aliases

table <pfb_top_v4>persist file "/var/db/aliastables/pfB_Top_v4.txt"
pfB_Top_v4 = "<pfb_top_v4>"

Gateways

GWWAN_DHCP = " route-to ( igb2 178.83.128.1 ) "
GWWAN_DHCP6 = " route-to ( igb2 178.83.128.1 ) "
GWLAN_DHCP6 = "  "

set loginterface igb0_vlan1

set skip on pfsync0

scrub on $WAN all    fragment reassemble
scrub on $LAN all    fragment reassemble
scrub on $LABNET all    fragment reassemble
scrub on $WLAN all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/"
nat-anchor "natrules/"

Outbound NAT rules (automatic)

Subnets to NAT

tonatsubnets = "{ 127.0.0.0/8 192.168.1.0/24 192.168.0.0/24 10.8.0.0/24 }"
nat on $WAN  from $tonatsubnets to any port 500 -> 178.83.131.6/32  static-port
nat on $WAN  from $tonatsubnets to any -> 178.83.131.6/32 port 1024:65535

Load balancing anchor

rdr-anchor "relayd/*"

TFTP proxy

rdr-anchor "tftp-proxy/*"

UPnPd rdr anchor

rdr-anchor "miniupnpd"

anchor "relayd/"
anchor "openvpn/"
anchor "ipsec/*"

block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,

and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but

route-to can override that, causing problems such as in redmine #2073

block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
#---------------------------------------------------------------------------

default deny rules

#---------------------------------------------------------------------------
block in log inet all tracker 1000000103 label "Default deny rule IPv4"
block out log inet all tracker 1000000104 label "Default deny rule IPv4"
block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"

IPv6 ICMP is not auxilary, it is required for operation

See man icmp6(4)

1    unreach        Destination unreachable

2    toobig          Packet too big

128  echoreq        Echo service request

129  echorep        Echo service reply

133  routersol      Router solicitation

134  routeradv      Router advertisement

135  neighbrsol      Neighbor solicitation

136  neighbradv      Neighbor advertisement

pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state

Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state

We use the mighty pf, we cannot be fooled.

block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"

Snort package

block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"

SSH lockout

block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"

webConfigurator lockout

block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"

allow our DHCPv6 client out to the WAN

pass in  quick on $WAN proto udp from fe80::/10 port = 546 to fe80::/10 port = 546 tracker 1000000561 label "allow dhcpv6 client in WAN"
pass in  quick on $WAN proto udp from any port = 547 to any port = 546 tracker 1000000562 label "allow dhcpv6 client in WAN"
pass out  quick on $WAN proto udp from any port = 546 to any port = 547 tracker 1000000563 label "allow dhcpv6 client out WAN"
antispoof log for $WAN tracker 1000001570

allow our DHCP client out to the WAN

pass in  on $WAN proto udp from any port = 67 to any port = 68 tracker 1000001591 label "allow dhcp client out WAN"
pass out  on $WAN proto udp from any port = 68 to any port = 67 tracker 1000001592 label "allow dhcp client out WAN"

Not installing DHCP server firewall rules for WAN which is configured for DHCP.

allow our DHCPv6 client out to the LAN

pass in  quick on $LAN proto udp from fe80::/10 port = 546 to fe80::/10 port = 546 tracker 1000001611 label "allow dhcpv6 client in LAN"
pass in  quick on $LAN proto udp from any port = 547 to any port = 546 tracker 1000001612 label "allow dhcpv6 client in LAN"
pass out  quick on $LAN proto udp from any port = 546 to any port = 547 tracker 1000001613 label "allow dhcpv6 client out LAN"

allow access to DHCP server on LAN

pass in  quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 tracker 1000002641 label "allow access to DHCP server"
pass in  quick on $LAN proto udp from any port = 68 to 192.168.1.1 port = 67 tracker 1000002642 label "allow access to DHCP server"
pass out  quick on $LAN proto udp from 192.168.1.1 port = 67 to any port = 68 tracker 1000002643 label "allow access to DHCP server"
antispoof log for $LABNET tracker 1000003670

allow access to DHCP server on LABNET

pass in  quick on $LABNET proto udp from any port = 68 to 255.255.255.255 port = 67 tracker 1000003691 label "allow access to DHCP server"
pass in  quick on $LABNET proto udp from any port = 68 to 192.168.0.1 port = 67 tracker 1000003692 label "allow access to DHCP server"
pass out  quick on $LABNET proto udp from 192.168.0.1 port = 67 to any port = 68 tracker 1000003693 label "allow access to DHCP server"

loopback

pass in  on $loopback inet all tracker 1000005811 label "pass IPv4 loopback"
pass out  on $loopback inet all tracker 1000005812 label "pass IPv4 loopback"
pass in  on $loopback inet6 all tracker 1000005813 label "pass IPv6 loopback"
pass out  on $loopback inet6 all tracker 1000005814 label "pass IPv6 loopback"

let out anything from the firewall host itself and decrypted IPsec traffic

pass out  inet all keep state allow-opts tracker 1000005815 label "let out anything IPv4 from firewall host itself"
pass out  inet6 all keep state allow-opts tracker 1000005816 label "let out anything IPv6 from firewall host itself"
pass out  route-to ( igb2 178.83.128.1 ) from 178.83.131.6 to !178.83.128.0/21 tracker 1000005911 keep state allow-opts label "let out anything from firewall host itself"

make sure the user cannot lock himself out of the webConfigurator or SSH

pass in  quick on igb0_vlan1 proto tcp from any to (igb0_vlan1) port { 80 22 } tracker 10000 keep state label "anti-lockout rule"

User-defined rules follow

anchor "userrules/*"
pass  in  quick  on $OpenVPN  from any to any keep state  label "USER_RULE: OpenVPN redgate wizard"
block  in log  quick  on $WAN reply-to ( igb2 178.83.128.1 ) inet from $pfB_Top_v4 to any tracker 1770009106  label "USER_RULE: pfB_Top_v4 auto rule"
pass  in  quick  on $WAN reply-to ( igb2 178.83.128.1 )  proto udp  from any to 178.83.131.6 port 1194 keep state  label "USER_RULE: OpenVPN redgate wizard"
pass  in  quick  on $LAN inet from 192.168.1.0/24 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"

at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"

pass  in  quick  on $LABNET inet from any to any tracker 1476796414 keep state  label "USER_RULE"

VPN Rules

anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></pfb_top_v4></pfb_top_v4></negate_networks></vpn_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>

[KOM]

Disable any IDS/IPS packages like Snort, Suricata, pfBlocker and try again.  Any time there is weirdness that belies the rules, I make sure these specific packages are disabled.

btw it's much easier for us to read screenshots of your rules & logs than the raw dumps.

[poshpaws]

Hi , all IDS tools were uninstalled and the device rebooted once or twice since so there should be nothing held over from pfblockerng install

• and thanks for the screenshot tip :)

[KOM]

I'm not sure why it's blocking traffic from 192.168.1.230 to 192.168.1.109 since it's local and shouldn't even hit the firewall.  The client and server are on the same VLAN?  There isn't something weird going on with your setup?

[poshpaws]

So it appears I must redesign my network somewhat as i am getting some asymmetric routing , which is why i guess pfsense blocks it.

:-[

[johnpoz]

" as i am getting some asymmetric routing"

How is that?  Do you have downstream routers without using transit networks?

[poshpaws]

No , no more downstream routers left

ISP===> cablemodem ig2 => pfsense appliance ig1 =>    cisco 3750 (vlans tagged on switch , no routing)  vlan 3= > esx host
                                                                  ig0 =>                                                                          vlan 4=> esx host

gateway is set to be the pf box.
anything that does not go near the esx seems to be fine , direct subnets no  issues , vm to vm traffic gets blocked same vlan or not

if i pcap my outgoing lan networks , I see packets from my esx vm's , these are then blocked by  default rules.

So slight change , I do see where it's getting blocked , I have yet to figure out why ….

[johnpoz]

Well if no downstream routing how would you be getting asymmetrical.. How you would you possible have more than 1 route to somewhere?  Do you have multihomed devices that have nic in more than 1 vlan?

"vm to vm traffic gets blocked same vlan or not"

How exactly would traffic on the same vlan get blocked?  Vlan traffic to same vlan has ZERO to do with pfsense.  If I had to guess your prob running multiple layer 3 over the same layer 2, which is just BORKED, etc..

Do a real drawing of your network - break out the crayons if you have to and label what networks/vlans and what is plugged into what at layer 1.  And your switch config for these ports..

[poshpaws]

Ok point taken about the diagram , attached should be a slight improvement

I would largely agree traffic on the same vlan should have zero to do with pfsense , yet the logs show the block and the traffic is well .. blocked.

[Derelict]

Your issue is in ESXi somewhere.

Traffic between 192.168.0.3 and 192.168.0.230 should all be handled in the vswitch and should never be sent to the router.

Your problem is not in the firewall.

[johnpoz]

if you have devices in esxi on the same vswitch that can not talk to each other and your seeing this traffic go to pfsense, my guess would be you have a mask wrong somewhere on one of the vms.  So the 1 vm sends him syn and he answers back syn,ack to pfsense because from his mask the IP talking to him is on a different network.