Need Help Bridging Vlans
-
Hello Everyone! ;D
This is my first post, but I am not new to the forum. I have read, and read, and read and have found the information in the Forums very, very useful. However, I have encountered a problem that I just can't figure out on my own. I HAVE searched the forums for some kind of information that would help me, but I have not been successful.I should start by saying that I am networking and firewall novice. The purpose of installing pfSense was to explore and learn.
I am running:
2.1-RELEASE (i386)
built on Wed Sep 11 18:16:22 EDT 2013
FreeBSD 8.3-RELEASE-p11Harware info: Re-purposed MSA security appliance (manufactured by Celestix) with 6 x Intel Pro 1000 NICs, Intel(R) Celeron(R) CPU 2.80GHz, 2GB of memory. pfSense is installed on a 4GB CF card in an on-board slot.
The Setup: since a picture is worth a thousand words, and I have trouble with verbosity, I will (hopefully) attach an image that shows the setup, rather than trying to write it all out [EDIT: apparently I can't attach/upload images, so here goes nothing…]
lagg0 = em2, em3Interface VLAN tag Description
lagg0 10 DMZ Server Management
lagg0 80 DMZ Web Server Machine
em1 10 VLAN 10 Tagged on LAN InterfaceInterface Network port
WAN em0
LAN em1
DMZ VLAN 80 on lagg0
DMZMGMT VLAN 10 on lagg0
LAN_VLAN10 VLAN 10 on em1BRIDGE0 WAN, DMZ Bridge WAN to DMZ
BRIDGE1 LAN, DMZMGMT LAN TO DMZ MGMTThe Problem:
I have another box, also a repurposed machine: an RSA Securid box with similar specs to the MSA appliance (also manufactured by Celestix), but with 2 Intel Pro 1000 NICs. I have Ubuntu 12.04 LTS installed on the RSA box. The 2 NICs on the RSA host box are bonded. I have two vlan interfaces on the bond: bond0.10 and bond0.80. bond0.10 has IP address 10.10.0.66 in the same subnet as the LAN. bond0.80 has no IP address. I have VirtualBox installed on the host and another Ubuntu 12.04 server installed as a virtual machine. The VirtualBox NIC is bridged to bond0.80. The virtual machine has an IP address in the same subnet as the DMZ interface on the pfSense box.On the pfSense box, I have a DMZ interface assigned to vlan 80 with an IP in the same subnet. The virtual machine can resolve names on the internet, access the internet for updates, etc., and per the firewall rules I have set up on that interface, ping the primary DNS server (WIN2K12 hyper-v guest on LAN) and secondary DNS server (pfSense LAN IP).
I have tried bridging the LAN and DMZMGMT interfaces. If I ping the RSA host machine IP from the pfSense box, I get a response. If I ping the pfSense LAN IP from the RSA host machine, I get a reply, but ONLY if I ping it from pfSense first, and only for a short period of time. I can also ping out to the internet during this time. But, after some time has passed, the connection disappears, and ping no longer works to any destination. At no time can I ping any other machine on the LAN or the DMZ from the RAS box from DMZMGMT.
Here are the LAN firewall rules:
Proto Source Port Destination Port Gateway Queue Schedule Description-
-
- LAN Address 443 * * Anti-Lockout Rule
80
22
IPv4 * * * DMZMGMT net * * none LAN to DMZMGMT ANY
IPv4 * DMZMGMT net * LAN net * * none Allow DMZMGT to LAN
IPv4 * LAN net * * * * none Default allow LAN to any rule
- LAN Address 443 * * Anti-Lockout Rule
-
Here are the DMZMGMT firewall rules:
Proto Source Port Destination Port Gateway Queue Schedule DescriptionIPv4 ICMP DMZMGMT net * 10.10.0.1 * * none DMZMGMT PING to Secondary DNS Server
IPv4 ICMP DMZMGMT net * 10.10.0.20 * * none DMZMGMT PING to Primary DNS Server
IPv4 UDP DMZMGMT net * 10.10.0.1 53 (DNS) * none DMZMGMT to Secondary DNS Server
IPv4 UDP DMZMGMT net * 10.10.0.20 53 (DNS) * none DMZMGMT to Primary DNS Server
IPv4 * DMZMGMT net * * * * none Allow DMZMGMT to any ruleI have tried changing BRIDGE1 to connect LAN_VLAN10, DMZMGMT and then changing the LAN interface to BRIDGE1, then rebooting, etc. with no success. Also, every time I change the bridge(s) that involve the lagg/vlans I get the message: bridge(#): error setting interface capabilities on lagg0_vlan##.
I'm really at a loss as to how to resolve this. Any help or input would be greatly appreciated, and please be gentle. : ) Thanks.
-
-
I found the option to add attachments!
Hopefully this will help generate a view or two? Maybe a response?
I really need some insight, a point in the right direction, whatever…Please take a look at the illustration. Maybe it will help clarify where my problem lies?
Thanks!
-
One more piece of information that I did not mention- If I add an IP address to the DMZMGMT interface in the current configuration (LAN interface on em1, not vlan_10 on em1, and DMZGMT interface assigned to vlan_10 on lagg0), I can ping all back and forth between the host RSA machine in the DMZMGMT zone and all of the machines in the LAN zone.
-
Ok. I take it back. No need to be gentle. ANY input is appreciated. If you tell me I'm an idiot and I can't read, or whatever, at least I have someplace to start!
I should also say that I have not JUST read the forums- I have also read the Wiki/FAQ/Tutorials. While they do touch on the subjects of Multi-LAN, VLANS, etc., I don't see anything that specifically addresses bridged vlans over a LAGG to connect to LAN segments.
I'm confused and don't know where to go next! :'( Wah!!! I'm starting to feel like I'm whining like a baby. :D
-
This seems like you've deliberately setup something complex. Like you're trying to use all the features for experimental purposes. That's fine if you are, I have a box setup like that for testing, but if you don't say so people are going to read this and either not want to get involved in something so complex or suggest a simpler setup.
Steve
-
stephenw10, thanks very much for your response. You are correct, I have deliberately set up something complex. I did say that I am using pfSense to explore and learn. But, the implementation that I am testing, does have roots in practical considerations- though they may be unneccessary, or there may be a different, better (or simpler) set up that addresses the same concerns, which I am too ignorant to know of.
First, I chose to use the lagg between the pfbox and the DMZ (host) server for failover. I wanted to make use of LACP (803.2ad, I think) failover, so that one connection could jump in and take over if the other fails. My thinking being that if this were a critical connection, say to a web server, this would avoid one potential point for down time.
At the same time, I wanted to be able to have the DMZ server virtualized, so that it could be easily restored from a snapshot in the event it was compromised. Fortunately, the DMZ setup seems to work properly.
With that, I wanted to have a separate connection to the host machine that would easily allow the transfer of files, communication, etc. between the machines on the LAN and the host machine. I also really want to use and explore vlans, so this seemed like the perfect place/opportunity to implement separate vlans for the DMZ and DMZMGMT zones.
And this is probably where my lack of networking knowledge shows most. In my mind, I should be able to bridge the DMZMGMT and LAN segments to, in effect, make them two parts of the same network to accomplish the stated goal above using vlans over the lagg connection. I mean, it works to bridge the DMZ and WAN connections with an IP address on the DMZ interface and none on the host machine NIC (on the vlan 80 interface for the DMZ, also bridged, to the NIC on the virtual machine, with it's own IP). Given that it seems it should work in reverse, with an IP on the host NIC on the same subnet as the LAN, no IP on the DMZMGMT interface, bridged to the LAN interface, and the LAN IP as the gateway for that machine.
I realize that I could use a different subnet on the DMZMGMT segment, set an IP address on that interface, use it as the default gateway for the host machine, and create rules to pass traffic from the host machine back to the LAN. And really, I'm not so ignorant to know that, probably this is the most secure means of implementing the setup that I am proposing, following the principles that a) an edge server, like the machine hosting the DMZ virtual machines, should probably not be in the same subnet as the rest of the LAN, and b) the most secure firewall/network setup only allows the minimum number/type of connections back to the LAN network necessary to fulfill it's purpose.
The problem is that I am super stubborn, and feel like, if it should work the way I have it currently set up and it doesn't, I want to make it work, or learn why it can't or won't work that way. Once I know I can make it work, or know I can't, I can move on to implementing another solution that (most likely) makes more sense. (Did I mention I'm stubborn?)
I guess the motivation here is the key- I want to learn. If I just give up and walk away from this setup (which really seems to me like it should work, then I won't have learned why it didn't. That, and I'm really stubborn. I mentioned that, right? I'm pretty sure I mentioned I'm verbose as well… :D Again, I really appreciate your response, and any further input feedback you may have.
-
OK, just read through that about 5 times and I think I have some grasp of your setup. ::)
Some questions:
What have you made lagg0 from in the pfSense box? I assume em2 and em3 (or similar)
What is VLAN_10 on em1 doing? It seems to serve no purpose.In my limited VLAN dealing I've never tried using the same VLAN tag on more than one pfSense interface. I don't why it shouldn't work but also I could imagine confusion somewhere so I would try either removing the VLAN interface from em1 or changing the tag to something not used elsewhere.
Why are you trying to bridge the LAN and DMZ Server Management interfaces? Do they need to be in the same subnet? It seems like two groups that would be better in different subnets.
If I add an IP address to the DMZMGMT interface in the current configuration (LAN interface on em1, not vlan_10 on em1, and DMZGMT interface assigned to vlan_10 on lagg0), I can ping all…
Not quite sure what you mean here. You add an IP the DMZMGMT interface and can then ping anything in LAN?
In your diagram you have labelled LAN GW. Does that mean you have a gateway on the LAN interface?
Your description of being able to ping and then not sounds like a dynamic routing issue or maybe a firewall state issue. Do you see anything in the firewall logs when the ping fails?
Steve
-
stephenw10, thanks, again for your response. I'll try to answer your questions below:
What have you made lagg0 from in the pfSense box? I assume em2 and em3
That is correct- em2 and em3.
What is VLAN_10 on em1 doing?
Nothing, at the moment.
First I tried adding vlan 10 tag to em1 (which created the VLAN_10_on_em1 network port), then I assigned the LAN interface to the VLAN_10_on_em1 port, instead of em1. But, when I did that I lost connection to the pf box.
Next, I tried creating a new interface: LAN_VLAN10 and assigning that to the VLAN_10_on_em1 network port. After that, I modified bridge1 to bridge LAN_VLAN10 and DMZMGMT. Once I did that, I assigned the LAN interface to bridge1. That didn't work either.
Why are you trying to bridge the LAN and DMZ Server Management interfaces? Do they need to be in the same subnet? It seems like two groups that would be better in different subnets.
I think I agree with you now. That is something that I have learned from this experiment. But, it doesn't answer why bridging two interfaces on the same subnet doesn't work. And I would like to know why it doesn't work, and, if possible learn how to make it work.
Not quite sure what you mean here. You add an IP the DMZMGMT interface and can then ping anything in LAN?
DMZMGMT interface has an IPv4 configuration of "none" There is no IP address assigned to the interface. If I add an IP address in the same subnet as the LAN interface (such as 10.10.0.50), the machine behind the VLAN_10_on_lagg0 network port can ping any machine in the LAN subnet (the machine is in the same subnet: 10.10.0.66).
In your diagram you have labelled LAN GW. Does that mean you have a gateway on the LAN interface?
No. I did not create any gateways. I only meant that the IP address assigned to the LAN gateway is handed out to machines on the LAN subnet as the default gateway.
Your description of being able to ping and then not sounds like a dynamic routing issue or maybe a firewall state issue. Do you see anything in the firewall logs when the ping fails?
Honestly, I've looked at the logs, but I don't always understand what I see there, and it's been a while since I did all this. I don't remember the result. I'll test it again as soon as I have the time. Work has been very busy and it was a struggle to get back here to write this response. I did however, also think (from some learning I did while researching!) that there might be a problem with the states, so I did clear the state several times during all of this reconfiguring I did, but it did not help. Perhaps this or some of the information I provided above will help in some way.
Anyway, thanks again for taking the time to read all my babbling and for trying to help. I really appreciate it. You're a very nice person to try to help.
-
Ok, that all makes sense, thanks for the concise replies. :)
I agree that even if a thing is not the best way go about something it's nice to know why it doesn't work. Doing so helps avoid making the same mistakes in the future.
I'm not sure why you can't bridge those two interfaces. You should be able to bridge any interfaces in pfSense it makes no difference what type of interface it is. That said some things are not bridgable, PPPoE interfaces for example. Also some combinations of interface type can give trouble if hardware offloading is used, there is no hardware to off load to on a LAGG interface. To work as a bridge the interface must be placed in promiscuous mode in order to pass all packets on the wire (or virtual wire), perhaps something is failing to enter promiscuous mode?
Normally in a set or bridged interfaces you would only have one that has an IP, that can be the bridge interface itself. The fact that you are seeing better traffic when you add an IP to a 2nd bridge member is odd. Is that traffic actually going via the bridge?
The fact that traffic starts and stops is probably a clue. That can often show when there is some subnet overlap between two (or more) interfaces and routing or packets becomes a bit hit and miss.
You can probably tell I'm guessing here. ;)
Steve
-
So, I'm playing around with this again today and decide to look at the logs again for more helpful information. To do that, I log into the DMZMGMT machine via SSH from a machine in LAN. When I login, the Ubuntu control panel tells me that I packages available for update, which implies to me that the server has contacted the repo servers to find this information out- odd because I cannot ping anything outside of the machines own IP address. So, i run apt-get update and it downloads the list of packages ready for update! Ok. Even more odd. So, I decide to run apt-get upgrade to see what happens. Lo!, and behold, it starts downloading and installing package updates. UNTIL- it suddenly stops. So I go into the logs and can see that DNS and DHCP traffice (UDP 53 and 138- see attached screenshots from firewall log) are clearly being blocked. Even though I have the rules in place (below) to pass at least the DNS traffic!??? (Though the last rule, should allow any connection as well, right?)
Here are the DMZMGMT firewall rules:
Proto Source Port Destination Port Gateway Queue Schedule DescriptionIPv4 ICMP DMZMGMT net * 10.10.0.1 * * none DMZMGMT PING to Secondary DNS Server
IPv4 ICMP DMZMGMT net * 10.10.0.20 * * none DMZMGMT PING to Primary DNS Server
IPv4 UDP DMZMGMT net * 10.10.0.1 53 (DNS) * none DMZMGMT to Secondary DNS Server
IPv4 UDP DMZMGMT net * 10.10.0.20 53 (DNS) * none DMZMGMT to Primary DNS Server
IPv4 * DMZMGMT net * * * * none Allow DMZMGMT to any ruleSo, I also see some very odd entries in the system log that are making me start to wonder if there is just something really wrong with my hardware. It seems like the link states keep going up and down prompting the system to keep running rc.newwanip, which results in errors.
Jan 18 13:26:57 php: rc.interfaces_wan_configure: The command '/sbin/ifconfig 'lagg0_vlan10' inet delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address' Jan 18 13:26:54 check_reload_status: Reloading filter Jan 18 13:26:54 php: rc.newwanip: rc.newwanip: on (IP address: 10.10.80.1) (interface: opt1) (real interface: lagg0_vlan80). Jan 18 13:26:54 check_reload_status: Configuring interface opt2 Jan 18 13:26:54 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan80. Jan 18 13:26:54 php: rc.newwanip: rc.newwanip: Failed to update opt2 IP, restarting... Jan 18 13:26:54 php: rc.newwanip: rc.newwanip: on (IP address: ) (interface: opt2) (real interface: lagg0_vlan10). Jan 18 13:26:54 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan10. Jan 18 13:26:51 check_reload_status: rc.newwanip starting lagg0_vlan80 Jan 18 13:26:51 check_reload_status: rc.newwanip starting lagg0_vlan10 Jan 18 13:26:51 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:51 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:49 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:49 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:48 kernel: em3: link state changed to UP Jan 18 13:26:48 check_reload_status: Linkup starting em3 Jan 18 13:26:47 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:47 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:47 check_reload_status: Linkup starting lagg0 Jan 18 13:26:47 check_reload_status: Linkup starting em2 Jan 18 13:26:47 kernel: lagg0_vlan10: link state changed to UP Jan 18 13:26:47 kernel: lagg0_vlan80: link state changed to UP Jan 18 13:26:47 kernel: lagg0: link state changed to UP Jan 18 13:26:47 kernel: em2: link state changed to UP Jan 18 13:26:46 php: rc.interfaces_wan_configure: The command '/sbin/ifconfig 'lagg0_vlan10' inet delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address' Jan 18 13:26:45 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:45 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:45 check_reload_status: Linkup starting lagg0 Jan 18 13:26:45 kernel: lagg0_vlan10: link state changed to DOWN Jan 18 13:26:45 kernel: lagg0_vlan80: link state changed to DOWN Jan 18 13:26:45 kernel: lagg0: link state changed to DOWN Jan 18 13:26:45 kernel: em3: link state changed to DOWN Jan 18 13:26:45 check_reload_status: Linkup starting em3 Jan 18 13:26:45 kernel: em2: link state changed to DOWN Jan 18 13:26:45 check_reload_status: Linkup starting em2 Jan 18 13:26:44 check_reload_status: Configuring interface opt2 Jan 18 13:26:43 php: rc.newwanip: rc.newwanip: Failed to update opt2 IP, restarting... Jan 18 13:26:43 check_reload_status: Reloading filter Jan 18 13:26:43 php: rc.newwanip: rc.newwanip: on (IP address: ) (interface: opt2) (real interface: lagg0_vlan10). Jan 18 13:26:43 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan10. Jan 18 13:26:43 php: rc.newwanip: rc.newwanip: on (IP address: 10.10.80.1) (interface: opt1) (real interface: lagg0_vlan80). Jan 18 13:26:43 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan80. Jan 18 13:26:40 check_reload_status: rc.newwanip starting lagg0_vlan80 Jan 18 13:26:40 check_reload_status: rc.newwanip starting lagg0_vlan10 Jan 18 13:26:40 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:40 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:38 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:38 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:37 check_reload_status: Linkup starting em2 Jan 18 13:26:37 kernel: em2: link state changed to UP Jan 18 13:26:37 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:37 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:37 check_reload_status: Linkup starting lagg0 Jan 18 13:26:37 kernel: lagg0_vlan10: link state changed to UP Jan 18 13:26:37 kernel: lagg0_vlan80: link state changed to UP Jan 18 13:26:37 kernel: lagg0: link state changed to UP Jan 18 13:26:37 kernel: em3: link state changed to UP Jan 18 13:26:37 check_reload_status: Linkup starting em3 Jan 18 13:26:35 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:35 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:35 check_reload_status: Linkup starting lagg0 Jan 18 13:26:35 kernel: lagg0_vlan10: link state changed to DOWN Jan 18 13:26:35 kernel: lagg0_vlan80: link state changed to DOWN Jan 18 13:26:35 kernel: lagg0: link state changed to DOWN Jan 18 13:26:35 kernel: em2: link state changed to DOWN Jan 18 13:26:35 check_reload_status: Linkup starting em2 Jan 18 13:26:34 kernel: em3: link state changed to DOWN Jan 18 13:26:34 check_reload_status: Linkup starting em3 Jan 18 13:26:27 php: rc.interfaces_wan_configure: The command '/sbin/ifconfig 'lagg0_vlan10' inet delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address' Jan 18 13:26:21 php: rc.newwanip: rc.newwanip: Failed to update opt2 IP, restarting... Jan 18 13:26:21 php: rc.newwanip: rc.newwanip: on (IP address: ) (interface: opt2) (real interface: lagg0_vlan10). Jan 18 13:26:21 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan10. Jan 18 13:26:21 php: rc.newwanip: rc.newwanip: on (IP address: 10.10.80.1) (interface: opt1) (real interface: lagg0_vlan80). Jan 18 13:26:21 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan80. Jan 18 13:26:20 php: rc.interfaces_wan_configure: The command '/sbin/ifconfig 'lagg0_vlan10' inet delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address' Jan 18 13:26:18 check_reload_status: rc.newwanip starting lagg0_vlan10 Jan 18 13:26:18 check_reload_status: rc.newwanip starting lagg0_vlan80 Jan 18 13:26:18 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:18 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:17 check_reload_status: Reloading filter Jan 18 13:26:17 check_reload_status: Configuring interface opt2 Jan 18 13:26:17 php: rc.newwanip: rc.newwanip: Failed to update opt2 IP, restarting... Jan 18 13:26:17 php: rc.newwanip: rc.newwanip: on (IP address: ) (interface: opt2) (real interface: lagg0_vlan10). Jan 18 13:26:17 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan10. Jan 18 13:26:17 php: rc.newwanip: rc.newwanip: on (IP address: 10.10.80.1) (interface: opt1) (real interface: lagg0_vlan80). Jan 18 13:26:17 php: rc.newwanip: rc.newwanip: Informational is starting lagg0_vlan80. Jan 18 13:26:16 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:16 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:15 check_reload_status: Linkup starting em2 Jan 18 13:26:15 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:14 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:14 kernel: em2: link state changed to UP Jan 18 13:26:14 check_reload_status: Linkup starting lagg0 Jan 18 13:26:14 check_reload_status: Linkup starting em3 Jan 18 13:26:14 kernel: lagg0_vlan10: link state changed to UP Jan 18 13:26:14 kernel: lagg0_vlan80: link state changed to UP Jan 18 13:26:14 kernel: lagg0: link state changed to UP Jan 18 13:26:14 kernel: em3: link state changed to UP Jan 18 13:26:14 check_reload_status: rc.newwanip starting lagg0_vlan80 Jan 18 13:26:14 check_reload_status: rc.newwanip starting lagg0_vlan10 Jan 18 13:26:14 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:14 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:12 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:12 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:12 check_reload_status: Linkup starting lagg0 Jan 18 13:26:12 check_reload_status: Linkup starting em3 Jan 18 13:26:12 kernel: lagg0_vlan10: link state changed to DOWN Jan 18 13:26:12 kernel: lagg0_vlan80: link state changed to DOWN Jan 18 13:26:12 kernel: lagg0: link state changed to DOWN Jan 18 13:26:12 kernel: em3: link state changed to DOWN Jan 18 13:26:12 kernel: em2: link state changed to DOWN Jan 18 13:26:12 check_reload_status: Linkup starting em2 Jan 18 13:26:12 php: rc.linkup: Hotplug event detected for DMZMGMT(opt2) but ignoring since interface is configured with static IP ( ) Jan 18 13:26:11 php: rc.linkup: Hotplug event detected for DMZ(opt1) but ignoring since interface is configured with static IP (10.10.80.1 ) Jan 18 13:26:10 check_reload_status: Linkup starting em2 Jan 18 13:26:10 kernel: em2: link state changed to UP Jan 18 13:26:10 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:10 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:10 check_reload_status: Linkup starting lagg0 Jan 18 13:26:10 kernel: lagg0_vlan10: link state changed to UP Jan 18 13:26:10 kernel: lagg0_vlan80: link state changed to UP Jan 18 13:26:10 kernel: lagg0: link state changed to UP Jan 18 13:26:10 kernel: em3: link state changed to UP Jan 18 13:26:10 check_reload_status: Linkup starting em3 Jan 18 13:26:08 check_reload_status: Linkup starting lagg0_vlan10 Jan 18 13:26:08 check_reload_status: Linkup starting lagg0_vlan80 Jan 18 13:26:08 check_reload_status: Linkup starting lagg0 Jan 18 13:26:08 kernel: lagg0_vlan10: link state changed to DOWN Jan 18 13:26:08 kernel: lagg0_vlan80: link state changed to DOWN Jan 18 13:26:08 kernel: lagg0: link state changed to DOWN Jan 18 13:26:08 kernel: em2: link state changed to DOWN Jan 18 13:26:08 check_reload_status: Linkup starting em2 Jan 18 13:26:08 kernel: em3: link state changed to DOWN Jan 18 13:26:08 check_reload_status: Linkup starting em3
Any ideas what any of this means? This in particular makes me wonder what is happening, since there is no IP address assigned to lagg0)vlan10.
php: rc.interfaces_wan_configure: The command '/sbin/ifconfig 'lagg0_vlan10' inet delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
What is the system trying to delete, and why is it even trying to assign an IP address, if none is technically required (there is an IP on the other side of the bridge on the LAN(em1) interface)?
-
Well, well, well. I think I have made a breakthrough.
I kept trying to pass traffic through the DMZMGT interface to the LAN and I kept getting the traffic blocked by the firewall. So, it made me start to think the problem had to be with the firewall rules, and not something else- like the bridge or the vlans. The more I looked at it, the more I thought that it looked right, except….
All of the rules configured to pass traffic from DMZMGT to the LAN subnet were set up to identify the source traffic as originating from the DMZMGMT Subnet. But....if there is no IP address assigned to the DMZMGMT interface, is there actually a DMZMGT subnet at all? How could the pf firewall identify traffic as originating from a DMZMGMT subnet without an IP address being assigned to that interface?
So, I get the idea to change the source to the IP address of the host machine behind the DMZMGMT interface on just one rule- the ping to primary DNS server, and voila!- it works. The ICMP traffic is getting to the DNS server and back lickety-split!
To test further, I change the source on this rule on the DMZMGMT interface:
Proto Source Port Destination Port Gateway Queue Schedule Description
IPv4 * DMZMGMT net * * * * none Allow DMZMGMT to any rulefrom DMZMGT net to any (*), and voila! Now I can ping from the host machine to any device in the LAN subnet without a problem. Also, I can ping any host on the internet, since the traffic can now pass out through the DMZMGT interface and traverse bridge1 to the LAN interface, then out through the default gateway and back.
Now, back to reality here- if I was simply bridging two LAN segments (may a WLAN interface?), this would be fine. But, since this machine is acting as a host machine to a virtual DMZ server, maybe this isn't the wisest idea? Probably, the virtual software layer between the virtual machine and the host (I guess that would be the hypervisor), and the separate vlans would not be significant enough segmentation between the DMZ server and machine hosting it to warrant this as a secure set up?
Then again, the firewall clearly was blocking traffic from the (host) machine behind the DMGMT interface to the LAN subnet, so maybe it would be enough to just create a rule or two using only the host's IP address as source to pass only the traffic that I need (DNS is the only one I can think of that would need to be open without limitation, and possibly ftp; then maybe only open up ports 80/443 as needed to update the packages on the host?) through to that machine?
Anyway, I'm glad it appears that the mystery has been solved. Bridging the two interfaces does seem to have worked. Although, it does seem that there should be some way to configure the bridge so that traffic can be identified as being on the same subnet on either side. Any suggestions on that?
-
Ha, as I was reading through those posts I started thinking exactly the same thing as you, there is no DMZMGMT subnet so the rule will never match.
Another thing I thought of, and this could explain your second log error, is that 2.1 has a known bug where it realoads interfaces that have no address when it shouldn't. It only seems to affect some NICs but I have no idea how that would apply to a LAGG interface. It is known to be a problem on em(4) NICs. This patch fixed it for me:
https://forum.pfsense.org/index.php/topic,66908.msg367991.html#msg367991
That and many other bugs have been squished in the new (first snapshots yesterday) 2.1.1 images. These are first snapshots though so further bugs will likely be found. I'm running it on my test box without issues though.Steve
-
Hello,
As as an update, I expiremented with changing the source for the rule to pass traffic from DMZMGMT to LAN, and instead of DMZMGMT subnet or the single IP of the machine behind the interface, I decided to use LAN subnet, which worked, since the machine behind DMZMGMT had an IP address in the same subnet.So, if for some reason (again, maybe briding WLAN?) one wanted to put some firewall rules in place between two segments of the same subnet, this would work. I did not leave it this way, however.
I set it up to only pass ICMP and DNS from the machine in DMZMGMT to LAN, to reject all (other) traffic from DMZMGMT to LAN subnet. So in practice, the machine can still reach the internet and download upgrades, etc. It is also still reachable from the LAN subnet. But, all other traffic to it should be blocked by default.
stephenw10, thanks for the link to the patch. I actually had read that thread when researching the possible issues with bridging the vlans, but had not seen the logs that showed the link state going up and down and the attempts at re-assigning the IP address, etc. I will give the patch a try, and hopefully it will help with the stability.
Thanks again for all of your help and input! I really appreciate it.