Problems with transparent bridge revisited..

wacko

Hi,

this is a follow-up to this old topic http://forum.pfsense.org/index.php/topic,5441.0.html (ignore the topic-title, the topic changed in the course of time)

By now I made some disturbing observations, which in my opinion should be considered as bugs.

Here my simplified setup:

Internet–-WAN --[pfSense]–-LAN---Internal network

LAN and WAN are bridged. (Maybe this was a bad choice in the beginning, and we would have better used an OPT Interface instead of the LAN.. read on)

We have a subnet of 128 ip adresses, lets call them x.y.z.w/25
The clients in the internal network are using the gateway upstream of pfSense, which is x.y.z.129

Another requirement is that we need DHCP running and serving the internal network.

The WAN IP address is set to x.y.z.130/25.

There is no rule at the WAN interface, hence everything from outside should be blocked (simplified)
There is an any->any rule on the LAN interface, hence everything from the LAN should be allowed.

Everything works from point of view like routing, dhcp etc. However a few things concern me:

Due to the fact, that we NEED the DHCP we need to set the IP of the LAN interface in the same subnet as the WAN. And this is where the problems start!

Case 1: Set LAN-IP to x.y.z.135/25
DHCP Server on LAN working–>OK
Packets from outside toward the LAN subnet are getting blocked-->OK
Packets from inside towards outside get passed-->OK
Packets from outside towards ip of LAN interface (x.y.z.135) get PASSED!!!! --->BUMMER
Hence, everything on the firewall is reachable from ouside.. this should not be. This is furthermore completely reconstructable, since it is always the case, but read on, more strange things are goind to come..
Sometimes (from time to time, sporadically) packets from inside the LAN towards the LAN address (x.y.z.135) get blocked. After a while, it works again and the same packets flow without problems.

This is really strange so I looked closer at the log entries and found the following: The packets coming from inside get blocked because pfSense "sees" them suddenly on the WAN interface. To clarify this, lets call the WAN interface em0 and the LAN interface em1. Hence sporadically I get a block on packets coming from LAN Subnet (identified by IP) but block is on IN on interface em0!!!! How the hack did the get to the other side?? I verified this by including a workarround rule to my WAN interace which says "let everything through, which comes from WAN-interface(em0) with the IP from the LAN-Subnet" - very strange rule (also a security issue --IP spoofing is possible with this). With this workarround rule, the block on my own packets are gone - I can still see it, when it happens, since I'm logging the workarround rule.. and it happens quite often.

Following the example from above, I suspected, that the reason why the LAN IP is reachable from outside lies in the same area. Hence in the PASS any->any rule on the LAN Interface. So I changed the Rule on LAN to "Packets coming from em1, whith IP in the LAN subnet to anywhere should pass" (previously also source was "any"). With this change also the LAN IP is no longer reachable from outside. Hence a rule on the LAN interface influences what packets from ouside do. It is with certainy that this is the same issue as above, packets from ouside "somehow" come IN over em1!! (And before you ask: NO THERE IS NO LOOP SOMEWHERE ELSE; pfSense is the only connection to the ouside!)

Case 2: Set LAN-IP to x.y.z.130/25
Hence, we use the SAME ip for LAN and WAN - a thing which actually should never be done. However, also this setup works with one additional benefit: Let's go back to the unmodified rules, hence no rule at all on WAN and a "any->any" rule on LAN.
Packets from inside get passed to the outside–>OK
Packets from outside towards LAN subnet get blocked -->OK
Packets from inside toward the ip x.y.z.130 get sporadically blocked--> Not OK, solution same as above!
Packets from outside towards the ip x.y.z.130 get sporadically through--> Slightly better then in case 1, but still not good. Solution is again as in case 1.
Probably this is tue to the fact, the IP on the ouside "masks" the same IP on the inside interface..

Case 3: Set LAN-IP to something which has nothing to do with any network, e.g. 10.0.0.1/32
I can only comment theoretically on this one, since I cannot use it right now, since it breaks DHCP support!!!
From theoretical point of view the issue, that the LAN-IP is reachable from outside should be gone, since you use anyway an unroutable Address for the LAN-IP. The issue, that packets from the inside get blocked at the LAN-IP should also be gone, since you would never try to connect this address…
However, no DHCP, hence not useable for us. If you don't need DHCP go for this!

Conclusion and Suggestions
1.) In a bridged mode, there should be only ONE single IP. Whenever bridging LAN with WAN the IP set on the LAN interface should be ignored, i.e. it sould be deleted from the LAN interface, which is a meber of bridge0. Then probably above problems would be gone!

2.) Enable in bridged mode the DHCP server on the bridge (alternatively on the WAN interface). (anyway right now the dhcp server is always complainig on start, that there are two interfaces with overalpping subnets.. em0 and em1. Yes, clearly, since each interface of the bridge has right now the same subnet. It would be much cleaner, if the bridge would have exactly one ip address, that of the WAN interface and DHCP would be started either on WAN or on bridge0.

Remarks:
Even though I found some workarrounds for my problems, it gives me strange feeling kowing that packets are "seen" on interfaces on the "other side". I mean, what if something like this happes also from time to time with packets from outside towards the LAN subnet. I never saw something like this, but the observed effects don't give me a warm feeling of confidence.
Additionally, using an OPT interface instead of the LAN interface to acomplish the same thing would also not work, since also then, the DHCP server could not be used. (I learned that with an OPT interface it is possible to use it WITHOUT an ip-address)

So, hopefully my observations help in isolating this issue. Keep up the good work guys :)

Best regards
Arno

PS: This is probably also the source of many issues which say something like "transparent bridge is sporadically blocking all traffic" - why? Well, if you are not carefull and you use the LAN-IP as gateway for the internal clients (instead of the upstream gateway) ,then sporadically everything is blocked to this interface, hence all traffic from internal network is blocked. Interesting is that if you use the correct upstream gateway and the block on the internal interface occurs, the traffic from internal LAN can flow without problems to the ouside. But note, using the LAN-IP as gateway DOES work (with AON enabeld) - it is just one hop more for routing (and not the way how a tranparent bridge should be used) - but this is only a side-information, the real issues are described above.

wacko

I manged to get DHCP working together with the transparent bridge setup (case 3 from above, i.e. LAN IP is set to something unrouteable). DHCP is housed on the WAN interface, and clients are served with public ips.

In order to get it working I patched pFSense. I hope that the contribution will be included in future releases.

First the logic with repsect to the question when should we be able to activate dhcp on an interface and when not?
The following changes yield the following behaviour:

1.) If an interface is bridged (with some other interface) dhcp cannot be activated (and is automatically ignored, in case it was previously activated). Only exception is the LAN interface - this can be bridged and also house dhcp. In my opinion this could also be dropped, but I letted it in - maybe there is some exotic setup which needs this.

2.) If some interface is bridged to WAN then it is allowed to host DHCP on WAN, i.e. the tab WAN becomes visible in the DHCP Server settings and the dhcp-configuration is written correctly to house DHCP on the WAN-interface.

In order to do so I needed to change two files:

/usr/local/www/services_dhcp.php - for activating/deactivating the settings-tabs. Very close to the beginning of the file:


....
/* if OLSRD is enabled, allow WAN to house DHCP. */
if($config['installedpackages']['olsrd']) {
        foreach($config['installedpackages']['olsrd']['config'] as $olsrd) {
                        if($olsrd['enable']) {
                                $iflist = array("wan" => "WAN", "lan" => "LAN");
                                $is_olsr_enabled = true;
                                break;
                        }
        }
}

/* changes start here */
$defaultif = "lan";

/* if LAN is bridged to WAN allow WAN to house DHCP */
if (!$iflist && ($config['interfaces']['wan']['ipaddr'] != "dhcp") && ($config['interfaces']['lan']['bridge'] == "wan") ){
        /* actually it should be enough to enable DHCP on WAN  */
        $iflist = array("wan" => "WAN", "lan" => "LAN");
        $defaultif = "wan";
}

if(!$iflist)
        $iflist = array("lan" => "LAN");

for ($i = 1; isset($config['interfaces']['opt' . $i]); $i++) {
        $oc = $config['interfaces']['opt' . $i];

        if (isset($oc['enable']) && $oc['if'] && (!$oc['bridge'])) {
                $iflist['opt' . $i] = $oc['descr'];
        }

        /* in case it is bridged to WAN, we should enable dhcp on WAN */
        if (!isset($iflist['wan']) && ($config['interfaces']['wan']['ipaddr'] != "dhcp") && ($oc['bridge'] == "wan") ){
                $iflist['wan'] = "WAN";
        }
}

if (!$if || !isset($iflist[$if]))
        $if = $defaultif;

/*unchanged from here */
$pconfig['range_from'] = $config['dhcpd'][$if]['range']['from'];
$pconfig['range_to'] = $config['dhcpd'][$if]['range']['to'];
....

The second file is /etc/inc/services.inc - the changes here reflect the above behaviour in wiriting the dhcp configuration file:


....
        $dhcpnum = 0;

        foreach ($dhcpdcfg as $dhcpif => $dhcpifconf) {

                $ifcfg = $config['interfaces'][$dhcpif];

/* changes from here */
                /*
                        Skip the interface in case of:
                        - dhcp is not enabled on this interface
                        - the interface gets its ip via dhcp
                        - the interface is not enabled and it is not a default interface (LAN/WAN)
                        - no device is set for this interface
                        - the interface is bridged and it is not LAN (DHCP can be activated on LAN even if bridged)
                */
                if (!isset($dhcpifconf['enable']) ||
                        ($ifcfg['ipaddr'] == "dhcp") ||
                        ( !isset($ifcfg['enable']) && ($dhcpif != "lan")  && ($dhcpif != "wan")  ) ||
                        !$ifcfg['if'] ||
                        ($ifcfg['bridge'] && ($dhcpif != "lan")) )
                        continue;

                /* just notfy user in case DHCP is started on a bridged interface */
                if($ifcfg['bridge'])
                        log_error("NOTE: DHCP Server on bridged {$dhcpif} is enabled.");

/* unchanged from here */
                $subnet = gen_subnet($ifcfg['ipaddr'], $ifcfg['subnet']);
                $subnetmask = gen_subnet_mask($ifcfg['subnet']);

                if($is_olsr_enabled == true)
                        if($dhcpifconf['netmask'])
                                $subnetmask = gen_subnet_mask($dhcpifconf['netmask']);

....

So, with these changes it is possible to house DHCP in a transparent bridge setup. Most of the problems described in the last post went away. I really hope this finds its way into the repository. In my opinion it improves the transparent bridge setup.

One puzzling thing remains: I still have to add a rule to my WAN interface (remember LAN is bridged to WAN with filtering on) which allows traffic from my internal ips via the external interface (WAN) to the WAN address in order to access anything on pfSense from my internal network. Without this rule, (even with a LAN rule wich allows on the LAN interface any->any) sometimes (not always!) no traffic from my internal network to the WAN address is possible (all other traffic works). Any idea?
I think somehow freebsd does some kind of balancing here, and this affects where packets are input to a bridge. Does that make any sense? For me it is still strange seeing packets on the "other side" coming IN….

Best regards
Arno

wacko

I'm currently using

1.2-TESTING-SNAPSHOT-07-21-2007
built on Thu Aug 9 19:22:35 EDT 2007

and I made patches of the above changes against my this snapshot.

Hence, the patch for the services.inc is against 1.72.2.57.2.10 from 2007/07/23 and the services_dhcp.php is against 1.38.2.18.2.2

If you need something else, or something is wrong with the patch, just let me know and I'll try to fix it.

Best regards
Arno

services.inc.patch.txt
services_dhcp.php.patch.txt

wacko

Is there no interest in this topic?

The patch solves some problems with transparent firewalling while at the same time allowing the usage of DHCP for the internal network. What is wrong with that?

Best regards
Arno