Working on getting OpenVPN server bridging to fly.

sullrich

You may be interested in this commit:

http://pfsense.com/cgi-bin/cvsweb.cgi/pfSense/usr/local/www/status_interfaces.php?rev=1.29.2.7;only_with_tag=RELENG_1

Shows the bridge status now under Status -> Interfaces

Numbski

Hmm. Is it safe for me to grab that one file and plug it in, or is there something more formal I should do? (ie, cvs?)

sullrich

Yeah, its safe. Simply replace /usr/local/www/status_interfaces.php with that new one.

Numbski

Cool. They both show learning. Of course, after 5 mins it still dies, but they both show learning. ;)

Seriously, have to put this to rest for now. I'll come back to it later. :)

Numbski

Testing remotely. Quick note - works great, except for a minor detail.

If you intend to use STP, DO NOT, I repeat, DO NOT, enable STP on the tap interface. Your actual hardware interface is fine, but doing so on the tap interface creates a really odd situation where traffic hits the endpoint tap interface, and gets to your bridge, but nothing ever returns. Disabling STP on the tap interface resolves that problem.

Otherwise all is well. Just need to figure out why CARP chokes after 5 mins.

Numbski

Another update. Looked like all was working just fine, until the firewall seized to a halt. Same behavior as before too. It responds to ctrl-alt-del by trying to shut down, but fails to actually do so. Has to be hard rebooted.

When I get a chance to power cycle it, I'll see if I can set up watchdog to mitigate this side effect until I can find the root cause. Again, if you have any speculations as to the cause, post up and I'll try it. Also, anyone without carp that wants to try this, see what happens, let me know.

Numbski

For the sake of discussion, I think I left off an option that might be causing an issue. Dunno yet:

dev-node tap-bridge

Here's the official OpenVPN docs on the matter. Suprised that I overlooked that directive.

http://openvpn.net/bridge.html

It claims that directive is only required under windows though. Another comment is this:

A common mistake that people make when manually configuring an Ethernet bridge is that they add their primary ethernet adapter to the bridge before they have set the IP and netmask of the bridge interface. The result is that the primary ethernet interface "loses" its settings, but the equivalent bridge interface settings have not yet been defined, so the net effect is a loss of connectivity on the ethernet interface.

So, despite what I was reading elsewhere, it appears that the openvpn folks would prefer we do this:

ifconfig sis0 up
ifconfig tap0 up
ifconfig bridge0 create
ifconfig bridge0 addm sis0 addm tap0
ifconfig bridge0 172.16.10.2 netmask 255.255.255.0

The problem here of course is the impact this would have on CARP. I have sis0 in carp3, and I cannot do addm carp3. I don't know (and can't easily test at this moment) whether I can ifconfig bridge0 instead of sis0, and still have it able to join a carp cluster. If anyone wants to speak up on that point as well, please do. It will be about a week before I can safely test that (I think?). I might have an opportunity while in Montreal.

If this is indeed correct, then from pfSense's point of view, we need to able to change the lan interface (or in my case, opt interface) to be bridge0 and not sis0. That way all rules are being applied to the bridge and not to the physical interface, unless someone wants to step up with more information to say otherwise. I'm honestly just not finding much info in regards to FreeBSD, bridging, and rules re: pf, only that you should only create rules for one interface and not both, as it screws things up. I haven't found any documentation on whether rules should be applied specifically to the bridge, or to the physical ints.

Also, I'm puzzled by STP hosing things up on tap0. Doesn't make sense to me.

Numbski

In Montreal now. Noticed that I can't actually set up a watchdog timer, as it requires kernel support (and it isn't in GENERIC), so oops. :)

Have to find another way for now.

Might I suggest we officially enable watchdog in the kernel? Seems like a very logical, sane thing to have in a firewall. If the kernel stops responding for x seconds, reboot the system.

sullrich

We already support the GEOD watchdog but I do not plan on adding the SW_WATCHDOG as it may interfere with systems this late in the testing cycle.

We may be able to add it to 1.1.

Numbski

Ah, cool. Thanks. Hopefully I'll have time later to rebuild with SW_WATCHDOG for my own purposes. Doesn't really fix the problem at hand, but makes me feel better to know the system will kick itself. ;)

Numbski

New observations. :D

I had the opportunity to do an openvpn bridge on a pfSense RC2i box without CARP. Worked 99% flawlessly with the current code.

Added server-bridge directive.
Assigned tap0 as an opt.
Bridged that opt to lan using the webui.
Set an any-any rule on the opt.

The only thing that didn't work? STP was enabled on the tap interface by default! ifconfig bridge0 -stp tap0, and all was well.

Really, REALLY screwy stuff here. Wonder if I should just re-load my firewalls when I get back and start clean? ???

Would help if someone could verify my findings.

Numbski

Been up for a couple of days, completely stable on bridging on everyone's pfSense boxes but my own.

Go fig. ;D

So yeah. Put in a statement to check if an interface is a tap interface, and if it is, don't enable STP. Do that, OpenVPN bridging is good to go. Works quite nicely with CARP too, despite my initial experiences. Just do "local (CARP IP)" on both boxes, and presuming you've used the same server crt, ca cert, server key, and dh, it will fail over gracefully.

Good stuff guys. Sorry I made a three page thread on it. At least someone else has issues that match mine, they'll have something to go on. When I return I'll try doing a fresh load on my boxes and see what my results are. I think you can safely say that OpenVPN bridging works though.

NOTE: The change needs to be made ~ line 144 in /etc/inc/interfaces.inc. Currently it looks like this:


if(!is_interface_wireless($lancfg['if']) and
                   !is_interface_wireless($config['interfaces'][$lancfg['bridge']]['if']))
                        mwexec("/sbin/ifconfig bridge{$bridges_total} stp {$config['interfaces'][$lancfg['bridge']]['if']} stp {$lancfg['if']}");

I'm thinking on that first line we need to add something along the lines of a negated regex, maybe !/tap/? Don't know precisely how that goes in php. Then after the mwexec line, we add an elsif block that says the same thing, only don't negate the regex, and on the mwexec line, leave off the stp part. Make sense?

sullrich

Good to hear. If you want to create a patch using diff -rub I'll get it commited.

Numbski

I actually don't know how to handle the php code for that situation. :'(

Numbski

Grrr.

The bridge doesn't hold after a reboot. bridge0 gets created, but the interfaces don't get added. I have to do it manually. At what point in the boot process do the bridges get brought up? Is it possible that it's being attempted prior to openvpn being launched, thus tap0 doesn't exist and the interfaces don't get added?

sullrich

You will most likely need to use <shell_cmds>and friends to do all your custom commands.

I wonder if Fernando can write in a bridging module at some point. I'll ask him.</shell_cmds>

Numbski

Care to expound? I've never seen that tag before. Can I place that manually into config.xml or something?

Also…boxes are randomly hanging after a period of openvpn use. These aren't low-end systems either. Some are custom boxes with 1.4Ghz-2.0Ghz P4's with lots and lots of RAM, and the others are from vendors listed on this site with plenty of RAM.

Hmm. Behavior is consistent across the board in that regard to. Nothing logged points to the failure though. Tends to make me think it is a FreeBSD issue and not pfSense-specific. I'll keep digging.

hoba

See http://faq.pfsense.com/index.php?action=artikel&cat=10&id=38&artlang=en&highlight=hidden for additional tags in the config.xml that can be used to trigger some additional actions for example on bootup or filter reload.

Numbski

Thank you!

(BTW, have I mentioned how frustrating it is to hunt down random lockups???)

Numbski

erm…hmmm

Heres's what I added in the system section of config.xml:


                <shellcmd>ifconfig bridge0 create</shellcmd>
                <shellcmd>ifconfig bridge0 addm sis0 addm tap0 stp sis0 up</shellcmd>

The commands appear to be ignored at bootup though. I know you posted shell_cmd, but the faq says it is just one word, shellcmd. Is the faq wrong?

The default config.xml from cvs doesn't help much either:

http://pfsense.com/cgi-bin/cvsweb.cgi/pfSense/conf.default/config.xml?rev=1.19;content-type=text%2Fplain

:)