SIXXS-Aiccu and pfSense

[logion]

As I wanted to setup an SIXXS AYIYA tunnel on my pfSense box and found this twitter conversation, I thought I would detail my pfSense + sixxs-aiccu findings here. SIXXS' nearest tunnel endpoint is located in my country(Brussels). The RTT to this POP is equal to ~20ms, which is 10ms faster than the closest HE tunnel endpoint in my area (Amsterdam). YMMV however. Configuring a non-aiccu tunnel is supported in pfSense's webinterface (cfr this guide); so you have to decide whether setting up a SIXXS aiccu tunnel is worth the hassle. Which solution is better for dynamic IPv4 endpoints is unknown to me, I'd speculate that the Aiccu software with its heartbeat mechanism has the edge here.

One heads-up before we start: at the time of writing the aiccu tunnel interface's name is fixed to tun0 which means that it doesn't show up in the webinterface. One result is that you can't configure the firewall to differentiate according to the tun0 interface. See the bottom of this post for more info, any help is greatly appreciated!

The installation/configuration is done via CLI (e.g. through SSH), to my knowledge there is no integration into the web-interface in pfSense 2.1-BETA.

In short you can install the sixxs-aiccu package (http://www.freebsd.org/cgi/ports.cgi?query=sixxs-aiccu&stype=all) using pkg_add -r sixxs-aiccu from CLI.

More elaborate guide:
SSH to machine (SSH needs to be enabled via webinterface/serial port):
ssh admin@ <pfsense-box>Mount filesystem as R/W:
/etc/rc.conf_mount_rw
Install package:
pkg_add -r sixxs-aiccu
You should see output similar to this:

Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/Latest/sixxs-aiccu.tbz… Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/p11-kit-0.11.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/gmp-5.0.4.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/nettle-2.4.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/pkg-config-0.25_1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/libiconv-1.13.1_2.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/gettext-0.18.1.1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/libgpg-error-1.10.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.3-release/All/gnutls-2.12.18.tbz... Done.

Aiccu created a configuration file under /usr/local/etc/aiccu.conf, the binary however looks for the config file at /etc/aiccu.conf so symlink it:
ln -s /usr/local/etc/aiccu.conf /etc/aiccu.conf

Update /etc/aiccu.conf according to the instructions on http://www.sixxs.net/wiki/Aiccu/InstallationOnFreeBSD#Configure_Aiccu. Note that creating the gif0 device isn't necessary, if you're using a AYIYA tunnel like me then aiccu will always created a tun0 interface. Unfortunately this seems to have some unwanted consequences, see the end of this post for more info.

The sixxs-aiccu package also created a startup script at /usr/local/etc/rc.d/sixxs-aiccu. As the pfSense startup system differs a bit from FreeBSD (see NOTE on startup scripts at http://doc.pfsense.org/index.php/Installing_FreeBSD_Packages), we have to alter the script as follows:
mv  /usr/local/etc/rc.d/sixxs-aiccu  /usr/local/etc/rc.d/sixxs-aiccu-start.sh
chmod  +wx /usr/local/etc/rc.d/sixxs-aiccu-start.sh
Apply the following patch:
diff -puN sixxs-aiccu sixxs-aiccu-start.sh

–- sixxs-aiccu 2013-05-09 12:34:42.000000000 +0200
+++ sixxs-aiccu-start.sh 2013-05-09 12:12:43.000000000 +0200
@@ -9,7 +9,7 @@

Add the following lines to /etc/rc.conf to enable sixxs-aiccu:

-#sixxs_aiccu_enable="YES"
+sixxs_aiccu_enable="YES"

. /etc/rc.subr
@@ -37,4 +37,4 @@ license_cmd="${command} license"
version_cmd="${command} version"
extra_commands="brokers tunnels test autotest license version"

-run_rc_command $1
+run_rc_command start

The resulting sixxs-aiccu-start.sh file should look like http://pastebin.com/mb9zKrhr.

When you reboot there should now be an active tun0 interface (or gif0 in case of a non-AYIYA tunnel) and you should be able to reach IPv6 internet from your pfSense box, you can test this via ping6 ipv6.google.com.

Now that you've successfully configured the tunnel you'll probably want your pfSense to act as an IPv6 Internet gateway for your home network. By default SIXXS provides one additional subnet that is routed over the tunnel that can be used inside your LAN. See the tunnelinfo page on sixxs.net for this subnet. You can setup a radvd server that distributes this Ipv6 prefix to all machines in your LAN or you can assign static addresses from the routed subnet to your LAN machines. I chose the latter approach and am now able to reach IPv6 Internet from my machines.

That's it, enjoy IPv6 internet via SIXXS provided by a POP near to you. Any questions? Leave a reply.

Issues:

• sixxs-aiccu creates a tun0 interface and this interface isn't configurable in the webinterface.
  I've found the following issue: http://redmine.pfsense.org/issues/2211, the suggested workaround doesn't seem to work however. I've applied the work-around as follows and rebooted:

diff /etc/inc/util.inc /nano/base/etc/inc/util.inc
1563,1564c1563
< if (preg_match("/^enc|^cua|^tap|^l2tp|^pptp|^ppp|^ovpn|^gif|^gre|^lagg|^bridge|vlan|_wlan/i", $ifcfg['if'])) { /* logion: enable tun interfaces in webinterface /
< / if (preg_match("/^enc|^cua|^tun|^tap|^l2tp|^pptp|^ppp|^ovpn|^gif|^gre|^lagg|^bridge|vlan|_wlan/i", $ifcfg['if'])) { */
–-

if (preg_match("/^enc|^cua|^tun|^tap|^l2tp|^pptp|^ppp|^ovpn|^gif|^gre|^lagg|^bridge|vlan|_wlan/i", $ifcfg['if'])) {

If anyone has some suggestions on how to fix this, please share them. You can still pass incoming IPv6 traffic to your LAN network, you just can't specify IPv6 rules specific to the AICCU tunnel (allthough the IPv6 subnet is fixed to the tunnel, so it's possible to differentiate on IP but not incoming/outgoing interface).

References:

• http://redmine.pfsense.org/issues/2580
• http://redmine.pfsense.org/issues/2211
• http://doc.pfsense.org/index.php/Installing_FreeBSD_Packages
• http://www.freebsd.org/cgi/ports.cgi?query=sixxs-aiccu&stype=all</pfsense-box>

[logion]

I managed to get the firewall working, assuming that the aiccu client creates a tunnel interface named tun0 you can apply the following patch to /usr/local/www/interfaces_assign.php:

diff -pNu /usr/local/www/interfaces_assign.php.bak /usr/local/www/interfaces_assign.php
–- ../../usr/local/www/interfaces_assign.php 2013-05-09 14:19:27.018264319 +0200
+++ /home/fvdabeele/Downloads/interfaces_assign.php 2013-05-09 15:57:43.070257996 +0200
@@ -143,6 +143,13 @@ if (is_array($config['ppps']['ppp']) &&
}
}

+/* add tun0 interface */
+                $portlist['tun0'] = array();
+                $portlist['tun0']['if'] = 'tun0';
+                $portlist['tun0']['istun'] = true;
+
+
+
$ovpn_descrs = array();
if (is_array($config['openvpn'])) {
if (is_array($config['openvpn']['openvpn-server']))

Then you have to create a new interface via the webinterface on  https://<pfsense>>/interfaces_assign.php. Edit the newly created interface and check 'Enable'. Next you can surf to Firewall > Rules and there you can now find a new tab for OPT1, here you can add rules for the IPv6 traffic that is tunneled via aiccu.
Here is an example that allows SSH over IPv6 to all machines inside my subnet: http://imgur.com/KkgO1xU.

Integrating all this in the pfSense web interface is feasible, but would be quite some work nonetheless. Apart from the php pages that would allow to configure the aiccu config, the aiccu package should be recompiled to use a separate tunnel interface named aiccu; which can then get a separate interface type (similar to the openvpn and gif tunnel types) inside pfSense.</pfsense>

[Luzemario]

Hi logion,

There is support for aiccu in current stable releases?

I am running pfsense 2.0.3 and I get  the following messages from package install:

"Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.1-release/Latest/sixxs-aiccu.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.1-release/Latest/sixxs-aiccu.tbz' by URL"

[logion]

Whether or not this guide will work in pfsense 2.0.3 I can't say (I'm running the Beta as you can see). But it would seem that the package repository structure has changed, a problem which should be more common to all 2.0.3 packages. Can you add a package via URL? You should install this package: http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.1-release/Latest/sixxs-aiccu.tbz; it might be that the dependencies will give a 4.04 as well though.
I'd suggest trying to install aiccu from the web interface: System -> Packages. If that doesn't work you might want to try changing package repository as mentioned here: http://doc.pfsense.org/index.php/Creating_Your_Own_Package_Repository#On_pfSense_2.0. I'd imagine the hostname would be ftp-archive.freebsd.org in this case.
Here is an other suggestion: http://forum.pfsense.org/index.php/topic,8272.msg46662.html#msg46662.
It really seems to be more pfSense related than aiccu related, can you install any other packages? pkg_add -r nano for instance?

[Luzemario]

I can add several packages and all work as expected. It appears to be aiccu package was intentionally not available to 2.0.x.

I need so much to try ipv6, so I decided to move to 2.1-Beta. PfSense team is clear about ipv6 being on roadmap of 2.1 only.

I will follow your guide and post my findings.

Thanks!

[Luzemario]

Hi logion,

I follow closely your guide, and I appeared to be successful. I can run

sixxs-aiccu autotest /usr/local/etc/aiccu.conf

and almost all tests passed. The only test unsuccessful was traceroute to remote IPv4 endpoint, but I can ping it. Some routers in path block traceroute's requests.

But unfortunately I cannot reproduce 'sixxs-aiccu' behavior. I cannot ping none remote IPv6 address, nor the remote tunnel endpoint. Ping6 shows the error:

[2.1-BETA1][root@fw.local]/root(21): ping6 noc.sixxs.net
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:ZZZZ::2 --> 2001:838:1:1:210:dcff:fe20:7c7c
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
^C
--- noc.sixxs.net ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

[2.1-BETA1][root@fw.local]/root(21): ping6 2001:XXXX:YYYY:ZZZZ::1
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:ZZZZ::2 --> 2001:XXXX:YYYY:ZZZZ::1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
^C
--- noc.sixxs.net ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

If I run sixxs-aiccu autotest again the tests (and ping6 and traceroute6) work inside testing. At after sixxs-aiccu ends, it fails again.

[Luzemario]

The 'easy' part…  ;D

I am not a programmer, but lastly I become very happy with my progress... ;)

I figured out why network was offline: sixxs-aiccu is not running!!!

/usr/local/etc/rc.d/sixxs-aiccu-start.sh start      
Cannot 'start' sixxs_aiccu. Set sixxs_aiccu_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.                                                                                               
[2.1-BETA1][root@fw.local]/root(30): /usr/local/etc/rc.d/sixxs-aiccu-start.sh onestart   
[2.1-BETA1][root@fw.local]/root(31): ping6 noc.sixxs.net                                 
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:ZZZZ::2 --> 2001:838:1:1:210:dcff:fe20:7c7c                        
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=0 hlim=54 time=247.998 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=1 hlim=54 time=248.104 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=2 hlim=54 time=252.881 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=3 hlim=54 time=248.184 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=4 hlim=54 time=254.536 ms      

I suspect pfsense do not use a '/etc/rc.conf' file to start services… so where is it?

[Luzemario]

logion,

Please be more especific in 'patch' section… not all folks here deal with 'diff'...  ???

[logion]

If you put daemonize=true in the aiccu config file (/etc/aiccu.conf), then you can just start aiccu from CMD line (/usr/local/sbin/sixxs-aiccu start) and it will start as a background process.
If you want to integrate it with the rc.d system (i.e. setup the tunnel when pfSense boots), then look at the second part of my first post. Your /usr/local/etc/rc.d/sixxs-aiccu-start.sh shouldn't accept command line arguments, you have to uncomment sixxs_aiccu_enable="YES" and change the last line to "run_rc_command start". See lines 12 and 40 in this pastebin file: http://pastebin.com/mb9zKrhr. If you change these two lines, then the aiccu tunnel should come online after pfSense boots.

The second patch file just adds these 4 lines at line 146 in the  /usr/local/www/interfaces_assign.php file (remove the plusses):
+/* add tun0 interface */

• $portlist['tun0'] = array();
• $portlist['tun0']['if'] = 'tun0';
• $portlist['tun0']['istun'] = true;

[logion]

I put the entire patched /usr/local/www/interfaces_assign.php file online at http://pastebin.com/yxgAmtT0. Look at the contents near line 146

[Luzemario]

logion,

Thank you very much. I am enjoying IPv6 now!  8) I believe your guide is enough for anyone wanting to try IPv6 until pFsense folks put it in the web interface. I am connected directly to a autonomous system (nearest PoP here is a AS one), with pings of marvelous 9ms for 1000km…  ;D

However, I noticed some commands gone from sixxs-aiccu after running the patch. For instance, I can't do anymore 'sixxs-aiccu status'. There is no error message, nor output. Maybe it went to the system log?

[logion]

There is only the onestatus command that you can run via the original /usr/local/etc/rc.d/sixxs-aiccu, it only seems to display the contents of /var/run/aiccu.pid though.
Could you post how stable the tunnel is by the way? I've noticed that the tunnel stopped working eventhough the sixxs-aiccu daemon process was still running. Restarting the daemon fixed it…

[Luzemario]

@logion:

Could you post how stable the tunnel is by the way? I've noticed that the tunnel stopped working eventhough the sixxs-aiccu daemon process was still running. Restarting the daemon fixed it…

The first of first: I am using heartbeat here. I will let AYIYA only for NATed scenarios. The idea is my pFsense box handle tunnel and firewall only. My machines will get fixed IPs in the /64 network I got.

That said, the tunnel is quite stable whenever your clock is synced via ntp and you allow ping6 from the remote side (very important). It go down silently if you do not allow v6 pings from remote side of tunnel after some time. Your side gets pinged every 60 seconds. Currently I could not open a rule to ICMPv6 ping since it is not on the protocols list to choose. I opened all protocols from remote IPv6 tunnel host (using single host or alias option), but it still is a issue. My pFsense box keeps blocking it. Maybe your case can be issues with delay, but I could not see it since my ISP is very good…  ;D My worst ping to remote side of tunnel was 12msec...

Disabling firewall is not an option for me. Still seeking a solution.

[logion]

NTP seems to be operating fine. I was dropping ICMPv6 traffic from the POP's endpoint however. Now that I've added a rule that should fix this, let's see how long the tunnel will last :). I know I should follow IETF's RFC 4890 on ICMP filtering, but has this per chance been applied to a pfSense setup before? I.e. can I use someone else's rules as a basis for ICMPv6 filtering? Maybe there should be a RFC 4890 compliant set of rules defined as a preset in pfSense?

How do you know that the pings from the remote side aren't coming through by the way? SIXXS supplies this testing tool: http://www.sixxs.net/tools/traceroute/ but it seems to be down for the moment… You say: "Currently I could not open a rule to ICMPv6 ping since it is not on the protocols list to choose.", when choosing IPv6 as TCP/IP  Version and then choosing ICMP as protocol; I would assume this filters ICMPv6 traffic and not ICMPv4. I've added a rule like this from ALL sources and pnzone.net is able to ping me using ICMPv6, so I think it's working. Are you reloading the filter table after you create new rules? It should work...

You can use AYIYA tunnels without NAT as well? At least that's what I'm doing and it seems to be working (although perhaps not so stable, the next few days should point this out).

[Luzemario]

I saw the ping packet being blocked by the 'default rule' in the firewall log. Check "show packets blocked by default rule" to see most of packets dropped. You will see the pings coming from the remote side of the tunnel being blocked. For sixxs tunnels it can't happen.

ICMPv4 and ICMPv6 are (very) different protocols. There are several features of IPv6 wich depend of new things added to ICMPv6 (neighboor discovery, for example). I would not assume choosing IPv6 and ICMP being ICMPv6. It's make sense, but I cannot figure out why it has been dropped, so I guess better make no assumptions until someone from pFsense team can confirm.

I believe allowing ICMPv6 from remote side is a thing more sixxs-related than needed by some RFC. They use it in their credit balance account (they mean this in the FAQ). SixXS is free at all, but has a interesting virtual credit system. If you make your tunnel stable, you gain credits to 'buy' more subnets, etc… ::) is like a sort of 'game'...

I make a rule to allow IPV6 and ICMP, but the packets still get being dropped, and the drop is logged in firewall log. I don't know if this is a bug or if my interface is not choosen correctly in the rule. Firewall logs show WAN interface, not GIF (heartbeat is configured as GIF, not TUN). I did not apply your last patch to see the new GIF interface in webconfigurator yet, so I can't choose it in the rule setup.

I made no testing with AYIYA yet. AYIYA have a bunch of complexity to help to connect, but I personally do not like when it fails, since there is a bunch of things to debug too... :(

[logion]

You definitely need to allow icmpv6 traffic via the tunnel interface, not via the WAN interface (IPv6 traffic is encapsulated in IPv4 traffic when it enters via the WAN interface), therefore you need to create a OPT interface in the pfSense webinterface. This requires my patch to interfaces_assign.php if the tunnel is named tun0, if it is named gif (like in your case) it might not need to be patched; or you can replace tun with gif in my patch file ;)

Just to demonstrate that it's working over here, my pfSense firewall rules for my tun0 interface look like this: https://www.dropbox.com/s/ldxqrx6li4frckf/Screenshot%20from%202013-05-13%2022%3A47%3A15.png. The first rule allows SSH to the public IPv6 address of my pfSense box, the second address allows incoming ICMPv6 traffic from the tunnel POP's endpoint. Note that these addresses are in their own separate tunnel subnet (important)! I'm 100% sure that IPv6+ICMP results to ICMPv6. Executing "pfctl -sa | grep icmp" gives :
pass in quick on tun0 inet6 proto ipv6-icmp from 2a02:578:xxxxx::1 to 2a02:578:xxxxx::2 keep state label "USER_RULE: Allow tunnel's POP endpoint to ping gatekeeper"
Notice that proto says ipv6-icmp, while the interface is IPv6 + ICMP.

Finally executing "tcpdump -n -i tun0 -vv icmp6" on pfSense shows ICMP6 echo requests/replies from the POP to my pfSense box every minute (as well as neighbor sollications for the tunnels POP from my pfSense box, strange that pfSense doesn't cache this for sending the echo reply packets :s).

I am aware that ICMP6 differs from ICMP4 and that IPv6 relies heavily on ICMP6 for its workings (in regards to routers dropping packets that are larger than their link's MTUs allow, for instance). I also don't see any harm in allowing ICMP6 from the POP's endpoint to the pfSense box (they are conceptually in the same subnet after all). I was referring to a policy for incoming ICMPv6 traffic that will be forwarded to my LAN subnet (including my pfSense LAN interface, not really forwarded in this case though). In this case RFC 4890 might provide some good guidance. Guess I'll have to read up a bit more on this.

[Luzemario]

Hi logion,

Thank you very much for your help. My pFsense box is correctly configured and my GIF interface is answering pings from remote side of tunnel.

I will play with subnet now…  :P

[s0lid]

I'm not quite sure why pfsense refuses to pass ICMPv6 from my POP to local End point on the tunnel if.
IPv6 tunnel works. I've tried to allow everything from POP address to End point address but no, RRD graphs show that there's constant IPv6 in-block for 409b/s which seems to be ICMPv6.

I've had this working earlier before i moved to routerboard, but had to come back to pfsense since ROS 6.1 managed to turn routerboard inaccessable… couple of times.

[MerlinWV]

By the way, thanks for posting this - it helped me set up a tunnel on 2.1 really quick.

Two problems left: the tunnel goes down after an undefined time, not passing any traffic. It seems to be realted to a tun-Problem in FreeBSD 8.0/8.1  . Should no longer present in 8.3, but sure show the same behaviour. Just upgraded to 2.1.1-PRERELEASE - let's see if that fixes it.

Also, I recommend disabling the pfscrub Option in Advanced / Firewall/NAT due to a bug in FreeBSD:
https://redmine.pfsense.org/issues/2762
Even if they talk about an IPv6-Options allow rule, probably the fix mentioned in
http://www.freebsd.org/cgi/query-pr.cgi?pr=172648
is better. I did not want to disable rfc1323 (TCP window scaling).

[eri--]

Why you guys need a tun device for this.

The SIXXS daemon is so damn simple that you just need a script to login and not need the daemon + the gif interface setup.

Also it is very simple to integrate this to pfSense.

If you do the php work on interfaces.php i will see to make this included in 2.2