SIXXS-Aiccu and pfSense

Luzemario

Hi logion,

There is support for aiccu in current stable releases?

I am running pfsense 2.0.3 and I get the following messages from package install:

"Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.1-release/Latest/sixxs-aiccu.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.1-release/Latest/sixxs-aiccu.tbz' by URL"

logion

Whether or not this guide will work in pfsense 2.0.3 I can't say (I'm running the Beta as you can see). But it would seem that the package repository structure has changed, a problem which should be more common to all 2.0.3 packages. Can you add a package via URL? You should install this package: http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.1-release/Latest/sixxs-aiccu.tbz; it might be that the dependencies will give a 4.04 as well though.
I'd suggest trying to install aiccu from the web interface: System -> Packages. If that doesn't work you might want to try changing package repository as mentioned here: http://doc.pfsense.org/index.php/Creating_Your_Own_Package_Repository#On_pfSense_2.0. I'd imagine the hostname would be ftp-archive.freebsd.org in this case.
Here is an other suggestion: http://forum.pfsense.org/index.php/topic,8272.msg46662.html#msg46662.
It really seems to be more pfSense related than aiccu related, can you install any other packages? pkg_add -r nano for instance?

Luzemario

I can add several packages and all work as expected. It appears to be aiccu package was intentionally not available to 2.0.x.

I need so much to try ipv6, so I decided to move to 2.1-Beta. PfSense team is clear about ipv6 being on roadmap of 2.1 only.

I will follow your guide and post my findings.

Thanks!

Luzemario

Hi logion,

I follow closely your guide, and I appeared to be successful. I can run

sixxs-aiccu autotest /usr/local/etc/aiccu.conf

and almost all tests passed. The only test unsuccessful was traceroute to remote IPv4 endpoint, but I can ping it. Some routers in path block traceroute's requests.

But unfortunately I cannot reproduce 'sixxs-aiccu' behavior. I cannot ping none remote IPv6 address, nor the remote tunnel endpoint. Ping6 shows the error:


[2.1-BETA1][root@fw.local]/root(21): ping6 noc.sixxs.net
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:ZZZZ::2 --> 2001:838:1:1:210:dcff:fe20:7c7c
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
^C
--- noc.sixxs.net ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

[2.1-BETA1][root@fw.local]/root(21): ping6 2001:XXXX:YYYY:ZZZZ::1
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:ZZZZ::2 --> 2001:XXXX:YYYY:ZZZZ::1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
ping6: sendmsg: Network is down
ping6: wrote noc.sixxs.net 16 chars, ret=-1
^C
--- noc.sixxs.net ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

If I run sixxs-aiccu autotest again the tests (and ping6 and traceroute6) work inside testing. At after sixxs-aiccu ends, it fails again.

Luzemario

The 'easy' part… ;D

I am not a programmer, but lastly I become very happy with my progress... ;)

I figured out why network was offline: sixxs-aiccu is not running!!!


/usr/local/etc/rc.d/sixxs-aiccu-start.sh start      
Cannot 'start' sixxs_aiccu. Set sixxs_aiccu_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.                                                                                               
[2.1-BETA1][root@fw.local]/root(30): /usr/local/etc/rc.d/sixxs-aiccu-start.sh onestart   
[2.1-BETA1][root@fw.local]/root(31): ping6 noc.sixxs.net                                 
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:ZZZZ::2 --> 2001:838:1:1:210:dcff:fe20:7c7c                        
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=0 hlim=54 time=247.998 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=1 hlim=54 time=248.104 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=2 hlim=54 time=252.881 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=3 hlim=54 time=248.184 ms                      
16 bytes from 2001:838:1:1:210:dcff:fe20:7c7c, icmp_seq=4 hlim=54 time=254.536 ms

I suspect pfsense do not use a '/etc/rc.conf' file to start services… so where is it?

Luzemario

logion,

Please be more especific in 'patch' section… not all folks here deal with 'diff'... ???

logion

If you put daemonize=true in the aiccu config file (/etc/aiccu.conf), then you can just start aiccu from CMD line (/usr/local/sbin/sixxs-aiccu start) and it will start as a background process.
If you want to integrate it with the rc.d system (i.e. setup the tunnel when pfSense boots), then look at the second part of my first post. Your /usr/local/etc/rc.d/sixxs-aiccu-start.sh shouldn't accept command line arguments, you have to uncomment sixxs_aiccu_enable="YES" and change the last line to "run_rc_command start". See lines 12 and 40 in this pastebin file: http://pastebin.com/mb9zKrhr. If you change these two lines, then the aiccu tunnel should come online after pfSense boots.

The second patch file just adds these 4 lines at line 146 in the /usr/local/www/interfaces_assign.php file (remove the plusses):
+/* add tun0 interface */

$portlist['tun0'] = array();
$portlist['tun0']['if'] = 'tun0';
$portlist['tun0']['istun'] = true;

logion

I put the entire patched /usr/local/www/interfaces_assign.php file online at http://pastebin.com/yxgAmtT0. Look at the contents near line 146

Luzemario

logion,

Thank you very much. I am enjoying IPv6 now! 8) I believe your guide is enough for anyone wanting to try IPv6 until pFsense folks put it in the web interface. I am connected directly to a autonomous system (nearest PoP here is a AS one), with pings of marvelous 9ms for 1000km… ;D

However, I noticed some commands gone from sixxs-aiccu after running the patch. For instance, I can't do anymore 'sixxs-aiccu status'. There is no error message, nor output. Maybe it went to the system log?

logion

There is only the onestatus command that you can run via the original /usr/local/etc/rc.d/sixxs-aiccu, it only seems to display the contents of /var/run/aiccu.pid though.
Could you post how stable the tunnel is by the way? I've noticed that the tunnel stopped working eventhough the sixxs-aiccu daemon process was still running. Restarting the daemon fixed it…

Luzemario

@logion:

Could you post how stable the tunnel is by the way? I've noticed that the tunnel stopped working eventhough the sixxs-aiccu daemon process was still running. Restarting the daemon fixed it…

The first of first: I am using heartbeat here. I will let AYIYA only for NATed scenarios. The idea is my pFsense box handle tunnel and firewall only. My machines will get fixed IPs in the /64 network I got.

That said, the tunnel is quite stable whenever your clock is synced via ntp and you allow ping6 from the remote side (very important). It go down silently if you do not allow v6 pings from remote side of tunnel after some time. Your side gets pinged every 60 seconds. Currently I could not open a rule to ICMPv6 ping since it is not on the protocols list to choose. I opened all protocols from remote IPv6 tunnel host (using single host or alias option), but it still is a issue. My pFsense box keeps blocking it. Maybe your case can be issues with delay, but I could not see it since my ISP is very good… ;D My worst ping to remote side of tunnel was 12msec...

Disabling firewall is not an option for me. Still seeking a solution.

logion

NTP seems to be operating fine. I was dropping ICMPv6 traffic from the POP's endpoint however. Now that I've added a rule that should fix this, let's see how long the tunnel will last :). I know I should follow IETF's RFC 4890 on ICMP filtering, but has this per chance been applied to a pfSense setup before? I.e. can I use someone else's rules as a basis for ICMPv6 filtering? Maybe there should be a RFC 4890 compliant set of rules defined as a preset in pfSense?

How do you know that the pings from the remote side aren't coming through by the way? SIXXS supplies this testing tool: http://www.sixxs.net/tools/traceroute/ but it seems to be down for the moment… You say: "Currently I could not open a rule to ICMPv6 ping since it is not on the protocols list to choose.", when choosing IPv6 as TCP/IP Version and then choosing ICMP as protocol; I would assume this filters ICMPv6 traffic and not ICMPv4. I've added a rule like this from ALL sources and pnzone.net is able to ping me using ICMPv6, so I think it's working. Are you reloading the filter table after you create new rules? It should work...

You can use AYIYA tunnels without NAT as well? At least that's what I'm doing and it seems to be working (although perhaps not so stable, the next few days should point this out).

Luzemario

I saw the ping packet being blocked by the 'default rule' in the firewall log. Check "show packets blocked by default rule" to see most of packets dropped. You will see the pings coming from the remote side of the tunnel being blocked. For sixxs tunnels it can't happen.

ICMPv4 and ICMPv6 are (very) different protocols. There are several features of IPv6 wich depend of new things added to ICMPv6 (neighboor discovery, for example). I would not assume choosing IPv6 and ICMP being ICMPv6. It's make sense, but I cannot figure out why it has been dropped, so I guess better make no assumptions until someone from pFsense team can confirm.

I believe allowing ICMPv6 from remote side is a thing more sixxs-related than needed by some RFC. They use it in their credit balance account (they mean this in the FAQ). SixXS is free at all, but has a interesting virtual credit system. If you make your tunnel stable, you gain credits to 'buy' more subnets, etc… ::) is like a sort of 'game'...

I make a rule to allow IPV6 and ICMP, but the packets still get being dropped, and the drop is logged in firewall log. I don't know if this is a bug or if my interface is not choosen correctly in the rule. Firewall logs show WAN interface, not GIF (heartbeat is configured as GIF, not TUN). I did not apply your last patch to see the new GIF interface in webconfigurator yet, so I can't choose it in the rule setup.

I made no testing with AYIYA yet. AYIYA have a bunch of complexity to help to connect, but I personally do not like when it fails, since there is a bunch of things to debug too... :(

logion

You definitely need to allow icmpv6 traffic via the tunnel interface, not via the WAN interface (IPv6 traffic is encapsulated in IPv4 traffic when it enters via the WAN interface), therefore you need to create a OPT interface in the pfSense webinterface. This requires my patch to interfaces_assign.php if the tunnel is named tun0, if it is named gif (like in your case) it might not need to be patched; or you can replace tun with gif in my patch file ;)

Just to demonstrate that it's working over here, my pfSense firewall rules for my tun0 interface look like this: https://www.dropbox.com/s/ldxqrx6li4frckf/Screenshot%20from%202013-05-13%2022%3A47%3A15.png. The first rule allows SSH to the public IPv6 address of my pfSense box, the second address allows incoming ICMPv6 traffic from the tunnel POP's endpoint. Note that these addresses are in their own separate tunnel subnet (important)! I'm 100% sure that IPv6+ICMP results to ICMPv6. Executing "pfctl -sa | grep icmp" gives :
pass in quick on tun0 inet6 proto ipv6-icmp from 2a02:578:xxxxx::1 to 2a02:578:xxxxx::2 keep state label "USER_RULE: Allow tunnel's POP endpoint to ping gatekeeper"
Notice that proto says ipv6-icmp, while the interface is IPv6 + ICMP.

Finally executing "tcpdump -n -i tun0 -vv icmp6" on pfSense shows ICMP6 echo requests/replies from the POP to my pfSense box every minute (as well as neighbor sollications for the tunnels POP from my pfSense box, strange that pfSense doesn't cache this for sending the echo reply packets :s).

I am aware that ICMP6 differs from ICMP4 and that IPv6 relies heavily on ICMP6 for its workings (in regards to routers dropping packets that are larger than their link's MTUs allow, for instance). I also don't see any harm in allowing ICMP6 from the POP's endpoint to the pfSense box (they are conceptually in the same subnet after all). I was referring to a policy for incoming ICMPv6 traffic that will be forwarded to my LAN subnet (including my pfSense LAN interface, not really forwarded in this case though). In this case RFC 4890 might provide some good guidance. Guess I'll have to read up a bit more on this.

Luzemario

Hi logion,

Thank you very much for your help. My pFsense box is correctly configured and my GIF interface is answering pings from remote side of tunnel.

I will play with subnet now… :P

s0lid

I'm not quite sure why pfsense refuses to pass ICMPv6 from my POP to local End point on the tunnel if.
IPv6 tunnel works. I've tried to allow everything from POP address to End point address but no, RRD graphs show that there's constant IPv6 in-block for 409b/s which seems to be ICMPv6.

I've had this working earlier before i moved to routerboard, but had to come back to pfsense since ROS 6.1 managed to turn routerboard inaccessable… couple of times.

MerlinWV

By the way, thanks for posting this - it helped me set up a tunnel on 2.1 really quick.

Two problems left: the tunnel goes down after an undefined time, not passing any traffic. It seems to be realted to a tun-Problem in FreeBSD 8.0/8.1 . Should no longer present in 8.3, but sure show the same behaviour. Just upgraded to 2.1.1-PRERELEASE - let's see if that fixes it.

Also, I recommend disabling the pfscrub Option in Advanced / Firewall/NAT due to a bug in FreeBSD:
https://redmine.pfsense.org/issues/2762
Even if they talk about an IPv6-Options allow rule, probably the fix mentioned in
http://www.freebsd.org/cgi/query-pr.cgi?pr=172648
is better. I did not want to disable rfc1323 (TCP window scaling).

eri--

Why you guys need a tun device for this.

The SIXXS daemon is so damn simple that you just need a script to login and not need the daemon + the gif interface setup.

Also it is very simple to integrate this to pfSense.

If you do the php work on interfaces.php i will see to make this included in 2.2

Luzemario

Using AICCU is not required, but is throughly recommended. There are some requisites for keeping SIXXS tunnel up, as stated in https://www.sixxs.net/tools/heartbeat/, and AICCU takes care of them automatically.

Based on logion's instructions and some research by myself, I made a kind of 'tutorial' for installing pfSense and use SIXXS without headaches, in portuguese language. You can take a look on it. It was made for Brazilian readers, but I added a 'Google translator' button to help readers around the world. Is not the best, but at least… it can be helpful.

http://blog.luzemario.net.br/2013/12/ipv6-pfsense.html

HTH

MichelZ

Hi guys

I have the following problem with this configuration.
I've setup AICCU a while ago (when 2.1 was still in Beta), everything worked fine for a while.
Until recently, I don't know the exact cause, it might have been the update to 2.1.1 or 2.1.2.
I now get this when I try to ping any external IPv6 address from the pfsense Box:


ping6: sendmsg: No buffer space available
ping6: wrote 2001:1620:f00:f::1 16 chars, ret=-1

The Tunnel seems to be up:

./usr/local/sbin/sixxs-aiccu test

Tunnel Information for xxxxxx:
POP Id      : chzrh02
IPv6 Local  : 2001:1620:f00:f::2/64
IPv6 Remote : 2001:1620:f00:f::1/64
Tunnel Type : ayiya
Adminstate  : enabled
Userstate   : enabled

Also, the SixXs status page tells me the tunnel is up:

Tunnel State: up 
Tunnel Type: ayiya 
Last Heartbeat: 2014-04-25 07:52:35 (1398412355; 0 days 00:00:24 ago) 
Packet In: 2014-04-25 07:29:35 (1398410975; 0 days 00:23:24 ago) 
Packet Out: 2014-04-25 07:52:17 (1398412337; 0 days 00:00:42 ago)

Upon reboot, I can briefly ping the remote endpoint.
Firewall logs do not show any blocks.

I also want to mention that I have a second pfSense installation with the same configuration working well (upgraded to 2.1.2 as well)
I have tried re-installing the sixxs-aiccu package which did not help.

Unfortunately, I'm not a FreeBSD guru…

Any ideas on how I can troubleshoot that, or any helpful clues at all are very welcome :)