Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Support for GENEVE protocol (can use in AWS as Gateway Load Balancer then)

    General pfSense Questions
    4
    20
    2.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      forbiddenera
      last edited by

      Can't edit my post anymore so..

      Perhaps the AWS sample is more useful than I thought..

      After some more reading and digging I realized this works differently than I originally assumed and perhaps could be ported without too much effort and without needing OVS either...

      I originally thought it was implementing it's own bridge/tun interface driver kind of thing but it's just creating regular tun interfaces..

      Seems it's just UDP listening for and parsing GENEVE packets, creating tun interfaces on the first packet received, pulling extra AWS metadata out of the options TLV and I assume stuffing ingress tun with the encaps'd packets and then pulling packets from the egress tun and encapsing and pushing them out to the next hop ..

      F 1 Reply Last reply Reply Quote 0
      • F
        forbiddenera @forbiddenera
        last edited by forbiddenera

        Making progress on porting.. just stuck on passing proper destination address to recvDispatcher() because I haven't done c socket coding in like 20+ years + never had to make something work on both *BSD+Linux when I did

        edit: past that now, onto tunnel interface differences, which might be a bit harder but fortunately there should be lots of googleable code for that (vpn software as an example)

        F 1 Reply Last reply Reply Quote 0
        • F
          forbiddenera @forbiddenera
          last edited by forbiddenera

          Got it to build..! (with a hammer)

          There's warnings and it doesn't work yet (insta coredump) but it's some progress..

          I'm almost tempted (for simplicity and laziness) to re-implement this from scratch in something I'm more familiar with like nodejs, the tuns even can seemingly be be created with ifconfig alone and otherwise it's just listening on udp, parsing the packet and reading/writing to/from to the tun fd..

          However, while pfSense has php (I could do it in php! lol), python and perl available, there's no node and dealing with that is a whole thing on it's own.. If I'm going to get this working and possibly package it up as a pfSense plugin/package then that's not something people are going to want to deal with either, probably even if it was included in the package.

          That and, considering the use-cases for this, I'm not sure any interpreted language is really going to be a performant enough solution - I can't say for sure as it shouldn't be too complicated to exec some ifconfig commands, listen to a socket and read/write to/from a fd but there's always going to be overhead compared to c/c++ and this could potentially be used in places where it needs to handle as many packets as possible.. Though I do think reasonable performance could be achieved with node especially when bytecode cached/optimized but even so that's not really ideal for a pfSense enviroment.

          Now, as I said, it was built with a hammer - I think some parts will work like the UDP server and parsing of the packets but I don't think my hammering will work for the tunnel creation so I'm sure there's more work there but as previously mentioned there's plenty of things that do portable tunnel creation.. worst case I could create the tunnel executing some ifconfig commands on FreeBSD but while this might be easier to implement I feel it's a bit of a cop out especially if I'm #ifdef'ing it in beside the C++ solution for Linux - I mean I said some AWS exsamples I've seen before were bad (and, actually, this aws sample repo gets no complaints from me other than not being portable so it definitely doesn't fall into that category - I'd almost call it less of an exsample than a simple, usable implementation for Linux that could be used as a starting point for something more complicated or otherwise demonstrating the process for implementation on proprietary stacks) - I couldn't do something like that (beyond maybe testing) and not be a hypocrite deserving of a slap, just like doing this in an interpreted language feels like.

          Not making any promises on whether I'll get this working or when but as I'd like to use it for infrastructure I'm WIP on deploying.

          If I get it working based off the AWS code, I'll fork it on my github and post the link and perhaps do a PR to the original repo. That's only part of it though and I'd also like to make a pfSense package to set it up and stuff.. but if it was working on FreeBSD, I think a motivated person could build it (ideally on a separate FreeBSD box like I am, unless you want to deal with setting up extra pkgs and build tools on your pfSense (or at worst, a separate dev pfSense install) - I personally try my best to avoid enabling the freebsd pkg repo as it can break stuff - IIRC this is best practice anyway and mentioned in the docs as well), run it on the shell and then manage the created tun interfaces w/pfSense from there.

          WML

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            You are using Open VSwitch for this?

            F 1 Reply Last reply Reply Quote 0
            • F
              forbiddenera @stephenw10
              last edited by

              @stephenw10 said in Support for GENEVE protocol (can use in AWS as Gateway Load Balancer then):

              You are using Open VSwitch for this?

              No; not right now.. initially I figured that might be a decent way forward, then I looked into it a bit closer (especially the the provided code from AWS) and realized a few things about how it's been implemented here, my initial thoughts were mostly assumption based knowing that OVS could potentially do it..

              So, while OVS apparently supports GENEVE, at least for AWS GWLB, it seems we need to (should?) parse the TLV options from the GENEVE packets as they contain I think some extra information like the ENI used for the GLWBe and stuff and I think the options TLV that AWS is using is potentially specific/proprietary for AWS, I'm not sure how required this information is but I believe this is partly/mainly why AWS provided the linked sample, as from my research it seems the sample code will create two tunnel interfaces, one for ingress, one for egress. It then starts listening on UDP for the GENEVE packets from the GLWBe, parses their TLV options and pushes them onto the ingress tunnel. Once the packets come back after forwarding/IPS/etc on the egress tunnel, it repacks them as GENEVE packets and pushes them out to the GLWBe for continuing along their path through the GLWBe to the which would decode them before pushing them to the final hop, I think.

              I can't say that this is 100% what happens but this seems to be the case from the research that I've done.

              I'd initially hoped that simply setting up some OVS capable of GENEVE would be enough but I'm not entirely sure that's the case after looking into the AWS sample code more, I also figured that it would be the simpler method forward and originally didn't have much faith in any AWS exsample code from past experience, however, after looking at this code, I've got no particular complaints other than the fact that C++ has changed quite a bit since the last time I really used it, but that was like 20 years ago!

              I was able to get the AWS exsample code to build on FreeBSD, albeit with warnings and of course building doesn't mean working.

              The main differences I believe I need to reconcile now are the differences in how Linux and FreeBSD handle creating tunnels, especially under C/C++, as I mentioned previously, it'd potentially be possible to create/destroy the tunnel interfaces by execing ifconfig but I feel like that's a bit of a cop out and I'm sure is not something that would ever get PR'd as a reasonable porting attempt anyhow.

              Fortunately, there's a ton of opensource software out there that uses/creates tunnels (as well as deals with UDP sockets) that's portable for both Linux and FreeBSD as well as others, so nothing that the AWS exsample is doing is particularly unique.

              Once the code AWS provided runs on FreeBSD, two tunnel interfaces should be created that we can manage and do firewall/IPS/whatever we want with pfSense. It'll also handle the health checks and such that AWS does to ensure the GWLB 'appliance' (being the VM that you're using for GWLB, in our case, our goal being to use pfSense as such an appliance) acts and responds the way the AWS GWLB system is expecting.

              Beyond that, if I get it working, I'll consider packaging it up as a pfSense package that can install the service and potentially log it and manage any configuration settings related to it.

              I can't say that I've gotten a 100% grasp on everything here but this is how I understand it so far, if I'm mistaken on anything I'm open to any corrections or direction to the proper information, of course.

              I spent last night brushing up on my C++ and learning a bunch of the newer language features that I wasn't familiar with that are used in the provided code.

              Looking at differences between tunnel creation code for FreeBSD and what's here for Linux, I've noticed that some of the flags it uses (eg IFF_UP, IFF_RUNNING) don't exist on FreeBSD and it also names the interfaces which isn't possible on FreeBSD either it seems. I'm not sure if the interface is automatically 'up/running' when created on FreeBSD, but my next effort is into consolidating the differences relating to that as well as opening up some debugging as currently it just core dumps immediately (except when doing a --help, heh).

              I'm not sure if or how it'd be possible for OVS to parse the AWS TLV options out of the GENEVE packet or if it'd be possible to work without that information, though I could at least see it being useful for logging. If it's not required in any way, then perhaps having OVS do the de/encapsulation alone might work as well but as I've already made progress with this code and I've not actually attempted any OVS stuff w/pfSense before, I'm going to continue down this route for now - it does seem easier to run a small service to handle it than trying to setup OVS under pfSense especially if only used for this purpose and nothing else.

              F 1 Reply Last reply Reply Quote 1
              • F
                forbiddenera @forbiddenera
                last edited by

                No more coredump .. was just an exception but clang was dumping instead of throwing anything useful, out of curiosity I switched to g++, actually thought cmake had picked up g++ in the first place.

                The coredump was due to SO_REUSE_PORT not being a thing on FreeBSD I guess..

                So I'm at the point where it seems to run and get into the main loop; again I definitely still need to work out the tunnel creation code as well, AFAICT the code doesn't attempt to create a tunnel even until it hears from a GWLBe first, otherwise I'm sure I'd have hit a further crash/exception already.

                However, now I'm at the point that unless I want to start really breaking up things to test that I should actually dump this on to an AWS environment and point a GWLB to a FreeBSD instance and go from there, currently the instance I'm using is just a vm on my local xen box...

                This is technically slightly more progress, but not anything I'd celebrate over yet.. ;-) But for someone who hasn't touched C++ in years and was sort of shocked that C++ has had so many additions/changes, the fact I can get it to compile and run even remotely is more progress than I thought I would've made yet.. (I haven't run into any C code in the last 20 years that made me scratch my head, but this C++ is different, namespaces, while existent, wasn't even really used much/at all last time I was really using it..)

                F 1 Reply Last reply Reply Quote 0
                • F
                  forbiddenera @forbiddenera
                  last edited by

                  @forbiddenera
                  The coredump was due to SO_REUSE_PORT not being a thing on FreeBSD I guess..

                  should be (after further looking)

                  The coredump was due to not being able to set multiple flags in a single call on a socket in FreeBSD, I guess..

                  1 Reply Last reply Reply Quote 0
                  • T
                    taliesins
                    last edited by

                    @forbiddenera it sounds like you making some excellent progress. Would be cool if you could share your attempts on github. Might even pickup a few guys that want to help.

                    I am totally willing to be a alpha tester and have access to as much AWS as we need.

                    For TUN name generation are you using the GWLBE/VPCE ID?

                    J 1 Reply Last reply Reply Quote 0
                    • J
                      jazzl0ver @taliesins
                      last edited by

                      @taliesins @forbiddenera do you have any news on this? seems to be a very desirable approach for aws workloads

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        How exactly would you expect to use this with pfSense if it supported it? I'm unclear what the use-case is here.

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          jazzl0ver @stephenw10
                          last edited by jazzl0ver

                          @stephenw10 the main use case is traffic inspection. we could use pfsense capabilities like Snort/Suricata or pfBlockerNG to inspect/filter traffic directed to or from the cloud workloads.

                          Here are the articles covering the architectures in details:
                          https://aws.amazon.com/blogs/aws/introducing-aws-gateway-load-balancer-easy-deployment-scalability-and-high-availability-for-partner-appliances/
                          https://aws.amazon.com/blogs/networking-and-content-delivery/how-to-integrate-third-party-firewall-appliances-into-an-aws-environment/
                          https://aws.amazon.com/blogs/networking-and-content-delivery/scaling-network-traffic-inspection-using-aws-gateway-load-balancer/
                          https://aws.amazon.com/blogs/networking-and-content-delivery/centralized-inspection-architecture-with-aws-gateway-load-balancer-and-aws-transit-gateway/

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            We are looking into this now. It would be a non-trivial task though so no promises at this time.

                            F 1 Reply Last reply Reply Quote 2
                            • F
                              forbiddenera @stephenw10
                              last edited by

                              @stephenw10 That would be AWESOME if it was supported by pfSense natively. If you do begin work on this, please let me know ASAP, I'd be happy to share what I've got and otherwise there's no sense in me working on a feature that will get implemented natively, well, I mean there's no sense in us both working on it!

                              That said I am still working on my implementation - I've had some other things take priority recently but hopefully will have some time to dedicate shortly to it.

                              As you say, it's not a trivial task - just porting the AWS example to FreeBSD alone isn't trivial, letalone anything else.

                              Unfortunately my pfSense instances on AWS will no longer update; but that's a separate off-topic issue I'll have to raise otherwise.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.