Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Support for GENEVE protocol (can use in AWS as Gateway Load Balancer then)

    Scheduled Pinned Locked Moved General pfSense Questions
    20 Posts 4 Posters 3.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      You are using Open VSwitch for this?

      F 1 Reply Last reply Reply Quote 0
      • F
        forbiddenera @stephenw10
        last edited by

        @stephenw10 said in Support for GENEVE protocol (can use in AWS as Gateway Load Balancer then):

        You are using Open VSwitch for this?

        No; not right now.. initially I figured that might be a decent way forward, then I looked into it a bit closer (especially the the provided code from AWS) and realized a few things about how it's been implemented here, my initial thoughts were mostly assumption based knowing that OVS could potentially do it..

        So, while OVS apparently supports GENEVE, at least for AWS GWLB, it seems we need to (should?) parse the TLV options from the GENEVE packets as they contain I think some extra information like the ENI used for the GLWBe and stuff and I think the options TLV that AWS is using is potentially specific/proprietary for AWS, I'm not sure how required this information is but I believe this is partly/mainly why AWS provided the linked sample, as from my research it seems the sample code will create two tunnel interfaces, one for ingress, one for egress. It then starts listening on UDP for the GENEVE packets from the GLWBe, parses their TLV options and pushes them onto the ingress tunnel. Once the packets come back after forwarding/IPS/etc on the egress tunnel, it repacks them as GENEVE packets and pushes them out to the GLWBe for continuing along their path through the GLWBe to the which would decode them before pushing them to the final hop, I think.

        I can't say that this is 100% what happens but this seems to be the case from the research that I've done.

        I'd initially hoped that simply setting up some OVS capable of GENEVE would be enough but I'm not entirely sure that's the case after looking into the AWS sample code more, I also figured that it would be the simpler method forward and originally didn't have much faith in any AWS exsample code from past experience, however, after looking at this code, I've got no particular complaints other than the fact that C++ has changed quite a bit since the last time I really used it, but that was like 20 years ago!

        I was able to get the AWS exsample code to build on FreeBSD, albeit with warnings and of course building doesn't mean working.

        The main differences I believe I need to reconcile now are the differences in how Linux and FreeBSD handle creating tunnels, especially under C/C++, as I mentioned previously, it'd potentially be possible to create/destroy the tunnel interfaces by execing ifconfig but I feel like that's a bit of a cop out and I'm sure is not something that would ever get PR'd as a reasonable porting attempt anyhow.

        Fortunately, there's a ton of opensource software out there that uses/creates tunnels (as well as deals with UDP sockets) that's portable for both Linux and FreeBSD as well as others, so nothing that the AWS exsample is doing is particularly unique.

        Once the code AWS provided runs on FreeBSD, two tunnel interfaces should be created that we can manage and do firewall/IPS/whatever we want with pfSense. It'll also handle the health checks and such that AWS does to ensure the GWLB 'appliance' (being the VM that you're using for GWLB, in our case, our goal being to use pfSense as such an appliance) acts and responds the way the AWS GWLB system is expecting.

        Beyond that, if I get it working, I'll consider packaging it up as a pfSense package that can install the service and potentially log it and manage any configuration settings related to it.

        I can't say that I've gotten a 100% grasp on everything here but this is how I understand it so far, if I'm mistaken on anything I'm open to any corrections or direction to the proper information, of course.

        I spent last night brushing up on my C++ and learning a bunch of the newer language features that I wasn't familiar with that are used in the provided code.

        Looking at differences between tunnel creation code for FreeBSD and what's here for Linux, I've noticed that some of the flags it uses (eg IFF_UP, IFF_RUNNING) don't exist on FreeBSD and it also names the interfaces which isn't possible on FreeBSD either it seems. I'm not sure if the interface is automatically 'up/running' when created on FreeBSD, but my next effort is into consolidating the differences relating to that as well as opening up some debugging as currently it just core dumps immediately (except when doing a --help, heh).

        I'm not sure if or how it'd be possible for OVS to parse the AWS TLV options out of the GENEVE packet or if it'd be possible to work without that information, though I could at least see it being useful for logging. If it's not required in any way, then perhaps having OVS do the de/encapsulation alone might work as well but as I've already made progress with this code and I've not actually attempted any OVS stuff w/pfSense before, I'm going to continue down this route for now - it does seem easier to run a small service to handle it than trying to setup OVS under pfSense especially if only used for this purpose and nothing else.

        F 1 Reply Last reply Reply Quote 1
        • F
          forbiddenera @forbiddenera
          last edited by

          No more coredump .. was just an exception but clang was dumping instead of throwing anything useful, out of curiosity I switched to g++, actually thought cmake had picked up g++ in the first place.

          The coredump was due to SO_REUSE_PORT not being a thing on FreeBSD I guess..

          So I'm at the point where it seems to run and get into the main loop; again I definitely still need to work out the tunnel creation code as well, AFAICT the code doesn't attempt to create a tunnel even until it hears from a GWLBe first, otherwise I'm sure I'd have hit a further crash/exception already.

          However, now I'm at the point that unless I want to start really breaking up things to test that I should actually dump this on to an AWS environment and point a GWLB to a FreeBSD instance and go from there, currently the instance I'm using is just a vm on my local xen box...

          This is technically slightly more progress, but not anything I'd celebrate over yet.. ;-) But for someone who hasn't touched C++ in years and was sort of shocked that C++ has had so many additions/changes, the fact I can get it to compile and run even remotely is more progress than I thought I would've made yet.. (I haven't run into any C code in the last 20 years that made me scratch my head, but this C++ is different, namespaces, while existent, wasn't even really used much/at all last time I was really using it..)

          F 1 Reply Last reply Reply Quote 0
          • F
            forbiddenera @forbiddenera
            last edited by

            @forbiddenera
            The coredump was due to SO_REUSE_PORT not being a thing on FreeBSD I guess..

            should be (after further looking)

            The coredump was due to not being able to set multiple flags in a single call on a socket in FreeBSD, I guess..

            1 Reply Last reply Reply Quote 0
            • T
              taliesins
              last edited by

              @forbiddenera it sounds like you making some excellent progress. Would be cool if you could share your attempts on github. Might even pickup a few guys that want to help.

              I am totally willing to be a alpha tester and have access to as much AWS as we need.

              For TUN name generation are you using the GWLBE/VPCE ID?

              J 1 Reply Last reply Reply Quote 0
              • J
                jazzl0ver @taliesins
                last edited by

                @taliesins @forbiddenera do you have any news on this? seems to be a very desirable approach for aws workloads

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  How exactly would you expect to use this with pfSense if it supported it? I'm unclear what the use-case is here.

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    jazzl0ver @stephenw10
                    last edited by jazzl0ver

                    @stephenw10 the main use case is traffic inspection. we could use pfsense capabilities like Snort/Suricata or pfBlockerNG to inspect/filter traffic directed to or from the cloud workloads.

                    Here are the articles covering the architectures in details:
                    https://aws.amazon.com/blogs/aws/introducing-aws-gateway-load-balancer-easy-deployment-scalability-and-high-availability-for-partner-appliances/
                    https://aws.amazon.com/blogs/networking-and-content-delivery/how-to-integrate-third-party-firewall-appliances-into-an-aws-environment/
                    https://aws.amazon.com/blogs/networking-and-content-delivery/scaling-network-traffic-inspection-using-aws-gateway-load-balancer/
                    https://aws.amazon.com/blogs/networking-and-content-delivery/centralized-inspection-architecture-with-aws-gateway-load-balancer-and-aws-transit-gateway/

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      We are looking into this now. It would be a non-trivial task though so no promises at this time.

                      F 1 Reply Last reply Reply Quote 2
                      • F
                        forbiddenera @stephenw10
                        last edited by

                        @stephenw10 That would be AWESOME if it was supported by pfSense natively. If you do begin work on this, please let me know ASAP, I'd be happy to share what I've got and otherwise there's no sense in me working on a feature that will get implemented natively, well, I mean there's no sense in us both working on it!

                        That said I am still working on my implementation - I've had some other things take priority recently but hopefully will have some time to dedicate shortly to it.

                        As you say, it's not a trivial task - just porting the AWS example to FreeBSD alone isn't trivial, letalone anything else.

                        Unfortunately my pfSense instances on AWS will no longer update; but that's a separate off-topic issue I'll have to raise otherwise.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.