Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    HAProxy: serve static robots.txt from Files section

    Scheduled Pinned Locked Moved Cache/Proxy
    1 Posts 1 Posters 94 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      decibel83
      last edited by

      I am using HAProxy 2.4.9-f8dcd9f on pfSense 2.6.0, although I am planning to upgrade to pfSense 2.8.0 within the next two weeks.

      I need to serve a static robots.txt file for every request to /robots.txt on an HAProxy frontend, so I created the robots.txt file in the Files section:

      Screenshot 2026-03-30 at 16.19.36.png

      Now I am having trouble creating the correct rule to serve that file for every GET /robots.txt request.

      If I create a Robots_File backend with the following rule, it works:

      http-request return status 200 content-type "text/plain; charset=utf-8" hdr Cache-Control "no-store, max-age=0" lf-string "User-agent: *\nDisallow: /\n"
      

      However, since I need a much more complex robots.txt, I would prefer to serve the static file instead of embedding the file content directly in the rule.

      Could someone please help me understand what the correct HAProxy rule would be in this case?

      Thank you very much.

      1 Reply Last reply Reply Quote 0
      • First post
        Last post
      Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.