• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Moving the pfSense® Documentation to GitHub

Scheduled Pinned Locked Moved Documentation
8 Posts 4 Posters 3.3k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    jdillard
    last edited by Jun 4, 2018, 10:52 PM

    We've moved the documentation from mediaWiki to GitHub. More information can be found on the following blog post: https://www.netgate.com/blog/moving-the-pfsense-documentation-to-github.html.

    There is still some clean up left to do from the conversion, so if you find anything content that broke in the process feel free to post the findings here.

    1 Reply Last reply Reply Quote 2
    • L
      lsilverman
      last edited by Jun 6, 2018, 2:14 PM

      I have a similar doc site I would like to convert from Mediawiki to GitHub. How did you execute the actual conversion of Mediawiki pages? What tooling did you use?

      Thanks.

      1 Reply Last reply Reply Quote 0
      • J
        jdillard
        last edited by jdillard Jun 6, 2018, 5:54 PM Jun 6, 2018, 3:01 PM

        I used a php script to grab all the wiki pages from the DB and save them as flat files (note: I forgot to add the title at the top of each page in this version):

        <?php
        $servername = "localhost";
        $username = "user";
        $password = "pa$$";
        $dbname = "mediaWiki";
        
        function slugify($text)
        {
          // replace non letter or digits by -
          $text = preg_replace('~[^\pL\d]+~u', '-', $text);
        
          // remove unwanted characters
          $text = preg_replace('~[^-\w]+~', '', $text);
        
          // trim
          $text = trim($text, '-');
        
          // remove duplicate -
          $text = preg_replace('~-+~', '-', $text);
        
          // lowercase
          $text = strtolower($text);
        
          if (empty($text)) {
            return 'n-a';
          }
        
          return $text;
        }
        
        // Create connection
        $conn = new mysqli($servername, $username, $password, $dbname);
        
        // Check connection
        if ($conn->connect_error) {
            die("Connection failed: " . $conn->connect_error);
        } 
        
        $sql = "SELECT page_title, page_touched, old_text FROM revision,page,text WHERE revision.rev_id=page.page_latest AND text.old_id=revision.rev_text_id AND page.page_namespace=0 AND substring(text.old_text,2,8) NOT IN ('REDIRECT')";
        $result = $conn->query($sql);
        
        if ($result->num_rows > 0) {
            // output data of each row
            while($row = $result->fetch_assoc()) {
                $myfile = fopen(slugify($row["page_title"]).".mw", "w") or die("Unable to open file!");
                fwrite($myfile, $row["old_text"]);
                fclose($myfile);
            }
        } else {
            echo "0 results";
        }
        $conn->close();
        
        ?>
        

        I also found some commands on StackOverflow to download all the images as a zip file.

        I used pandoc to convert the mediaWiki syntax to RST syntax (you could do markdown or whatever here and go in a different direction):

        files=($(find . -type f -name '*.mw'))
        for item in ${files[*]}
        do
          filename=${item##*/}
          #printf "   %s\n" $filename
          
          pandoc $filename -f mediawiki -t rst -o ./output/${filename%.*}.rst || {  printf "   %s conversion failed\n" $filename ; }
        done
        

        Then massaged all that into the desired sphinx formatting that I wanted...it took several custom python/bash scripts to clean up the pandoc conversion (it isn't perfect).

        Then I built the sphinx docs as HTML and ran the npm package broken-link-checker-local against it to check for broken links (more python scripts involved to fix them).

        I also used git as a backup so I could git checkout if my scripts blew anything up along the way.

        That's about all the advise I can offer... It's a lot of work, but worth it in the end. Good Luck!

        1 Reply Last reply Reply Quote 0
        • L
          lsilverman
          last edited by Jun 6, 2018, 3:06 PM

          Great, thanks for sharing!

          How did you come to choose Sphinx over the other many possible options (Jekyll, Hugo, many other choices)?

          1 Reply Last reply Reply Quote 0
          • J
            jdillard
            last edited by jdillard Jun 6, 2018, 3:34 PM Jun 6, 2018, 3:21 PM

            I'd say the big reason is that Sphinx (restructuredText) has a syntax geared towards documentation (info/warning boxes, etc) so we didn't have to adopt a custom markdown flavor to achieve the same results. We also have the option to easily build other variants such as PDFs, man pages, texinfo, etc should we choose. In a perfect world, I would prefer if there was a "standard" markdown flavor that existed for documentation that mimics the key directives of the RST syntax. The markdown ecosystem is getting there, but until then sphinx works for us.

            We currently use Jekyll (possibly Gatsby in the future) for www.netgate.com and www.pfsense.org so we are familiar with those SSGs as well.

            1 Reply Last reply Reply Quote 0
            • J
              jimp Rebel Alliance Developer Netgate
              last edited by Jun 6, 2018, 3:53 PM

              Plus we've been using Sphinx for years for the pfSense book and other internal documentation, so it made sense to keep the converted wiki consistent.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • S
                ScionIV
                last edited by Jun 12, 2018, 9:36 PM

                There used to be documentation for installing pfSense on watchguard firebox appliances; this seems to have disappeared. I had bookmarked the link at https://doc.pfsense.org/index.php/PfSense_on_Watchguard_Firebox which now redirects to your docs page, and I can't seem to find it via your search box. What's happened here?

                1 Reply Last reply Reply Quote 0
                • J
                  jdillard
                  last edited by Jun 13, 2018, 2:40 PM

                  Some of the pages were considered outdated, redundant, or misplaced and are still being manually organized and that might be one of them. We are still working on cleaning up everything from the move. Until then you may want to use the web archive: https://web.archive.org/web/20171205175929/https://doc.pfsense.org/index.php/PfSense_on_Watchguard_Firebox

                  1 Reply Last reply Reply Quote 0
                  8 out of 8
                  • First post
                    8/8
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                    This community forum collects and processes your personal information.
                    consent.not_received