Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Trouble getting current mitmproxy working on 2.2.2

    Scheduled Pinned Locked Moved General pfSense Questions
    8 Posts 3 Posters 4.4k Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      paftdunk
      last edited by

      I'm trying to get mitmproxy-0.12.1 from http://mitmproxy.org working on pfSense 2.2.2. FreeBSD ports still includes mitmproxy-0.10.1, where Transparent mode isn't supported. I understand this was fixed in mitmproxy-0.11.

      I'm a noob at massaging build dependencies manually so I may well be missing some obvious -devel library.

      All attempts to either 'pip install mitmproxy' or to build from source fail at the same point.

      Here's netlib failing to build.

      running build_ext
building '_cffi__x88a8fd90x94e38fe2' extension
creating /tmp/pip-build-SrTHd4/netlib/netlib/__pycache__/netlib
creating /tmp/pip-build-SrTHd4/netlib/netlib/__pycache__/netlib/__pycache__
cc -fno-strict-aliasing -O2 -pipe -fstack-protector -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/local/include/python2.7 -c netlib/__pycache__/_cffi__x88a8fd90x94e38fe2.c -o /tmp/pip-build-SrTHd4/netlib/netlib/__pycache__/netlib/__pycache__/_cffi__x88a8fd90x94e38fe2.o -w
In file included from /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/syslimits.h:7:0,
                 from /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h:34,
                 from /usr/local/include/python2.7/Python.h:19,
                 from netlib/__pycache__/_cffi__x88a8fd90x94e38fe2.c:2:
/usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
compilation terminated.

      Full log: http://pastebin.com/YYzz9qCX

      (I've also tried with gcc-4.8.4 and gcc49-4.9.3, it didn't make a difference.)

      Here's cffi failing to build:
      http://pastebin.com/PRaDaFM1
      Here's the dump of all the packages I currently have installed.
      http://pastebin.com/Czt0F8cH

      edit: here's what I have installed via pip  http://pastebin.com/yiZ9jWVx

      Although I have py27-cffi-0.8.6_3 and libffi-3.2.1 installed, but neither seems to satisfy the dependency.

      I also filed this with the mitmproxy folks at
      https://github.com/mitmproxy/mitmproxy/issues/609
      Any pointers on getting this sorted would be greatly appreciated.

      1 Reply Last reply Reply Quote 0
      • F Offline
        firewalluser
        last edited by

        Have you looked for the limits.h file on your system in the folder/directory location its looking in?

        The GCC library is fairly common imo, dont know about freebsd but in Linux its common, so it could be missing from the library package and or the mitm script might need tweaking. Have you looked to see if limits.h is in a different location on your machine?

        I havent looked at your links as this machine I'm on only allows access to pfsense forum webpages out of the whole web.

        Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

        Asch Conformity, mainly the blind leading the blind.

        1 Reply Last reply Reply Quote 0
        • P Offline
          paftdunk
          last edited by

          @firewalluser:

          Have you looked for the limits.h file on your system in the folder/directory location its looking in?

          The GCC library is fairly common imo, dont know about freebsd but in Linux its common, so it could be missing from the library package and or the mitm script might need tweaking. Have you looked to see if limits.h is in a different location on your machine?

          The only limits.h I can find is /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h itself. The line referenced in the error is just the include:

          #ifdef _GCC_NEXT_LIMITS_H
#include_next <limits.h>                /* recurse down to the real one */
#endif</limits.h>

          @firewalluser:

          I havent looked at your links as this machine I'm on only allows access to pfsense forum webpages out of the whole web.

          Heh, very impressive. Given how much of the net I'm blocking now, I'm not far from that. Thanks for the response!

          1 Reply Last reply Reply Quote 0
          • F Offline
            firewalluser
            last edited by

            @paftdunk:

            The only limits.h I can find is /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h itself. The line referenced in the error is just the include:

            #ifdef _GCC_NEXT_LIMITS_H
#include_next <limits.h>                /* recurse down to the real one */
#endif</limits.h>

            Ok so the file exists, but in the code its just got #include_next which would suggest a path to the folder/directory might not exist. Might be worth checking the paths, alternatively what would happen if you put the full path into
            #include_next
            making it
            #include_next

            The error message No such file or directory is the clue at this stage.

            Heh, very impressive. Given how much of the net I'm blocking now, I'm not far from that. Thanks for the response!

            PC's can still communicate through DNS in ways and the same with other network protocols as you might find/work out in time.  ;)

            Google and others dont provide a DNS services just for everyone's pleasure.

            Edit. Phishing is also quite effective with multiple profiles as well.

            Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

            Asch Conformity, mainly the blind leading the blind.

            1 Reply Last reply Reply Quote 0
            • P Offline
              paftdunk
              last edited by

              Gotcha.

              Fwiw, I was able to uninstall nearly all of the extraneous cruft and get mitmproxy-0.12 working using python virtualenv.

              On both my pfsense box and on a virgin VMware copy of FreeBSD 10.1 from ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.1-RELEASE/amd64/Latest/

              I installed the following (pkg first, then python, then I believe py27-lxml, which pulls in most of the rest.)

              gettext-runtime-0.19.4
indexinfo-0.2.3
libffi-3.2.1
libgcrypt-1.6.3
libgpg-error-1.19_1
libxml2-2.9.2_2
libxslt-1.1.28_6
pkg-1.5.3
py27-lxml-3.4.1_1
py27-setuptools27-5.5.1_1
python-2.7_2,2
python2-2_3
python27-2.7.9_1

              Also on both pfSense and VMware I grabbed https://bootstrap.pypa.io/get-pip.py and ran 'python get-pip.py' to get that set up.

              I ran 'pip install virtualenv lxml' giving me the following with pip:

              lxml (3.4.4)
pip (7.0.3)
setuptools (17.0)
virtualenv (13.0.3)
wheel (0.24.0)

              Then in VMware:

              # mkdir mitmproxy_env
# virtualenv --no-site-packages mitmproxy_env
# source mitmproxy_env/bin/activate.csh

              At this point, we're inside the mitmproxy_env virtualenv (read more at https://virtualenv.pypa.io/en/latest/userguide.html.)

              pip install cryptography pyOpenSSL mitmproxy

              This should result in the following python modules installed:

              # pip list
backports.ssl-match-hostname (3.4.0.2)
blinker (1.3)
certifi (2015.4.28)
cffi (1.1.0)
ConfigArgParse (0.9.3)
cryptography (0.9)
enum34 (1.0.4)
hpack (1.0.1)
idna (2.0)
ipaddress (1.0.7)
lxml (3.4.4)
mitmproxy (0.12.1)
netlib (0.12.1)
passlib (1.6.2)
Pillow (2.8.1)
pip (7.0.3)
pyasn1 (0.1.7)
pycparser (2.13)
pyOpenSSL (0.15.1)
pyperclip (1.5.11)
setuptools (17.0)
six (1.9.0)
tornado (4.2)
urwid (1.3.0)
wheel (0.24.0)

              It should now be configured and operational inside VMware:

              # which mitmproxy
/root/mitm_env/bin/mitmproxy
# mitmproxy --version
mitmproxy 0.12.1
# mitmproxy -T --host

              After confirming that it does in fact run, type 'deactivate' to exit the virtualenv. At this point you can 'tar zcvf mitmproxy_env.tar.gz ./mitmproxy_env' and move the tarball over to the pfSense machine.

              On the pfSense box:

              # tar zxvf mitmproxy_env.tar.gz
# source mitmproxy_env/bin/activate.csh
# setenv LANG en_US.UTF-8 ; mitmproxy -T --host

              And it should be up and running. This lets you avoid setting up full dev toolchain and messing with build dependencies like I started out. Hopefully this will be of some use to others.

              1 Reply Last reply Reply Quote 0
              • F Offline
                firewalluser
                last edited by

                Thanks, this is an approach which will probably work for getting Dtrace working on pfsense.  :D

                Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                Asch Conformity, mainly the blind leading the blind.

                1 Reply Last reply Reply Quote 0
                • S Offline
                  Supermule Banned
                  last edited by

                  Fookme…

                  This is over my head.

                  #sigh

                  1 Reply Last reply Reply Quote 0
                  • F Offline
                    firewalluser
                    last edited by

                    Its not hard, just ask the right questions, give it a go.  ;)

                    What dont you know?

                    Edit:

                    Worth starting a getting Dtrace to work on pfsense thread at this point?

                    Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                    Asch Conformity, mainly the blind leading the blind.

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.