Trouble getting current mitmproxy working on 2.2.2



  • I'm trying to get mitmproxy-0.12.1 from http://mitmproxy.org working on pfSense 2.2.2. FreeBSD ports still includes mitmproxy-0.10.1, where Transparent mode isn't supported. I understand this was fixed in mitmproxy-0.11.

    I'm a noob at massaging build dependencies manually so I may well be missing some obvious -devel library.

    All attempts to either 'pip install mitmproxy' or to build from source fail at the same point.

    Here's netlib failing to build.

    running build_ext
    building '_cffi__x88a8fd90x94e38fe2' extension
    creating /tmp/pip-build-SrTHd4/netlib/netlib/__pycache__/netlib
    creating /tmp/pip-build-SrTHd4/netlib/netlib/__pycache__/netlib/__pycache__
    cc -fno-strict-aliasing -O2 -pipe -fstack-protector -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/local/include/python2.7 -c netlib/__pycache__/_cffi__x88a8fd90x94e38fe2.c -o /tmp/pip-build-SrTHd4/netlib/netlib/__pycache__/netlib/__pycache__/_cffi__x88a8fd90x94e38fe2.o -w
    In file included from /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/syslimits.h:7:0,
                     from /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h:34,
                     from /usr/local/include/python2.7/Python.h:19,
                     from netlib/__pycache__/_cffi__x88a8fd90x94e38fe2.c:2:
    /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
    compilation terminated.
    

    Full log: http://pastebin.com/YYzz9qCX

    (I've also tried with gcc-4.8.4 and gcc49-4.9.3, it didn't make a difference.)

    Here's cffi failing to build:
    http://pastebin.com/PRaDaFM1
    Here's the dump of all the packages I currently have installed.
    http://pastebin.com/Czt0F8cH

    edit: here's what I have installed via pip  http://pastebin.com/yiZ9jWVx

    Although I have py27-cffi-0.8.6_3 and libffi-3.2.1 installed, but neither seems to satisfy the dependency.

    I also filed this with the mitmproxy folks at
    https://github.com/mitmproxy/mitmproxy/issues/609
    Any pointers on getting this sorted would be greatly appreciated.



  • Have you looked for the limits.h file on your system in the folder/directory location its looking in?

    The GCC library is fairly common imo, dont know about freebsd but in Linux its common, so it could be missing from the library package and or the mitm script might need tweaking. Have you looked to see if limits.h is in a different location on your machine?

    I havent looked at your links as this machine I'm on only allows access to pfsense forum webpages out of the whole web.



  • @firewalluser:

    Have you looked for the limits.h file on your system in the folder/directory location its looking in?

    The GCC library is fairly common imo, dont know about freebsd but in Linux its common, so it could be missing from the library package and or the mitm script might need tweaking. Have you looked to see if limits.h is in a different location on your machine?

    The only limits.h I can find is /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h itself. The line referenced in the error is just the include:

    #ifdef _GCC_NEXT_LIMITS_H
    #include_next <limits.h>                /* recurse down to the real one */
    #endif</limits.h>
    

    @firewalluser:

    I havent looked at your links as this machine I'm on only allows access to pfsense forum webpages out of the whole web.

    Heh, very impressive. Given how much of the net I'm blocking now, I'm not far from that. Thanks for the response!



  • @paftdunk:

    The only limits.h I can find is /usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd10.1/5.1.0/include-fixed/limits.h itself. The line referenced in the error is just the include:

    #ifdef _GCC_NEXT_LIMITS_H
    #include_next <limits.h>                /* recurse down to the real one */
    #endif</limits.h>
    

    Ok so the file exists, but in the code its just got #include_next which would suggest a path to the folder/directory might not exist. Might be worth checking the paths, alternatively what would happen if you put the full path into
    #include_next
    making it
    #include_next

    The error message No such file or directory is the clue at this stage.

    Heh, very impressive. Given how much of the net I'm blocking now, I'm not far from that. Thanks for the response!

    PC's can still communicate through DNS in ways and the same with other network protocols as you might find/work out in time.  ;)

    Google and others dont provide a DNS services just for everyone's pleasure.

    Edit. Phishing is also quite effective with multiple profiles as well.



  • Gotcha.

    Fwiw, I was able to uninstall nearly all of the extraneous cruft and get mitmproxy-0.12 working using python virtualenv.

    On both my pfsense box and on a virgin VMware copy of FreeBSD 10.1 from ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.1-RELEASE/amd64/Latest/

    I installed the following (pkg first, then python, then I believe py27-lxml, which pulls in most of the rest.)

    gettext-runtime-0.19.4
    indexinfo-0.2.3
    libffi-3.2.1
    libgcrypt-1.6.3
    libgpg-error-1.19_1
    libxml2-2.9.2_2
    libxslt-1.1.28_6
    pkg-1.5.3
    py27-lxml-3.4.1_1
    py27-setuptools27-5.5.1_1
    python-2.7_2,2
    python2-2_3
    python27-2.7.9_1
    

    Also on both pfSense and VMware I grabbed https://bootstrap.pypa.io/get-pip.py and ran 'python get-pip.py' to get that set up.

    I ran 'pip install virtualenv lxml' giving me the following with pip:

    lxml (3.4.4)
    pip (7.0.3)
    setuptools (17.0)
    virtualenv (13.0.3)
    wheel (0.24.0)
    
    

    Then in VMware:

    # mkdir mitmproxy_env
    # virtualenv --no-site-packages mitmproxy_env
    # source mitmproxy_env/bin/activate.csh
    
    

    At this point, we're inside the mitmproxy_env virtualenv (read more at https://virtualenv.pypa.io/en/latest/userguide.html.)

    pip install cryptography pyOpenSSL mitmproxy
    

    This should result in the following python modules installed:

    # pip list
    backports.ssl-match-hostname (3.4.0.2)
    blinker (1.3)
    certifi (2015.4.28)
    cffi (1.1.0)
    ConfigArgParse (0.9.3)
    cryptography (0.9)
    enum34 (1.0.4)
    hpack (1.0.1)
    idna (2.0)
    ipaddress (1.0.7)
    lxml (3.4.4)
    mitmproxy (0.12.1)
    netlib (0.12.1)
    passlib (1.6.2)
    Pillow (2.8.1)
    pip (7.0.3)
    pyasn1 (0.1.7)
    pycparser (2.13)
    pyOpenSSL (0.15.1)
    pyperclip (1.5.11)
    setuptools (17.0)
    six (1.9.0)
    tornado (4.2)
    urwid (1.3.0)
    wheel (0.24.0)
    
    

    It should now be configured and operational inside VMware:

    # which mitmproxy
    /root/mitm_env/bin/mitmproxy
    # mitmproxy --version
    mitmproxy 0.12.1
    # mitmproxy -T --host
    

    After confirming that it does in fact run, type 'deactivate' to exit the virtualenv. At this point you can 'tar zcvf mitmproxy_env.tar.gz ./mitmproxy_env' and move the tarball over to the pfSense machine.

    On the pfSense box:

    # tar zxvf mitmproxy_env.tar.gz
    # source mitmproxy_env/bin/activate.csh
    # setenv LANG en_US.UTF-8 ; mitmproxy -T --host
    

    And it should be up and running. This lets you avoid setting up full dev toolchain and messing with build dependencies like I started out. Hopefully this will be of some use to others.



  • Thanks, this is an approach which will probably work for getting Dtrace working on pfsense.  :D


  • Banned

    Fookme…

    This is over my head.

    #sigh



  • Its not hard, just ask the right questions, give it a go.  ;)

    What dont you know?

    Edit:

    Worth starting a getting Dtrace to work on pfsense thread at this point?


Log in to reply