Unofficial E2guardian package for pfSense
-
pkg v 5.0.3 is out. Main changes on this version:
-
Apply firewall limiters to direct connect clients (wpad or proxy manual config).
-
New button to clear cert MITM cache data.(useful when you change MITM CA)
Unfortunately re-doing the configs didn't fix the crashing issue. I've just upgraded to the latest 5.0.3 version, and checked with pkg info. I'll see if this fixes it, thanks!
Also, could we apply the limiter to specific groups?
-
-
Also, could we apply the limiter to specific groups?
For transparent proxy, the rules are manual, so l guess yes.
-
@Marcelloc Just an update on the situation. The latest update fixed some of the crashes I had, everything's been running better today. 2500 workers, and I was able to use over 1000 without any huge problems. Its also been faster! Let's hope it stays that way, and improves until we can eradicate all crashes. There was time when my entire pfsense would crash, so I'm grateful for the improvements made.
I have a few suspicions that the crashes may perhaps be related to using pfBlockerNG along with transparent mode. I am trying to explore all the avenues to try locate the fault but it isn't easy without proper debugging. I have disabled pfBlockerNG for the meantime to try and isolate the problem, let's see if there's any improvement.
That being said, I think it's best to keep some debug code that can be enabled easily when required as we can come across quite a bit of unforseen issues with FreeBSD as the developers of E2G use Linux.
-
So disabling pfBlockerNG did nothing, I really thought that would've been it since the transparent proxy is sending all the pfBlocker domains through the proxy and crashing it or something.
I've checked and redone pretty much all the ACLs, removed all test and unnecessary ACLs and I'm having issues. I've even tried multiple restarts and reinstalls…
I'm pretty surprised and shocked no one else is having this.
-
I am also having issues with Transparent proxy mode. I messed with this for quite awhile and it would at times work with only https connections which seems weird. Then reinstalling and or rebooting some times would change things and it would work correctly for a little bit and then it would quit. Sometimes it would not catch any connections at all and connections would bypass E2 completly. But mostly it would not complete http connections while catching https. I know it would try because nothing would happen for awhile till browser would show error.
Also if I configure proxy settings on my computer things work correctly. It seems there are issues with transparent mode.
-
I am also having issues with Transparent proxy mode. I messed with this for quite awhile and it would at times work with only https connections which seems weird. Then reinstalling and or rebooting some times would change things and it would work correctly for a little bit and then it would quit. Sometimes it would not catch any connections at all and connections would bypass E2 completly. But mostly it would not complete http connections while catching https. I know it would try because nothing would happen for awhile till browser would show error.
Also if I configure proxy settings on my computer things work correctly. It seems there are issues with transparent mode.
I've never had issues with it oniy loading HTTPS sites, however I have come across instances where HTTPS traffic just bypasses E2 Guardian. Usually giving E2 Guardian a restart fixes it for the most part. I'm not sure if this is somehow conflicting with pfBlockerNG. I've tried various things but the fact that both E2G and pfBlockerNG both forward traffic to themselves could somewhat cause an issue.
@Marcelloc do you use pfBlockerNG and have you come across any issues?
-
On my labs I do not use pfblockerNG but on a network with 850 workstations yes.
pfblockerNG basically create firewall rules, so the only way I see that can conflict with e2guardian is preventing it to connect to a remote site.
-
Try to disable watch guard and see on access log which was the last site or url accessed before crash. Repeat few times to see if we can get close to a way to reproduce the crash on any system.
-
to install a debug version of e2guardian, paste this under console
pkg add -f https://e-sac.websiteseguro.com/e2g/e2guardian-5.0.2_5.txz
then run
/usr/local/sbin/e2guardian -QN
to reinstall current Unofficial repo version use
pkg install -f e2guardian
-
Try to disable watch guard and see on access log which was the last site or url accessed before crash. Repeat few times to see if we can get close to a way to reproduce the crash on any system.
Great idea, I'll definitely give that a go now. Thanks for your input!
-
to install a debug version of e2guardian, paste this under console
pkg add -f https://e-sac.websiteseguro.com/e2g/e2guardian-5.0.2_3.txz
then run
/usr/local/sbin/e2guardian -QN
to reinstall current Unofficial repo version use
pkg install -f e2guardian
Can we not integrate this into the unofficial rep? And allow the debug mode to be turned on or off? I think that maybe easier to manage and use.
I've tried a lot of things last night to try combat the crashes, I've even changed my pfBlocker ports again even though I've never set them up to conflict with each other. I did see in logs that E2 Guardian was saying that port 8080 was in use. I'm not sure if it was just ghost instances of itself or another service. Nothing else is set to use my E2G ports.
-
Can we not integrate this into the unofficial rep? And allow the debug mode to be turned on or off? I think that maybe easier to manage and use.
No. It's a compile option.
-
I did see in logs that E2 Guardian was saying that port 8080 was in use. I'm not sure if it was just ghost instances of itself or another service. Nothing else is set to use my E2G ports.
Probably yes. Parallel e2guardian proceses due watch guard script trying to start e2guardian during an apply config or another apply config while process is reloading…
-
So I've just had E2Guardian crash on me again… This time I have snapped all the logs. Here's the last few lines on the access.log
18.04.15 16:19:11 172.16.1.6 172.16.1.6 https://r1---sn-8pgbpohxqp5-ajtl.googlevideo.com:443 *TRUSTED* Site match: googlevideo.com CONNECT 138829 0 - 3 200 - 172.16.1.6 wireless_users - - - - - 18.04.15 16:19:13 172.16.1.3 172.16.1.3 https://use.fontawesome.com - - 5060 0 - 2 0 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:15 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=01c53a8c-ce4e-4a87-948a-bb420093349e - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:18 172.16.1.6 172.16.1.6 https://m.youtube.com/api/stats/watchtime?ns=yt&el=detailpage&cpn=Bnm_BnpPYGGmvaw6&docid=9YZgy1dpesY&ver=2&referrer=https%3A%2F%2Fm.youtube.com%2Fwatch%3Fv%3DQFe1PcLZGQE%26itct%3DCBYQpDAYAiITCPGnsKHIvNoCFRekFQodY%252BkPo1IKbWFkcyBsZXdpcw%253D%253D%26lact%3D12&cmt=155.448&plid=AAVp5J4QQ2VIWDNA&ei=RG3TWvv3LdbzWtWSgZgM&fmt=134&fs=0&rt=49.006&of=sidReqaeP4yRcfYsI6hTkw&euri&lact=39362&cl=192651238&state=playing&vm=CAEQABgE&volume=100&c=MWEB&cver=1.20180412&cplayer=UNIPLAYER&cbrand=generic&cbr=Firefox&cbrver=59.0&cmodel=android%204.4&cos=Android&cosver=4.4.4&cplatform=TABLET&hl=en_GB&cr=GB&len=796.261&rtn=89&feature=related&afmt=251&idpj=-4&ldpj=-25&rti=49&muted=0&st=136.465&et=155.448&conn=3 - GET 0 0 - 3 204 - 172.16.1.6 wireless_users - - - - - 18.04.15 16:19:20 172.16.1.6 172.16.1.6 https://m.youtube.com/api/stats/qoe?event=streamingstats&fmt=134&afmt=251&cpn=Bnm_BnpPYGGmvaw6&ei=RG3TWvv3LdbzWtWSgZgM&el=detailpage&docid=9YZgy1dpesY&ns=yt&fexp=23707874%2C23708904%2C23708906%2C23708910%2C23710476%2C23712544%2C23712994%2C23713711%2C23716256%2C23716972%2C23718632%2C23720634%2C23721898%2C23723618%2C23723926%2C23724493%2C23725949%2C23726426%2C23728352%2C23731308%2C23731896%2C23732330%2C23732385%2C23732729%2C23733751%2C23734535%2C23735135%2C23735347%2C23736055%2C23736249%2C23736874%2C9422596%2C9449243%2C9475691%2C9477414%2C9485000%2C9486081%2C9487194%2C9489759&cl=192651238&seq=4&c=MWEB&cver=1.20180412&cplayer=UNIPLAYER&cbrand=generic&cbr=Firefox&cbrver=59.0&cmodel=android%204.4&cos=Android&cosver=4.4.4&cplatform=TABLET&vps=50.154:PL&bwm=50.154:3569482:1.432&bwe=50.154:1955673&cmt=50.154:156.525&bh=50.154:51.736 - POST 0 0 - 3 204 - 172.16.1.6 wireless_users - - - - - 18.04.15 16:19:21 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=6a8e596b-71f5-4665-91c3-b18339d68777 - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:21 172.16.1.6 172.16.1.6 https://endpoint1.collection.eu.sumologic.com/receiver/v1/http/ZaVnC4dhaV3f1zhS5xhQcAjcICG2O_Vtd8mu-39iqkdF1iAhhFWHpSA5bUBK_SrbgvG8GdcqjH_DGs-NplOisVzYbr-qoEfcff0MhUJH4RV79DiuIFEF0w== - POST 0 0 - 3 200 - 172.16.1.6 wireless_users - - - - - 18.04.15 16:19:22 172.16.1.3 172.16.1.3 https://easylist.to - - 148068 0 - 2 0 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:25 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=01c53a8c-ce4e-4a87-948a-bb420093349e - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:29 172.16.1.7 172.16.1.7 https://play.googleapis.com *TRUSTED* Site match: play.googleapis.com - 1166 0 - 4 0 - 172.16.1.7 wireless_users - - - - - 18.04.15 16:19:29 172.16.1.6 172.16.1.6 https://s3-eu-west-1.amazonaws.com/smhw-frontend-prod/VERSION.json - GET 0 0 - 3 304 - 172.16.1.6 wireless_users - - - - - 18.04.15 16:19:31 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=6a8e596b-71f5-4665-91c3-b18339d68777 - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:34 172.16.1.3 172.16.1.3 http://go.microsoft.com/fwlink/?LinkID=252669&clcid=0x409 - POST 0 0 - 2 302 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:35 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=01c53a8c-ce4e-4a87-948a-bb420093349e - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:36 172.16.1.6 172.16.1.6 https://m.youtube.com/youtubei/v1/log_event?alt=json&key=8Q4STEHLGCilw_Y9_11qcW8 - POST 48 0 - 3 200 - 172.16.1.6 wireless_users - - - - - 18.04.15 16:19:41 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=6a8e596b-71f5-4665-91c3-b18339d68777 - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:45 172.16.1.3 172.16.1.3 http://push.bitdefender.net/poll?push_id=01c53a8c-ce4e-4a87-948a-bb420093349e - GET 16 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:45 172.16.1.3 172.16.1.3 https://nexus.officeapps.live.com:443 - CONNECT 152 0 - 2 200 - 172.16.1.3 Forid - - - - - 18.04.15 16:19:46 172.16.1.3 172.16.1.3 https://beacons5.gvt3.com - - 3362 0 - 2 0 - 172.16.1.3 Forid - - - - -
We've got a tasty collection of enormous URL's, post requests and a tonne of random telemetry. @Marcelloc does this help?
-
Can you run e2g with debug binaries, browse these latest sites to get the error message?
what authentication are you using to get you user on logs?
-
Can you run e2g with debug binaries, browse these latest sites to get the error message?
what authentication are you using to get you user on logs?
I just had another crash, see below:
18.04.15 19:08:18 172.16.1.3 172.16.1.3 https://client.wns.windows.com - - 4430 0 - 2 0 - 172.16.1.3 Forid - - - - - 18.04.15 19:09:05 172.16.1.17 172.16.1.17 http://www.msftconnecttest.com/connecttest.txt - GET 22 -40 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:12 172.16.1.17 172.16.1.17 http://cdn.content.prod.cms.msn.com/singletile/summary/alias/experiencebyname/today?market=en-GB&source=appxmanifest&tenant=amp&vertical=finance - GET 1008 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:12 172.16.1.17 172.16.1.17 http://cdn.content.prod.cms.msn.com/singletile/summary/alias/experiencebyname/today?market=en-GB&source=appxmanifest&tenant=amp&vertical=sports - GET 1137 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:13 172.16.1.17 172.16.1.17 http://cdn.content.prod.cms.msn.com/singletile/summary/alias/experiencebyname/today?market=en-GB&source=appxmanifest&tenant=amp&vertical=news - GET 1248 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:16 172.16.1.17 172.16.1.17 http://cdn.content.prod.cms.msn.com/singletile/summary/alias/experiencebyname/today?market=en-GB&source=appxmanifest&tenant=amp&vertical=sports - GET 1137 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:16 172.16.1.17 172.16.1.17 http://cdn.content.prod.cms.msn.com/singletile/summary/alias/experiencebyname/today?market=en-GB&source=appxmanifest&tenant=amp&vertical=finance - GET 1008 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:16 172.16.1.17 172.16.1.17 http://cdn.content.prod.cms.msn.com/singletile/summary/alias/experiencebyname/today?market=en-GB&source=appxmanifest&tenant=amp&vertical=news - GET 1248 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:21 172.16.1.17 172.16.1.17 https://client-office365-tas.msedge.net - - 12400 0 - 2 0 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:21 172.16.1.17 172.16.1.17 https://config.edge.skype.com *TRUSTED* Site match: skype.com - 6894 0 - 2 0 - 172.16.1.17 Forid - - - - - 18.04.15 19:09:41 - 172.16.1.10 https://mmg-fna.whatsapp.net:443 *TRUSTED* Site match: whatsapp.net CONNECT 358 0 - 1 200 - 172.16.1.10 Default - - - - - 18.04.15 19:09:58 172.16.1.17 172.16.1.17 https://client.wns.windows.com - - 4430 0 - 2 0 - 172.16.1.17 Forid - - - - - 18.04.15 19:10:05 - 172.16.1.14 http://clients3.google.com/generate_204 *TRUSTED* Site match: clients3.google.com GET 0 0 - 1 204 - 172.16.1.14 Default - - - - - 18.04.15 19:10:05 - 172.16.1.14 http://clients3.google.com/generate_204 *TRUSTED* Site match: clients3.google.com GET 0 0 - 1 204 - 172.16.1.14 Default - - - - - 18.04.15 19:10:07 172.16.1.17 172.16.1.17 https://nexus.officeapps.live.com:443 - CONNECT 2904 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:10:07 172.16.1.17 172.16.1.17 https://nexusrules.officeapps.live.com:443 - CONNECT 2904 0 - 2 200 - 172.16.1.17 Forid - - - - - 18.04.15 19:10:07 - 172.16.1.14 https://api.samsungcloud.com:443 - CONNECT 6400 0 - 1 200 - 172.16.1.14 Default - - - - - 18.04.15 19:10:08 - 172.16.1.14 https://api.samsungcloud.com:443 - CONNECT 383 0 - 1 200 - 172.16.1.14 Default - - - - - 18.04.15 19:10:12 172.16.1.17 172.16.1.17 https://v20.vortex-win.data.microsoft.com - - 4215 0 - 2 0 - 172.16.1.17 Forid - - - - -
I am using IP authentication, with groups. I've got static IP for anyone who needs any "special" ACL's added. I will install the debug binaries now.
-
@Maracello, I have installed the debug binaries and it's made the mystery deeper! I have tried all the last sites in the access.log and had no crashes… I would set the debug logs to be written to a file but I can tell that they would become huge in a matter of minutes.
Have you got any other suggestions?
-
Leave it open without redirect on a ssh console.
When it crashes, latest info on ssh console will be the relevant information to send to e2guardian dev team.
-
Leave it open without redirect on a ssh console.
When it crashes, latest info on ssh console will be the relevant information to send to e2guardian dev team.
Got it, thanks bro! I'll keep a SSH session open with this on my ESXi VM :)
EDIT:
GOTCHA!! Found the problem, it's the HOSTS in the block page. Removing that broken feature right away.
hw2390: Start of request header:in hw2390: header:in before getLine - timeout:55000 Line: 1938 Function: in hw2390: firstime: header:in after getLine Line: 1942 Function: in hw2390: header:size = 7 hw2390: first line = POST /vpninfo/servers HTTP/1.1 Line: 981 Function: checkheader: Host: www.privateinternetaccess.com Line: 981 Function: checkheader: Accept-Encoding: identity,gzip,deflate Line: 981 Function: checkheader: Accept: */* Line: 981 Function: checkheader: User-Agent: Ruby hw2390: tp =Content-Length: 49 hw2390: tp =49 Contentlen.int =49 Line: 981 Function: checkheader: Content-Length: 49 hw2390: Header value from client: Content-Type: application/x-www-form-urlencode Line: 981 Function: checkheader hw2390: CheckHeader: HTTP/1.1 detected Line: 990 Function: checkheader hw2390: CheckHeader flags before normalisation: AP=1 PPC=0 1.1=1 connectionclose=1 CL=1 Line: 1051 Function: checkheader hw2390: CheckHeader flags after normalisation: AP=0 WP=0 Line: 1074 Function: checkheader hw2390: CheckHeader: Adding our own Proxy-Connection: Close Line: 1093 Function: checkheader hw2390: HTTPHeader size: 8 hw2390: from header url:http://www.privateinternetaccess.com/vpninfo/servers Line: 1235 Function: getUrl hw2390: setURL: header.front() changed from: POST http://www.privateinternetacce Line: 540 Function: setURL/1.1 Line: 548 Function: setURLw.privateinternetaccess.com/vpninfo/servers HTTP/1.1 Line: 553 Function: setURLne changed from: Host: www.privateinternetaccess.com Line: 562 Function: setURLteinternetaccess.com hw2390: isProxyRequest is 0 hw2390: firsttime =1ismitm =0 clientuser = group = 1 hw2390: Compiling ,*[a-z|A-Z].* hw2390: ...with PCRE hw2390: HTTPHeader size: 8 hw2390: from header url:http://www.privateinternetaccess.com/vpninfo/servers Line: 1235 Function: getUrl hw2390: decoding url Line: 1519 Function: decode hw2390: 38420Start URL http://www.privateinternetaccess.com/vpninfo/serversis_ssl=0ismitm=0 hw2390: inIPList no match for 172.16.1.17 hw2390: inIPList no match for 172.16.1.17 hw2390: inURLList: www.privateinternetaccess.com/vpninfo/servers hw2390: inURLList (processed): www.privateinternetaccess.com/vpninfo/servers After StoryA pre-authcheck0 mess_no 0 hw2390: isProxyRequest is 0 only_ip_auth is 1 hw2390: -Not got persistent credentials for this connection - querying auth plugins hw2390: -Querying next auth plugin... hw2390: -Auth plugin found username 172.16.1.17 (), now determining group hw2390: Matched IP 172.16.1.17 to straight IP list hw2390: Auth plugin found username & group; not querying remaining plugins hw2390: -Identity found; caching username & group Auth plugin is for a connection-based auth method - keeping credentials for entire connection hw2390: -username: 172.16.1.17 hw2390: -filtergroup: 1 hw2390: -About to check for bypass... hw2390: Found cookie: Line: 1309 Function: getCookie hw2390: No bypass cookie Line: 1337 Function: isBypassCookie hw2390: -Finished bypass checks. hw2390: inURLList: www.privateinternetaccess.com/vpninfo/servers hw2390: inURLList (processed): www.privateinternetaccess.com/vpninfo/servers hw2390: inURLList: www.privateinternetaccess.com/vpninfo/servers hw2390: inURLList (processed): www.privateinternetaccess.com/vpninfo/servers hw2390: After StoryB checkrequest isexception 0 gomitm 0 mess_no 500 hw2390: -Forwarding body to client : Upfailure is 0 hw2390: -reporting level is 3 hw2390: -Enabling filter bypass hash generation hw2390: mime type: - Line: 217 Function: isContentType hw2390: mimes result : false Line: 244 Function: isContentType hw2390: Displaying TEMPLATE hw2390: -HOST- placeholder encountered but hostname currently unknown; lookup forced. Segmentation fault
-
Happened and fixed on 4.1. maybe not merged on 5.0
I'll se if I find the issue number on 4.1 branch and post on #375