Necessary traffic being blocked, how to identify and pass

zaileion

I am running a Storj node. Its decentrilized storage www.storj.io if you are interested.

It work like this:
I have 10TB of storage that i allocated to s Storj Node which makes my drive sapce available to those using the service.

There are satellites that kind of act like a directory for companies or people to go through to access the storj nodes made available.

I have port forwarded the correct port through my firewall but some of the traffic is still blocked. The storj support agents said that my pfsense router thinks that some of the traffic from these satellites and companies accessing my node as a DDOS attack and blocking the traffic.

Im new to pfsense, got fed up with these all in one routers and switched.

I need to be able to identify this blocked traffic and allow it tp pass but have no idea how and I'm at risk of being disqualified from Storj if i dont get it fixed. It took me a long time to get the hardware and setting up the node so I surely dont want that to happen.

Can anyone help provide some guidance on identifying this traffic and allowing it to pass? I would be most grateful!

*Here is the email communication from the storj node support agent:

Storj Support:*
Your firewall is blocking connections from US-Central-1, Europe-West-1 and us2. Please, check your DDOS settings and configure it to do not block audits requests from the satellites.
Now your node have only 2 audit requests registered and only one is responded.
If your firewall would keep blocking audit requests, your node can be disqualified.

Best regards,
Storj Labs Support

Me:
I have it ported through my pfsense firewall > port: 28967
Is there something specifically I need to do in the firewall for the US.Central Satellite? Please let me know how to do this!

Storj Support:
Pfsense have an ability to prevent DDOS attacks, it doing that by analyzing frequency of requests from the same or several IPs. I do not know details, I just know it's possible to configure it like this.
It blocks requests from IPs of tardigrade satellites, you can see them there: https://tardigrade.io/trusted-satellites but not totally, but it's enough to consider your node offline by these satellites. The firewall should not block or throttle anything to the port of your storagenode.
It blocks traffic with some throttling and this happening not for all satellites, thus your node shows Online, because it has confirmation from few satellites. But in the detailed view you can see how each satellite is seeing your node and you see a yellow and red warning marks next to the online score of blocked satellites.

Best regards,
Storj Labs Support

Please keep in mind, the node is a p2p service, it will have a lot of connections from all the world, the data transfers are happening between a customer and the node directly. The satellites are only address book, audit service, repair service and payment processor, they do not proxying customer's data.
So, please, configure your firewall to do not block any connections from any addresses to the port of your storagenode.

I'm not a specialist of pfsense, we can give a support only for our products but not how to configure your OS or services (except basic level).

But you can ask for help with pfsense on our forum: https://forum.storj.io, our friendly Community can help with almost anything.

Best regards,
Storj Labs Support

johnpoz

@zaileion said in Necessary traffic being blocked, how to identify and pass:

Storj Support:
Pfsense have an ability to prevent DDOS attacks, it doing that by analyzing frequency of requests from the same or several IPs

No it doesn't... Not sure where he would of gotten that idea.. Are you even running IPS?

Lets see these blocks.. That your seeing in the logs..

zaileion

Well it gave me an error when i tried to paste the logs here so i had to put them in pastebin:

https://pastebin.com/jh0Tb4Lq

I hope that helps, its a lot of information and I'm really new to this advanced firewall thing...

My storj node is Ubuntu 192.168.1.246 on an ESXi server with mgmt IP 192.168.1.251

and im not sure that any of these blocks are actually blocks of the storj node. I just have no idea what to do, and im trying to keep my storj node running...

johnpoz

You have some issue talking to your gateway, that is not blocking inbound traffic - but that would for sure cause you issues with inbound traffic - if your internet connection is down.

No I mean the firewall logs where its blocking this traffic they say its blocking. Post us a picture of it.

I don't see any blocks in what you posted to this port 28967

Post your wan rules, and your port forward you setup.

zaileion

@johnpoz said in Necessary traffic being blocked, how to identify and pass:

You have some issue talking to your gateway, that is not blocking inbound traffic - but that would for sure cause you issues with inbound traffic - if your internet connection is down.

What do you mean by this:
"You have some issue talking to your gateway, that is not blocking inbound traffic - but that would for sure cause you issues with inbound traffic - if your internet connection is down."

My internet is up and running, as I' typing to you right now. How do I identify this issue talking to my gateway, like you said, and how do i fix it?!

I really hate to sound so incompetent... I'm learning as I go here...

It could be this issue you mention with talking to my internet. My storj node is online and running, but this internet issue you mention might be causing short temporary outages or otherwise causing dropped packets making the storj node inaccessible for a few seconds or minutes here and there

johnpoz

Feb 22 11:51:21	kernel		arpresolve: can't allocate llinfo for 73.133.106.1 on em0
Feb 22 11:51:21	kernel		arpresolve: can't allocate llinfo for 73.133.106.1 on em0
Feb 22 11:51:21	kernel		arpresolve: can't allocate llinfo for 73.133.106.1 on em0
Feb 22 11:51:21	kernel		arpresolve: can't allocate llinfo for 73.133.106.1 on em0
Feb 22 11:51:21	kernel		arpresolve: can't allocate llinfo for 73.133.106.1 on em0

Then this..

Feb 10 19:22:47	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 73.133.106.1 bind_addr 73.133.106.251 identifier "WAN_DHCP "

zaileion

yup. I'm seeing it and I thought it could be a problem. but... I have absolutely 0 idea what any of that means. I know em0 is the esxi NIC assigned to PFsense WAN and I know the .251 IP is the management IP for the ESXi server.

zaileion

Anyone have any ideas what it means?

SteveITS

Feb 10 19:24:23 dpinger WAN_DHCP 73.133.106.1: sendto error: 65

per https://docs.netgate.com/pfsense/en/latest/troubleshooting/gateway-errors.html

"sendto error: 65

65 EHOSTUNREACH
No route to host.
A socket operation was attempted to an unreachable host.

Either there is no possible route to the target locally, or status information was received from an upstream router that indicated the same condition elsewhere along the path to the target.

This can happen due to a lack of default route, missing interface link route, or similar conditions."

Feb 12 03:43:14 dpinger WAN_DHCP6 fe80::201:5cff:fe6b:c246%em0: Alarm latency 11087us stddev 3007us loss 21%
Feb 12 03:43:15 dpinger WAN_DHCP 73.133.106.1: Alarm latency 11191us stddev 3087us loss 22%
Feb 12 03:45:53 dpinger WAN_DHCP 73.133.106.1: Clear latency 8856us stddev 2151us loss 11%

11-22% packet loss. IOW it looks like you're logging a connection/packet loss problem.

You can have pfSense email you alerts: https://docs.netgate.com/pfsense/en/latest/config/advanced-notifications.html

Gertjan

@zaileion

Your hosting a service.

I have 10TB of storage ...... available to those using the service.

If possible : your WAN connection : make it twice as fast as the fastest "client" that is using your service. So, If someone uses a true fibre connection, and comes in your service with a ful one Giga bit, make your WAN at least more then one giga, like 2.

This probably means you'll have to $$$$ a lot.
( but hey, a free service is never free ... for some one )

Because :
The same client comes in, storing something.
Your WAN goes 100 %.
Less priority stuff like ... ICMP, gets dropped.

packet loss problem .....

which makes dpinger think the connection is bad : gateway (or the to be pinged host) becomes (less) reachable. dpinger can even pull the plug for a while to re establish the connection.

Or worse, the NIC goes KO for a while. remember : it's a virtual NIC, they do not have the same speed as what is advertised on the - real - chip set.

Again :

I have 10TB of storage ...... available to those using the service.

Alternative : What about some serious traffic shaping ? This will make your storage less attractive as it becomes 'slower' for all it's users, but at least your WAN will be able to follow.

zaileion

ok. I have a 1GB WAN from Comcast. it runs at about 750MB, its never a Gig. anyway, I am in the process of aggregating 2 nic's into my switch from my docs 3.1 modem and 2 WAN ports into pfsense (if thats possible), and added a 2nd virtual and physical nic to the storj node. Now I'm trying to do some traffic shaping to prioritize traffic from the node in both the switch, esxi and pfsense. This is new to me as i said so its going to take me a minute to figure out how to do it. Also, it seems the storj node is working much better already and the satellite online % has increased significantly overnight.

I am having loss and lag on the WAN port still between 8% and 22% I have an appointment scheduled for a tech to come out but of course they will say. "everything looks fine..." because unfortunately Comcast field techs get paid poorly and thus are minimally knowledgeable which is a corporate decision and is a bad one to say the least, for both the field techs and the customer but good for the share holders and board members. Right? Anyway, thank you to everyone for the help identifying my issues. So far pfsense and the community has been great!

EDIT: I just wanted to say, that so far everything open source and Linux related is just awesome. I have been on several forums, this one is the latest and its just great how everyone helps out and i dunno. its just a great way to do things... Thanks everyone.