Disaster accessibility

dfarquharson

I actively used pfSense during the early 2000's, up until January 12, 2010. That was the date of the catastrophic earthquake in Haiti. At that point I ended up in charge of the 'hardware' setup for a medical relief centre at Quisqueya Christian School in Port-au-Prince.

I chose the rooms to use for the Relief HQ, servers, battery banks and inverter systems. I installed all of the electrical wiring, network cabling, servers, routers and the complete VSAT internet setup (the bandwith was provided by some person or organization in England, but I never knew who it was).

One of my immediate concerns was network secuity, for which I planned on using pfSense because I was familiar with it. I immediately needed 4 different access groups.

The school administration
The relief center administration
The doctors and nurses being housed and dispatched from the school
Various visitors (who eventually included a huge range of people)
All of these required access to the Internet but widely varying access to various parts of our local networks.
Compounding this were my fears about outside hackers. Haiti has an EXTREMELY high unemployment and under employment rate, and many of these young people had enthusiastically embraced hacking.

Unfortunately, I was not able to get any reply from this community. After a couple of days, during which I was frantically installing equipment, I had to start searching other avenues. I'm an Amateur Radio operator so I sent messages out through it. One of my contacts found that he had a BSD expert as a neighbour so put me into contact with him.

We initially contacted through email, where immediately sent messages to each other listing the chat programs we each had installed and used. It was kind of hilarious (if I hadn't been in such desperate, frantic need. We each had over a dozen 'messenger' programs installed and the only one both of us had was Yahoo Messenger. Neither of us like it but that was what we had as there was no time to waste while learning another program.

He had never heard of pfSense but had years of using the BSD pf system. So he set himself to downloading pfSense and seeing how to set it up to do what he already knew how to do directly in BSD. I went to sleep for a couple of hours (it was after midnight and I had not had any sleep for the previous 3 days.

At 7 AM we were back communicating through Yahoo Messenger.I had already set up a computer with pfSense. I set up for him get through our firewall and connect directly to my computer and we worked our way through exactly what I needed, he set up all of this onto a BSD computer and then would set up pfSense and look to see how this compared to his BSD code. When he had what looked like it would do what I needed he passed the information to me to set up my pfSense machine. By that afternoon I had a functioning firewall.

My point in all of this rambling is that I think your group should see if it is possible to set up some sort of easy way for anyone who ends up on the ground when the unthinkable happens, to get hold of you. My experience taught me that when disaster hits, almost ALL the fancy equipment you might wish for is not only not available, there isn't even any way of getting it in any sort of reasonable time. We were lucky in having a whole bunch of Linksys WRT wifi routers. It took over a month to get some low end Cisco routes in. Meanwhile we fried another Linksys every few days.

pfSense is a perfect solution for this sort of thing - as long as there is a team of people available to assist the poor individual who is on the ground and going crazy with an unending stream of emergencies and a bunch of bosses who have absolutely NO idea of how to do whatever it is that they are wanting you to do and even LESS idea of the things that can (and WILL if you don't prepare for them) go wrong.

I'm retired now and back in Canada. I still get the Digest of freebsd-pf@FreeBSD.org messages and when I noticed the one today I thought of this rant that has been percolating in my head for a dozen years. Now I can safely forget the rant and I'm curious to see if I get any replies.

David Farquharson

SteveITS

Sooty to hear of your difficulties. All I can say is, the past several years I’ve been active on this forum, there are several very active people who would have helped as they do every day.

stephenw10

Mmm, that sounds like a nightmare!

I'm not sure exactly what you're suggesting. Some way to connect to the community that doesn't rely on a functioning internet connection?

Things are different now than they were in 2010. There are multiple ways to reach out of you do have a connection of some sort. But we are always looking for ways to improve that if you have a suggestion.

Steve

dfarquharson

I missed out on saying that I attempted multiple messages to this forum but never got a reply. After posting that message yesterday I checked my history and the posts I made then are still there and have never been replied to. So it very encouraging that this time I have received two replies in half a day.

What I'm trying to ask (I think) is if there is any way that someone who is in a position similar to what I was in back then, to actually get an answer. It might be that this forum is more active now than it was back then and my posts would not be ignored now.

I also think it would be worthwhile to have a place (maybe as a topic on this forum) where information is available on setting up pfSense in an emergency situation, bearing in mind the the person on the ground is unlikely to have any direct experience with using pfSense.

One of the things I did was do a few quick searches but found nothing helpful.

As for things being different now - not really. I had been installing direct satellite Internet connections for almost 10 years at that point. And my experience is Haiti shows that there are probably a great many countries around the world who rely much more on wireless communications than the USA does, because they do not have much in the way of installed hard infrastructure.

A couple of examples from Haiti.

When I arrived in Haiti as a teacher at Quisqueya Christian School I found that many of the students carried portable trunking radios with them. This was in 1986, before there was any Internet. The installed telephone system in the country was totally inadequate - there were only ever about 60,000 lines installed and many of those lines had been destroyed. So telephones were reasonably rare and were often unreliable. So kids carried radios so they could be in contact with (and be contacted by) their parents.
A great many businesses used wireless systems to communicate between their various establishments. A favoured system was to use C-band. Normally C-band was used for satellite communications and satellite television, so C-band beams were vertical (ground to geostationary satellite). But the US Marines discovered their backbone communications systems didn't work when they invaded in 1994. There were already many hundreds (maybe thousands) of horizontal C-band beams crisscrossing the country using all available frequencies.
In 1998 I was flying across the USA to spend Christmas with friends near Seattle when I heard an announcement about the first installation of a demo WiMAX system in Baltimore. I laughed. I had already installed country wide WiMAX systems in Grand Cayman and Barbados and 3 demo systems in Haiti.
For those who don't know what WiMAX is - it is like super WiFi. WiFi was originally designed for about 200 foot distances. It has been significantly enhanced by now but is still short distance. I installed Alvarion WiMax system and they had an internal setting that needed to be turned on if you were going to exceed 105 kilometers.

On a different note:
One of the things I mentioned in my post yesterday was being worried about Haitian hackers trying to get into our system. As it turned out, we had absolutely no problems along those lines. Two things led to that: 1: the people of Haiti very quickly spread the word around that we were sending out over 200 medical people every day to dozens of field hospitals and thousands of tons of medical supplies. So they did their best to help and at least no interfere. 2: the US Southern Command took over the bottom half of our campus and were in charge of the whole country. NO ONE parked on the streets immediately around the school as there were always soldiers out there guarding the place, and they regularly patrolled the streets close to the campus, so no 'bad guys' (if there were any) bothered trying either.

But it is unlikely that any future distasters will be so fortunate.

stephenw10

Hmm, where are you looking? I see this thread:
https://forum.netgate.com/topic/20954/haiti-emergency
But you had a reply there in 11 minutes so I assume it was not that.

The forum was quire active back then but there are more Netgate staff present now who I would hope would jump into a thread like that.

Our documentation is certainly better today than it was then.
What sort of thing do you think would help in an emergency beyond the existing install docs?
https://docs.netgate.com/pfsense/en/latest/install/index.html

Steve

dfarquharson

Hello Steve
I just re-read all those messages and reply's from that long ago disaster. Looking at the dates and times of the postings I can see it was 12 days into the disaster and I was getting close 'burn out'. I actually hit the wall on day 16 (as I ran solidly into the concrete walls several times while trying to get back into the crisis centre. I calculated later that I'd manager 39 hours of sleep in the first 16 days.

So I suspect I wasn't exactly reading (and understanding) things correctly.

What I think would be handy for someone in that position would be to have a 'ready-to-install' download available that had been configured with a series LANs (probably 4 to 6 of them) with different access and rights for each of the LANs. This installation would probably not use the latest version but would use the most recent 'ultra-stable' version. It would include instructions for the basic install and a written 'walk-through' of the setup explaining what the settings of each LAN were, how they affected access rights, etc and how to change them.

Something like this would allow the person on the ground to get everything downloaded and installed within a very short time and most likely at least one of those pre-set LANs would be a close approximation for at least part of what was needed. That person could then get that part running and would now have a bit of time to work with someone at netgate.com to discuss what they wanted and work out how to get the settings correct. I found that there was ALWAYS people willing to help out. The problem came in tracking down the people that had the knowledge that was needed right now.

With such a package available there could be a usable system functioning very quickly and a duplicate system could be set up at the netgate.com end so when help was needed both ends could be working from the same page. The problem in a disaster area is that there are an almost unlimited number of things that need to be done, they need to be done NOW and there is probably only one person available who has the knowledge to even begin doing it. The Internet is wonderful in being able to connect you to other people who want to help, but taking the time to find them, then taking time to discuss what is needed, then having to learn (mostly through trial and error) how to modify things to fit the specific needs is literally killing other people.

Of course, if you were to develop such a package, you would need to find a way to 'advertise' that it was available, otherwise no one would know what to look for or how to find it. Those sorts of things are completely beyond my skill set. You also have to realize that the very best thing that can possibly happen if you spend time developing such a 'disaster' package is that it is never needed.

I was the defacto vice president of the Radio Club d'Haiti at the time. At about the same time I was contacting this forum I got my home amateur radio set up again. The first thing I did was try to contact my friend, the president of the club - NOTHING, on any band. He lived about 3 kilometers away and my heart sank. Then I made contact with an amateur operator in New Hampshire. The first thing I asked was if he had heard anything from Jean Robert. His answer was, "Yes, I'm talking to him right now." So I had a short conference with my friend, relayed by the operator in New Hampshire and we each found out that the other was fine and we were both up past our ears in work. It only took about a 4,000 mile trip to get a message over the 2 mile gap that actually separated us.

Reviewing all this stuff is making me cry so I better get off for a while.

David

stephenw10

Urgh, that sounds brutal!

It would be tough to have a pre-configured setup if only because in that situation you have no idea what hardware is going to be available. It's conceivable that it could just expand on what we do already where if at least two recognised NICs are present they will configured as WAN and LAN. The installer could have an option to pre-assign all available NICs so they are available at boot. However that can actually make it more difficult in some situations where it's not obvious which NIC is which.

Assuming that could be negated though what sort of default config would you want to see there?
I assume you would not want all the internal interfaces configured the same as LAN where hosts on them could reach any other local IP by default?
If all NICs are assigned as internal interfaces they would need to be reconfigured if you have more than one WAN as you did.

Steve

dfarquharson

Hello Steve

As soon as I read your first 10 words I realized that it wouldn't work. So I've been sitting here trying to sort through my memories to see if I can figure out how to set something up that could actually be helpful. I think I'm probably going to have to start back when I had first started using pfSense and work my way up through the earthquake and steps we went through to get the medical crisis centre set up and running. That way I can get some idea in my mind for what my experience with pfSense had included and why I was not able to get it configured as I needed. Maybe then I might be able to see how something could be set up that would be useful to someone who had no previous pfSense experience.

I know this will take some time because just in sitting here for a little while and trying to think this through I find gaps and redundancies in my memories. I'll get back to you when I've got my basics straightened out.

David