My pfSense keeps breaking (novel inside…)

soteriologist

I've already uninstalled all the packages.

The current state I'm in is:
No packages installed.
Reset to factory defaults.
Only the most basic settings have been applied in order to get an internet connection up on it.

And yet I still see no traffic. :-\

It's as I were creating a super-massive star when all of a sudden it imploded into a supernova and warped into a blackhole. :'(
I want my super-massive pfSense star back.

soteriologist

The only thing left for me to do is re-install and start from scratch… AGAIN.

But before I do that, I figured I'd post on here to see if someone had a suggestion to diagnose this shit and hopefully stop others from running into the same problem AND hopefully prevent me from running through it all over again a third time.

I figured if there truly is a horribad bug somewhere in the code, someone would want to know about it and get it fixed.

soteriologist

ok, so I turned on accessing the web configurator from the wan side. One of my internet connections is DSL which uses a wired/wireless router/modem combo. So I've plugged my laptop into one of the other wired ports on the little DSL router and can access my pfSense box through the WAN port there. So at least SOME traffic is flowing through that connection. But it's not showing up in the states/firewall logs.

Still no pings, webpages, email, etc are going through it though. :'(
Can't get a connection to the internet through the pfSense. :-\

wallabybob

@soteriologist:

The only thing left for me to do is re-install and start from scratch… AGAIN.

There are still a number of alternatives, including plugging your laptop into the DSL router and attempting to access the pfSense WAN port.

@soteriologist:

One of my internet connections is DSL
What are the others?

@soteriologist:

So I've plugged my laptop into one of the other wired ports on the little DSL router and can access my pfSense box through the WAN port there.

Can you also access the management interface on the DSL router? What does it tell you about the WAN interface of the DSL router?

What is the interface type of your pfSense WAN interface? (Static? DHCP? PPP?)

Please post the output of of the pfSense shell command```

netstat -rn -f inet; traceroute -n 8.8.8.8

soteriologist

Tried pinging with pfSense's web configurator (under Diagnostics >> Ping) to both 4.2.2.2 and google.com (along with a handful of other sites) and get no response. Tried pinging the DSL router, and get a response. Tried pinging my laptop that is also plugged into the same router and get a response form my laptop's ip address.

Still nothing shows in states/firewall logs though?

I'll try that traceroute command.

soteriologist

Here are the results:

netstat -rn -f inet ; traceroute -n 8.8.8.8
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
127.0.0.1 link#12 UH 0 3412 lo0
192.168.2.0/24 link#7 U 0 10355 em2
192.168.2.2 link#7 UHS 0 0 lo0
192.168.168.0/24 link#5 U 0 341 em0
192.168.168.1 link#5 UHS 0 0 lo0
traceroute: findsaddr: failed to connect to peer for src addr selection.

wallabybob

You don't have a default route hence most of the traffic that would normally go out the WAN interface doesn't go out the WAN interface because there isn't a route saying that is where it should go.

Your pfSense WAN interface type is? (Depending on that I might be able to give you a pfSense shell command to add a default route.) But that won't help if the upstream link from your DSL router is broken. Can you get status of the upstream (to the Internet) link on the DSL router?

What version of pfSense are you running? Please post the version information from the home page of your pfSense box.

stephenw10

Hmm, you have no default route and no route to anywhere outside your network. Problem!
Is your WAN connection up? (or was it when you did this).
@soteriologist:

I've already uninstalled all the packages.

The reason I suspected packages is that they sometimes either overwrite things they shouldn't or remove things they shouldn't when you uninstall them.

Uninstalling all the packages is not necessarily the same thing as never having installed them! ::)

Something has messed up your routing table, either directly or by messing up something that controls the routing table.

Steve

Edit: Typed too slow.

soteriologist

I'm able to get an internet connection fine through all of my WAN devices I've had in the past (and currently have) attached to pfSense. Even when I plug in using the same cables/ports that pfSense would use to those devices.

I ruled out any hardware problems at the get-go.

As for my current version:
Version 2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:16:13 EST 2011
FreeBSD 8.1-RELEASE-p6

As for the default route, I can check mark that box for the interface. Right now it's unchechked because I had a loadbalancing group created and had "Allow default gateway switching" under "System >> Advanced >> Miscellaneous >> Load Balancing" checked.

I can recheck to have just that default DSL line checked as the "Default Gateway" and uncheck the other setting… brb.

cmb

You need to set a default gateway even if you're policy routing your egress traffic. And uncheck the default gateway switching.

What type of WANs?

soteriologist

I just rechecked to have just that default DSL line checked as the "Default Gateway" and unchecked "Allow default gateway switching" under "System >> Advanced >> Miscellaneous >> Load Balancing".
Still no ping response beyond the router/modem it's plugged into.
Stil no internet connection.
Still nothing in state/firewall logs.

Re-ran traceroute and this is what I have now:

netstat -rn -f inet; traceroute -n 8.8.8.8

Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.2.2 US 0 51 em2
127.0.0.1 link#12 UH 0 3524 lo0
192.168.2.0/24 link#7 U 0 19094 em2
192.168.2.2 link#7 UHS 0 0 lo0
192.168.168.0/24 link#5 U 0 341 em0
192.168.168.1 link#5 UHS 0 0 lo0
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets

soteriologist

The three WANs that I have are:
One DSL connection through a Verizon router/modem
One T1 through an AdTran DSU/CSU
One T1 through a Cisco DSU/CSU

AT THE MOMENT I'm ONLY using the DSL for testing. Just to simplify things and because the entire company is actively using the two T1s at office. But when I had everything plugged in during off hours, they were all working fine until… well... everything stopped working. So I had to put everything back they way I had it in the very late hours of the night before everyone came back in the next day and BACK TO THE DRAWING BOARD!

wallabybob

@soteriologist:

Re-ran traceroute and this is what I have now:

Your traceroute output is incomplete.

soteriologist

@wallabybob:

@soteriologist:

Re-ran traceroute and this is what I have now:

Your traceroute output is incomplete.

Ya… just realized that I hadn't copied everything, SORRY!

Here we go:

netstat -rn -f inet; t raceroute -n 8.8.8.8
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.2.2 US 0 201 em2
127.0.0.1 link#12 UH 0 3568 lo0
192.168.2.0/24 link#7 U 0 22223 em2
192.168.2.2 link#7 UHS 0 0 lo0
192.168.168.0/24 link#5 U 0 341 em0
192.168.168.1 link#5 UHS 0 0 lo0
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 * * *
2 * *traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*
traceroute: sendto: Host is down
3 traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*
traceroute: sendto: Host is down
4 traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*
traceroute: sendto: Host is down
5 traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*
traceroute: sendto: Host is down
6 traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*
traceroute: sendto: Host is down
7 traceroute: wrote 8.8.8.8 52 chars, ret=-1
^C

wallabybob

@soteriologist:

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 * * *
2 * *traceroute: sendto: Host is down
traceroute: wrote 8.8.8.8 52 chars, ret=-1
*
traceroute: sendto: Host is down

1 looks like your DSL router doesn't reply to the traceroute probes - that's allowed
2 "Host is down" suggests the DSL router's WAN link is down or for some other reason (also lost its default route?) it doesn't know where to forward the traceroute probes.

soteriologist

I'm posting this connected through the DSL router. So I know for a fact the router is working great.

Here's traceroute from my laptop attached to one of the other ports on the DSL router:

C:\Users\Administrator>tracert 8.8.8.8

Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.2.1
2 45 ms 45 ms 43 ms 10.39.5.1
3 50 ms 50 ms 49 ms at-3-2-0-1715.LAX01-CORE-RTR2.verizon-gni.net [1
30.81.194.2]
4 50 ms 51 ms 125 ms so-1-1-1-0.LAX01-BB-RTR2.verizon-gni.net [130.81
.16.130]
5 54 ms 51 ms 51 ms 0.so-6-0-0.XT2.LAX9.ALTER.NET [152.63.10.157]
6 122 ms 121 ms 121 ms 0.so-1-0-0.XT2.NYC4.ALTER.NET [152.63.64.126]
7 128 ms 133 ms 137 ms TenGigE0-7-0-0.GW8.NYC4.ALTER.NET [152.63.22.45]

8 128 ms 127 ms 129 ms Internet-gw.customer.alter.net [152.179.72.66]
9 128 ms 129 ms 129 ms 72.14.238.232
10 122 ms 119 ms 133 ms 209.85.252.2
11 131 ms 132 ms 130 ms 72.14.239.93
12 126 ms 127 ms 129 ms 72.14.236.200
13 142 ms 143 ms 129 ms 216.239.49.145
14 129 ms 129 ms 128 ms google-public-dns-a.google.com [8.8.8.8]

Trace complete.

chpalmer

Can you show a picture of your WAN interface configuration page? The one connected to your DSL modem??

soteriologist

Here are printscreens of both my Interface and Gateway setup for the WAN.

GatewaySetup.png_thumb

WAN.png_thumb

cmb

Your gateway cannot be the same as the IP on its own interface.

soteriologist

SORRY ABOUT THAT! Slight oversight the last time I reset to defaults. >_<

I've set it to the proper 192.168.2.1 now. (which it always has been except for the very last time I was setting it back up in a hurry and didn't pay attention). =/

Ok, so with that properly in place this is the situation:

Still can't get a connection THROUGH pfSense. (even though it seems to be FULLY communicating over both the WAN port and the LAN port).

When I run a traceroute from PFSENSE this is what I get:

netstat -rn -f inet; traceroute -n 8.8.8.8
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.2.1 UGS 0 19379 em2
4.2.2.2 192.168.2.2 UHS 0 14429 em2
10.39.5.1 192.168.2.1 UGHS 0 172195 em2
127.0.0.1 link#12 UH 0 10979 lo0
192.168.2.0/24 link#7 U 0 1989 em2
192.168.2.2 link#7 UHS 0 0 lo0
192.168.168.0/24 link#5 U 0 18742 em0
192.168.168.1 link#5 UHS 0 0 lo0
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 192.168.2.1 1.178 ms 0.298 ms 0.274 ms
2 10.39.5.1 41.901 ms 40.120 ms 39.915 ms
3 130.81.194.2 48.084 ms 48.009 ms 47.786 ms
4 130.81.16.130 52.014 ms 47.993 ms 48.036 ms
5 152.63.112.49 48.072 ms 48.021 ms 48.001 ms
6 152.63.64.126 113.913 ms 113.986 ms 114.065 ms
7 152.63.21.125 125.921 ms
152.63.16.125 134.030 ms
152.63.21.65 116.058 ms
8 152.179.72.66 127.841 ms 126.109 ms 127.908 ms
9 209.85.255.68 114.056 ms 116.028 ms
72.14.232.244 118.041 ms
10 209.85.251.37 127.852 ms
209.85.252.2 130.120 ms 130.018 ms
11 72.14.239.93 127.940 ms 128.042 ms 127.902 ms
12 72.14.236.200 126.165 ms 123.836 ms 125.851 ms
13 216.239.49.145 126.229 ms 125.828 ms 125.908 ms
14 8.8.8.8 124.284 ms 125.769 ms 128.827 ms

When I run a ping from pfSense this is what I get:

Ping output:
PING 8.8.8.8 (8.8.8.8) from 192.168.2.2: 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=54 time=125.634 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=124.157 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=125.198 ms

–- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 124.157/124.996/125.634/0.620 ms

But when I run a traceroute from my COMPTUER I get:
C:\Users\Administrator>tracert 8.8.8.8

Tracing route to 8.8.8.8 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.168.1
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * ^C

I also can't ping, or make any other connection to the outside world.

And AGAIN still not showing ANY states or ANYTHING in the firewall logs (even with logging turned on for the default rule. It won't even show what it's NOT blocking. It's just empty as always.)