Internet connection sharing and routing

luckylinux · Feb 11, 2012, 11:18 PM

Hello.
After a lot of troubles I successfully managed to install Pfsense on my new router (AMD E-350, 8GB of RAM and 2x500GB HDD in GEOM GMIRROR configuration) from an USB key (a kernel panic appears when using the CD's ISO).
I can get to the web interface and tried to fix the problem myself.

Now let be the computer where Pfsense is installed be the "router" (though it will be a switch and other things as well) and a PC inside the LAN be the "client".
This is a home based setup, made because I wanted link aggregation and I got tired of the Zyxel's limitations on number of rules for the firewall I could create (only 10). My internet connection comes from an ISP cable's provider that gets converted to a Ethernet port through a modem the ISP gave me. After the modem the ethernet port gives unrestricted access to the WAN (I do not need to authenticate myself !).

Now the problems are as follows:
a) The SERVER can only be accessed by the CLIENT if the SERVER's WAN (yes, WAN !) interface is connected to the CLIENT (directly or in this transition phase, through the original Zyxel router I was using before). If I connect SERVER and CLIENT through an external switch or by using a crossover cable nothing happens. Using the OPT1 (backup / redundancy LAN port) doesn't help at all. I find this pretty odd …

b) Because of point a no internet connection sharing is possible. I don't know the exact term for this (I think on GNU/Linux it's called Internet Connection Sharing, but correct me if I'm wrong) no internet connection is accessible to the CLIENT. I think it's kinda a bridging issue, i.e. addressing packets from the WAN to the CLIENT inside the LAN. I'm no expert in networking but I don't think this feature is called NAT (isn't that related to "open" ports and directly forward incoming packets from the WAN to the LAN). If I don't connect the WAN port of the ROUTER to the Zyxel router, but connect it to the ISP cable's provider's modem's ethernet port (the connection is said to be "Ethernet bridging" on my Zyxel router, i.e. nothing like PPoE or PPoA) no connection is available.

c) Moreover than the WAN access problems of point b, if the WAN interface is connected to the cable's ISP modem I can ping all devices I want (network printers, Zyxel router, ...) except the ROUTER where Pfsense is installed. Strangely enough the LAN and OPT1 ports seem to be pretty invisible to the external world.

Surely it's a very simple problem (for you), but for me this is quite hard to figure out.
Do you have any suggestions ? I attached the two configurations cited above as a network map (made in Dia). The "pfsense_tomakeitwork.png" refers to make it ping on the LAN. If a client is directly connected through the ROUTER (the pfsense one) this does not make the WAN translation work.

Thank you very much in advance.

EDIT: I'm not using right now the HP Procurve 1810-24g switch, but I'm gonna install it as soon as I can get Pfsense to work.

EDIT: sometimes even the "funky" method doesn't work (as presented in the diagram) and I'm completely locked out of the Pfsense ROUTER. From it I can correctly ping the CLIENT's ip address (LAN) when the WAN (!) interface of the Pfsense ROUTER is connected to a LAN port of the Zyxel router (see pfsense_tomakeitwork.png). With the WAN disconnected on the Pfsense ROUTER and the LAN cable connected from the Pfsense ROUTER's LAN / OPT1 interface to one of Zyxel's LAN ports, when trying to ping from the ROUTER I get "ping: sendto: No buffer space available". Pretty strange … I don't think my RAM is full. To sum it up ... the ROUTER can see the CLIENT if and only if the WAN-PFSENSE port is connected to the LAN-Zyxel port and the CLIENT can (SOMETIMES) see the Pfsense ROUTER in that condition.

stephenw10 · Feb 12, 2012, 2:33 AM

Hmm, where to start. ;)

'Internet Connection Sharing' is a Windows term. It's not usually used in either Linux or FreeBSD (which is what pfSense is built on). It generally refers to a combination of services including DHCP and NAT.

You talk about a SERVER but it's not in your diagram. I assume you mean the pfSense box.

I think you almost certainly have a IP address conflict. If you have both pfSense and your Zyxel router connected they are both trying to hand out IP addresses on your network. It's possible that they are both on the same subnet, maybe even using the same IP, causing all sorts of problems.

When you connect your pfSense WAN interface to the cable modem does it receive an address? You may have to power cycle the modem.

Steve

luckylinux · Feb 12, 2012, 8:06 AM

@stephenw10:

'Internet Connection Sharing' is a Windows term. It's not usually used in either Linux or FreeBSD (which is what pfSense is built on). It generally refers to a combination of services including DHCP and NAT.

My bad … sorry. Although for now let's leave DHCP out of the problem as far as the LAN is concerned (I only use static IPs in the LAN, the WAN interface is DHCP though)

@stephenw10:

You talk about a SERVER but it's not in your diagram. I assume you mean the pfSense box.

Again my bad … must've been very tired yesterday after struggling for a few hours ??? ??? ... sorry

@stephenw10:

I think you almost certainly have a IP address conflict. If you have both pfSense and your Zyxel router connected they are both trying to hand out IP addresses on your network. It's possible that they are both on the same subnet, maybe even using the same IP, causing all sorts of problems.

Very likely it's "only" a subnet conflict as normally I take good track of the static IPs allocated in my network and there are not so many computers either.
However one thing isn't clear at all to me: in the web interface sometimes subnet is referred to something as 255.255.255.0 and other times as /24 or /30. Is that the same thing ? During the initial configuration with keyboard on the Pfsense box it says "number of bits … (/24 for eg. 255.255.255.0 or /30 for 255.255.255.255)" and I selected /24. There shouldn't be a conflict however if the Pfsense ROUTER and the Zyxel router are not connected by any means (e.g. Pfsense ROUTER connected to the WAN cable's modem ethernet port and to the CLIENT directly through a CROSSOVER cable. Or am I (so much) wrong ??? ?

I think this is definitively an issue since I think 4 users including me posted problems about the same thing the same day ... :o

@stephenw10:

When you connect your pfSense WAN interface to the cable modem does it receive an address? You may have to power cycle the modem

I get an IP address (on the WAN interface) which is 84.AB.CD.EF so it seems fine. No need to power off the modem. I can successfully ping e.g. google.com. However as I told before I get locked out on the client. With only the Pfsense box and a CROSSOVER connection to the client (tried even with switch …) I cannot ping the LAN (192.168.1.1) or OPT1 (192.168.1.200) interface of the ROUTER, let alone google.com and the likes :'(.

Thank you very much for your help. At least it's not a WAN problem :).
I'm positive with your help I'd be able to fix this problem ;)

EDIT: please note I had to replug the Zyxel router WAN interface on the cable's modem before I could reply ;)

stephenw10 · Feb 12, 2012, 1:33 PM

@luckylinux:

However one thing isn't clear at all to me: in the web interface sometimes subnet is referred to something as 255.255.255.0 and other times as /24 or /30. Is that the same thing ? During the initial configuration with keyboard on the Pfsense box it says "number of bits … (/24 for eg. 255.255.255.0 or /30 for 255.255.255.255)" and I selected /24.

Yes they are two ways of defining the subnet mask. I agree it's confusing.
See: http://en.wikipedia.org/wiki/IPv4_subnetting_reference

@luckylinux:

I get an IP address (on the WAN interface) which is 84.AB.CD.EF so it seems fine. No need to power off the modem. I can successfully ping e.g. google.com. However as I told before I get locked out on the client. With only the Pfsense box and a CROSSOVER connection to the client (tried even with switch …) I cannot ping the LAN (192.168.1.1) or OPT1 (192.168.1.200) interface of the ROUTER, let alone google.com and the likes :'(.

Hmm, OK.
So the WAN interface is connecting and getting it's IP correctly and you can ping out. All good.
Is your client being given an IP address by the pfSense box correctly?
I suggest you concentrate on getting the LAN interface working correctly before connecting the OPT interface.
Check the DHCP server settings. Make sure it's handing out the correct gateway and dns addresses and subnet mask. It's easy to accidentally set the subnet mask to /32 in the dhcp sever which will cause a problem.
If your client is a Windows box it should have settings something like:


C:\Documents and Settings\Steve>ipconfig /all

Windows IP Configuration

        Host Name . . . . . . . . . . . . : NewTuring
        Primary Dns Suffix  . . . . . . . :
        Node Type . . . . . . . . . . . . : Mixed
        IP Routing Enabled. . . . . . . . : No
        WINS Proxy Enabled. . . . . . . . : No
        DNS Suffix Search List. . . . . . : fire.box

Ethernet adapter Local Area Connection:

        Connection-specific DNS Suffix  . : fire.box
        Description . . . . . . . . . . . : Realtek RTL8139/810X Family PCI Fast Ethernet NIC
        Physical Address. . . . . . . . . : 00-30-1B-AB-18-C3
        Dhcp Enabled. . . . . . . . . . . : Yes
        Autoconfiguration Enabled . . . . : Yes
        IP Address. . . . . . . . . . . . : 192.168.2.10
        Subnet Mask . . . . . . . . . . . : 255.255.255.0
        Default Gateway . . . . . . . . . : 192.168.2.1
        DHCP Server . . . . . . . . . . . : 192.168.2.1
        DNS Servers . . . . . . . . . . . : 192.168.2.1
        Lease Obtained. . . . . . . . . . : 12 February 2012 12:58:10
        Lease Expires . . . . . . . . . . : 12 February 2012 14:58:10

Try running ipconfig /all on your box what does it show?

Steve

luckylinux · Feb 12, 2012, 1:46 PM

@stephenw10:

@luckylinux:

However one thing isn't clear at all to me: in the web interface sometimes subnet is referred to something as 255.255.255.0 and other times as /24 or /30. Is that the same thing ? During the initial configuration with keyboard on the Pfsense box it says "number of bits … (/24 for eg. 255.255.255.0 or /30 for 255.255.255.255)" and I selected /24.

Yes they are two ways of defining the subnet mask. I agree it's confusing.
See: http://en.wikipedia.org/wiki/IPv4_subnetting_reference

@luckylinux:

I get an IP address (on the WAN interface) which is 84.AB.CD.EF so it seems fine. No need to power off the modem. I can successfully ping e.g. google.com. However as I told before I get locked out on the client. With only the Pfsense box and a CROSSOVER connection to the client (tried even with switch …) I cannot ping the LAN (192.168.1.1) or OPT1 (192.168.1.200) interface of the ROUTER, let alone google.com and the likes :'(.

Hmm, OK.
So the WAN interface is connecting and getting it's IP correctly and you can ping out. All good.
Is your client being given an IP address by the pfSense box correctly?
I suggest you concentrate on getting the LAN interface working correctly before connecting the OPT interface.
Check the DHCP server settings. Make sure it's handing out the correct gateway and dns addresses and subnet mask. It's easy to accidentally set the subnet mask to /32 in the dhcp sever which will cause a problem.
If your client is a Windows box it should have settings something like:
C:\Documents and Settings\Steve>ipconfig /all

Windows IP Configuration

        Host Name . . . . . . . . . . . . : NewTuring
        Primary Dns Suffix  . . . . . . . :
        Node Type . . . . . . . . . . . . : Mixed
        IP Routing Enabled. . . . . . . . : No
        WINS Proxy Enabled. . . . . . . . : No
        DNS Suffix Search List. . . . . . : fire.box

Ethernet adapter Local Area Connection:

        Connection-specific DNS Suffix  . : fire.box
        Description . . . . . . . . . . . : Realtek RTL8139/810X Family PCI Fast Ethernet NIC
        Physical Address. . . . . . . . . : 00-30-1B-AB-18-C3
        Dhcp Enabled. . . . . . . . . . . : Yes
        Autoconfiguration Enabled . . . . : Yes
        IP Address. . . . . . . . . . . . : 192.168.2.10
        Subnet Mask . . . . . . . . . . . : 255.255.255.0
        Default Gateway . . . . . . . . . : 192.168.2.1
        DHCP Server . . . . . . . . . . . : 192.168.2.1
        DNS Servers . . . . . . . . . . . : 192.168.2.1
        Lease Obtained. . . . . . . . . . : 12 February 2012 12:58:10
        Lease Expires . . . . . . . . . . : 12 February 2012 14:58:10
Try running ipconfig /all on your box what does it show?

Steve

To be honest right now I'm using GNU/Linux and I was trying to get a static IP (with subnet 255.255.255.0 and default gateway 192.168.1.1, which should point to the Pfsense ROUTER). DHCP on the LAN is not one of the most important things right now. Anyway I cannot ping anything when I connect the CLIENT to the Pfsense ROUTER.

stephenw10 · Feb 12, 2012, 2:16 PM

Hmm OK,
Have you changed the LAN settings in the pfSense box at all?
If your client is in the same subnet with the correct mask and gateway (which it looks like it has) you should be able to ping the LAN interface.
Can you check the firewall logs? Are you running any sort of firewall on your client?

Otherwise I'd start looking at hardware problems.
Try re-assigning the NICs in the pfSense box. Use the NIC you are currently using for WAN (since you know it is working) as LAN.

Steve

luckylinux · Feb 12, 2012, 2:53 PM

@stephenw10:

Hmm OK,
Have you changed the LAN settings in the pfSense box at all?
If your client is in the same subnet with the correct mask and gateway (which it looks like it has) you should be able to ping the LAN interface.
Can you check the firewall logs? Are you running any sort of firewall on your client?

Otherwise I'd start looking at hardware problems.
Try re-assigning the NICs in the pfSense box. Use the NIC you are currently using for WAN (since you know it is working) as LAN.

Steve

When things weren't working I even tried to bridge the LAN over the WAN but anyway since the LAN didn't work before and it doesn't work right now …
Still ... I changed the IP on the lan interface .. nothing changed
I tried to use the WAN interface on the LAN: I can ping my old Zyxel router but not the client (and the client can ping the Zyxel router but not the Pfsense box).

Now I resetted to factory defaults and I can even access google.com from my CLIENT computer connected to the Pfsense Box. Still I find pretty strange because I had not configured anything "strange". Well ... thanks anyway ... for now at least it seems to work ... not so sure until when though ...

But I have another small problem: do you think it's normal that out of two PCIe NIC of the same model (INTEL, see my signature) only one is working correctly with autoconfiguration ? Plus the PLANET NIC (I think it's a Realtek 8169 ?) always states "write failed" or something like this when trying the autoconfigurator.
Right now it's not so critical but I wanted to setup link aggregatin for the LAN (3x) and failover backup for the WAN (2x), since I have 1 OnBoard NIC plus 4 plugged in as PCIe / PCI. I looked at LAGG and did not quite understand ... what does the panel mean by "Only unassigned interfaces can be added to LAGG. " ?

Which are the commands from the shell to check the NICs ? On GNU/Linux you usually have ifconfig / lspci / dmesg that tells almost everything you need but the lspci command does not work. Is it possible that the PCI(e) card(s) is/are not plugged in fully / enough in their respectives slots ?
Thank you very much.

stephenw10 · Feb 12, 2012, 3:34 PM

The equivalent BSD command to lspci is pciconf.

The bootup messages from the last boot are held in /var/log/dmesg.boot

@luckylinux:

But I have another small problem: do you think it's normal that out of two PCIe NIC of the same model (INTEL, see my signature) only one is working correctly with autoconfiguration ? Plus the PLANET NIC (I think it's a Realtek 8169 ?) always states "write failed" or something like this when trying the autoconfigurator.

No it's probably not normal. Are they both known working NICs?
By 'autoconfiguration' do you mean the autodetect feature when you first assign the NICs?

@luckylinux:

Right now it's not so critical but I wanted to setup link aggregatin for the LAN (3x) and failover backup for the WAN (2x), since I have 1 OnBoard NIC plus 4 plugged in as PCIe / PCI. I looked at LAGG and did not quite understand … what does the panel mean by "Only unassigned interfaces can be added to LAGG. " ?

Unassigned NICs are just that. When you first install pfSense you have to assign NICs to WAN and LAN but any others are optional. You can have many NICs in your box, and they will be shown by both ifconfig and pciconf, but pfSense will ignore them until you assign them to a pfSense interface.
You can do that at any time in the webGUI under Interface: (assign):
A LAGG interface requires two or more NICs and these must not have already been assigned to another interface.

It is possible that the NICs are not fully seated in their slot but I would not expect them to show at all if that was the case.

Steve

luckylinux · Feb 12, 2012, 4:56 PM

@stephenw10:

No it's probably not normal. Are they both known working NICs?
By 'autoconfiguration' do you mean the autodetect feature when you first assign the NICs?

I think they are (at least the one in the PCIe x1 slot). To make sure I swapped another INTEL NIC (same model) and put it in the PCIe x16 (x4 electrical) slot. Same problem: green LED is on (NIC active), but no yellow LED (LINK 10/100/1000), same as with the other NIC. I suspect therefore that the cause relies in the x16 slot … quite strange ... it doesn't seem damaged at all and I successfully put x1 cards in x16 slots on other computers and they're working fine. Am I missing something ? In the BIOS there is no option to set the x16 speed as far as I can tell. Could be a BIOS update needed ? Maybe I'll put a graphic card (x16) in there and see if it works ...

Autoconfiguration I mean the autodetect feature to automatically find the names of the network interfaces (re1, em0, em1, re0 in my case).

@stephenw10:

Unassigned NICs are just that. When you first install pfSense you have to assign NICs to WAN and LAN but any others are optional. You can have many NICs in your box, and they will be shown by both ifconfig and pciconf, but pfSense will ignore them until you assign them to a pfSense interface.
You can do that at any time in the webGUI under Interface: (assign):
A LAGG interface requires two or more NICs and these must not have already been assigned to another interface.

So … If I'd want to do link aggregation and combine 3 NICs to make "one big 3gbps" NIC I'd rather not assign the LAN interface and use the LAN, OPT1 and OPT2 NICs as unassigned to make LAGG ? Or is the LAN configuration mandatory ?

@stephenw10:

It is possible that the NICs are not fully seated in their slot but I would not expect them to show at all if that was the case.

After swapping the INTEL NIC (see above) with an identical one it didn't get fixed. However after swapping the realtek one (PCI) with an identical one the autodetec feature worked like a charm and the two LED flashed just fine (with the faulty one I continued to get a lot of failed writes to PHY).
I surely have bad luck with components these days …

stephenw10 · Feb 12, 2012, 5:27 PM

I seem to remember something about the AMD E350 cpu and PCI-e slots. I'm not sure but because the CPU has a GPU built into it it somehow had an effect. :-\ Hmm, I could have that wrong.

LAN, OPT1 and OPT2 are all pfSense interface names to which NICs have already been assigned.
I have never used LAGG so I'm as unfamiliar as you are but your would create the LAGG interface and assign to it re0, re1 and em0, for example.

Why are you using LAGG? Just to experiment?

Steve

luckylinux · Feb 12, 2012, 5:33 PM

You may be right about the x16 slot. A bios update may make things more clear though.

Using LAGG only to experiment right now, however I'd like to have a faster network than 1gbps (even if it is a REAL 200MB/s I'd be pretty happy with it), mainly for doing backups on the LAN without affecting other traffic (media server, etc). Since I heard you can get half the speed of a 10gbps equipement for 1/10 of the price I think that's worth a try … 10gbps switches and NICs are so much expensive (> 700$ NICs and >1000$ switches). That''s my home network so I don't really want to spend that kind of money ... plus if I have multiple parallel lines I can have redundancy as well ;)

I'm gonna post the results of the bios update ;)

stephenw10 · Feb 12, 2012, 5:43 PM

I think you may be confusing the flow of traffic on the network.
If you have your procurve switch connected to the pfSense LAN interface and then you have your clients and other servers also connected to that switch then traffic from one client to a server does not go via the pfSense box at all. This means there would be no advantage to having a double width pipe between the switch and the pfSense box.
If you are talking about doing backup to a remote server out on the internet somewhere then the speed will be limited by your WAN connection and will make almost no difference to internal traffic.

Steve

luckylinux · Feb 12, 2012, 6:28 PM

Mmm … Fortunately somebody more informed than me pointed that out. Thanks for that ;)

Well then ... that pfsense box will only do routing / firewalling / NAT / VPN / ... not backups. I have a dedicated machine that does backups, another one which will handle media, etc

Glad to hear that ... And I think it wouldn't changed much in my bills either, because I had another E-30 (an MSI) which has a single x16 slot and probably has the same problem. Therefore the microATX case I chose (FRACTAL DESIGN ARC MINI, rather nice I think) seems to be a good choice.

If it is as you pointed out then I'll limit myself to doing only failover and leave link aggregation on the "client" machines and switches.
I think you're right since one user in another forum pointed that out as well ... but as I said to him I think something's odd, because with a 100BaseT router (Zyxel) and everything else gigabit ethernet (1000Mbps) I couldn't get transfers above 12MB/s (instead of the theoricals 125MB/s). Will have to test that out again with this setup.

In that case I'll just use 3 interfaces as failover plus one WAN interface OR 2xWAN failovers and 2xWAN failovers connected to an average switch (definitively not a procurve).

As a side note, do you know why the HP Procurves 1810-[8/24]g switches I bought costs a lot less than the 1800-[8/24]g ? Both support link aggregation and the 1810-* seems more energy efficient. Do you know anything about it ?

Last but not least now the problem got reversed: I can perfectly access the WAN but couldn't get access to the web configurator but I can ping the LAN interface of the Pfsense box. This is so screwed up … could it be the OPT interfaces I enabled (though with different ip addresses) ?

Many many thanks stephenw10 ;). Today you really helped me getting started with this router !