Routing traffic through PPPOE interfaces

ivanmar

My configuration is based on vlan interfaces:
WAN (wan) -> re0_vlan5 -> 85.xx.xx.xx
LAN (lan) -> re0_vlan10 -> 192.168.100.100
LANP (opt1) -> re0_vlan2 -> 192.168.5.1
WANT (opt2) -> re0_vlan3 -> 85.xx.xx.xx
DSL (opt3) -> pppoe3 -> NONE (PPPoE)

BTW: status of PPPOE on web interface is CONNECTED with assigned ip address.
Traffic can't be routed through any pppoe interface.
Name of dynamic gateways sometimes are shown with GW_ prefix, sometimens only interface name

I have 3 installations with latest builds and 3 same problems.
I think this problem is not related with vlans because one installation does not use vlans and still have same problem.
Maybe user GoldServe has same problem but different manifestation. It seems to me it is related.
In my logs I can only find few instances :
php: : The gateway: opt3 is invalid/unkown not using it.

Routing table that is referencing pppoe link is:
xx.178.192.1 link#11 UH 0 726 1412 pppoe3
xx.178.192.220 link#11 UHS 0 0 16384 lo0
First one is dynamic gateway
Why is it on lo0 interface. Should it be on re0_vlan4 ?

eri--

not enough info.

ivanmar

Little guidance would be helpful. What to look for?

This is output of route monitor : if# 11 is single pppoe interface
got message of size 96 on Wed Jun 9 12:24:33 2010
RTM_IFINFO: iface status change: len 96, if# 11, link: unknown, flags:<up,ptp,running,noarp,simplex,multicast></up,ptp,running,noarp,simplex,multicast>

On web interface same interface is marked as "Online" Status -> Gateways

Still noting passes through that interface. I have setup filtering rule on LAN interface to use that interface as a gateway for all outgoing traffic.
If I change gateway (we have 2 other permanent links), everything works fine. It is not DNS related problem because I tried to access IPs too.

ivanmar

I think there is some problem with routing through dynamic gateways.
Here some more info:

2.0-BETA2
built on Tue Jun 8 02:45:24 EDT 2010 I386

*** Welcome to pfSense 2.0-BETA2-pfSense (i386) on ns1 ***

WAN (wan) -> re0_vlan5 -> 88.114.48.110
LAN (lan) -> re0_vlan10 -> 192.168.100.100
LANPGM (opt1) -> re0_vlan2 -> 192.168.5.1
WANTMP (opt2) -> re0_vlan3 -> 88.114.55.134
DSL (opt3) -> pppoe3 -> NONE (PPPoE)

ifconfig -a

re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=389b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:1c:c0:d0:de:8d
inet6 fe80::21c:c0ff:fed0:de8d%re0 prefixlen 64 scopeid 0x1
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
pfsync0: flags=0<> metric 0 mtu 1460
syncpeer: 224.0.0.240 maxupd: 128
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
nd6 options=3 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33200
re0_vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:1c:c0:d0:de:8d
inet6 fe80::21c:c0ff:fed0:de8d%re0_vlan2 prefixlen 64 scopeid 0x6
inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 2 parent interface: re0
re0_vlan3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:1c:c0:d0:de:8d
inet6 fe80::21c:c0ff:fed0:de8d%re0_vlan3 prefixlen 64 scopeid 0x7
inet 88.114.55.134 netmask 0xfffffffc broadcast 85.114.55.135
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 3 parent interface: re0
re0_vlan4: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:1c:c0:d0:de:8d
inet6 fe80::21c:c0ff:fed0:de8d%re0_vlan4 prefixlen 64 scopeid 0x8
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 4 parent interface: re0
re0_vlan5: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:1c:c0:d0:de:8d
inet6 fe80::21c:c0ff:fed0:de8d%re0_vlan5 prefixlen 64 scopeid 0x9
inet 88.114.48.110 netmask 0xfffffffc broadcast 85.114.48.111
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 5 parent interface: re0
re0_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:1c:c0:d0:de:8d
inet6 fe80::21c:c0ff:fed0:de8d%re0_vlan10 prefixlen 64 scopeid 0xa
inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 10 parent interface: re0
pppoe3: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1412
inet 99.178.192.220 –> 99.178.192.1 netmask 0xffffffff
inet6 fe80::21c:c0ff:fed0:de8d%pppoe3 prefixlen 64 scopeid 0xb
nd6 options=3 <performnud,accept_rtadv># netstat -rn
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 88.114.48.109 UGS 0 6 re0_vl
88.114.48.108/30 link#9 U 4 53 re0_vl
88.114.48.110 link#9 UHS 0 0 lo0
88.114.55.132/30 link#7 U 4 53 re0_vl
88.114.55.134 link#7 UHS 0 0 lo0
99.178.192.1 link#11 UH 0 6794 pppoe3
99.178.192.220 link#11 UHS 0 0 lo0
127.0.0.1 link#4 UH 0 23 lo0
127.0.0.2 127.0.0.1 UHS 0 0 lo0
192.168.5.0/24 link#6 U 0 901976 re0_vl
192.168.5.1 link#6 UHS 0 0 lo0
192.168.100.0/24 link#10 U 4 155288 re0_vl
192.168.100.100 link#10 UHS 0 0 lo0
216.34.181.60 88.114.55.133 UGHS 2 6439 re0_vl

I can ping gateway of PPPOE connection:

ping 99.178.192.1

PING 99.178.192.1 (99.178.192.1): 56 data bytes
64 bytes from 99.178.192.1: icmp_seq=0 ttl=254 time=3.379 ms
64 bytes from 99.178.192.1: icmp_seq=1 ttl=254 time=3.044 ms

Packet Capture shows gateway pings and TCP initialization in one direction (from my host to web pages). Noting is captured in opposite direction or logged in the firewall logs.

13:13:03.026818 AF IPv4 (2), length 64: (tos 0x0, ttl 64, id 33784, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.50.42413 > 209.62.12.163.80: Flags , cksum 0x16ef (correct), seq 1293308779, win 5840, options [mss 1460,sackOK,TS val 7524499 ecr 0,nop,wscale 7], length 0
13:13:03.037551 AF IPv4 (2), length 64: (tos 0x0, ttl 64, id 22037, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.50.43462 > 140.239.191.10.80: Flags , cksum 0x618a (correct), seq 1245157756, win 5840, options [mss 1460,sackOK,TS val 7524500 ecr 0,nop,wscale 7], length 0
13:13:03.291553 AF IPv4 (2), length 64: (tos 0x0, ttl 64, id 33911, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.50.39476 > 83.138.145.146.80: Flags ~~, cksum 0x198d (correct), seq 1294816474, win 5840, options [mss 1460,sackOK,TS val 7524525 ecr 0,nop,wscale 7], length 0~~</performnud,accept_rtadv></up,pointopoint,running,noarp,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,simplex,multicast>

eri--

Can you show the filter rules?

ivanmar

This is setup where my local network (192.168.100.0/24) is router through pppoe connection - not working.
Just to repeat that I have 2 other instalations (almost same but with different dsl providers) and same problem.

FILTER RULES:
scrub in on re0_vlan5 all max-mss 1460 fragment reassemble
scrub in on re0_vlan10 all max-mss 1460 fragment reassemble
scrub in on re0_vlan2 all max-mss 1460 fragment reassemble
scrub in on re0_vlan3 all max-mss 1460 fragment reassemble
anchor "relayd/" all
anchor "firewallrules" all
block drop in log all label "Default deny rule"
block drop out log all label "Default deny rule"
block drop in quick inet6 all
block drop out quick inet6 all
block drop quick proto tcp from any port = 0 to any
block drop quick proto tcp from any to any port = 0
block drop quick proto udp from any port = 0 to any
block drop quick proto udp from any to any port = 0
block drop quick from <snort2c>to any label "Block snort2c hosts"
block drop quick from any to <snort2c>label "Block snort2c hosts"
anchor "packageearly" all
anchor "carp" all
block drop in log quick proto tcp from <sshlockout>to any port = 3322 label "sshlockout"
block drop in quick from <virusprot>to any label "virusprot overload table"
anchor "wanbogons" all
block drop in log quick on re0_vlan5 from <bogons>to any label "block bogon networks from WAN"
block drop in on ! re0_vlan5 inet from 88.114.48.108/30 to any
block drop in inet from 88.114.48.110 to any
block drop in on re0_vlan5 inet6 from fe80::21c:c0ff:fed0:de8d to any
block drop in log quick on re0_vlan5 inet from 10.0.0.0/8 to any label "block private networks from wan block 10/8"
block drop in log quick on re0_vlan5 inet from 127.0.0.0/8 to any label "block private networks from wan block 127/8"
block drop in log quick on re0_vlan5 inet from 172.16.0.0/12 to any label "block private networks from wan block 172.16/12"
block drop in log quick on re0_vlan5 inet from 192.168.0.0/16 to any label "block private networks from wan block 192.168/16"
block drop in on ! re0_vlan10 inet from 192.168.100.0/24 to any
block drop in inet from 192.168.100.100 to any
block drop in on re0_vlan10 inet6 from fe80::21c:c0ff:fed0:de8d to any
anchor "dhcpserverLAN" all
pass in on re0_vlan10 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server"
pass in on re0_vlan10 inet proto udp from any port = bootpc to 192.168.100.100 port = bootps keep state label "allow access to DHCP server"
pass out on re0_vlan10 inet proto udp from 192.168.100.100 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
block drop in on ! re0_vlan2 inet from 192.168.5.0/24 to any
block drop in inet from 192.168.5.1 to any
block drop in on re0_vlan2 inet6 from fe80::21c:c0ff:fed0:de8d to any
anchor "opt2bogons" all
block drop in log quick on re0_vlan3 from <bogons>to any label "block bogon networks from WANTMP"
block drop in on ! re0_vlan3 inet from 88.114.55.132/30 to any
block drop in inet from 88.114.55.134 to any
block drop in on re0_vlan3 inet6 from fe80::21c:c0ff:fed0:de8d to any
block drop in log quick on re0_vlan3 inet from 10.0.0.0/8 to any label "block private networks from wan block 10/8"
block drop in log quick on re0_vlan3 inet from 127.0.0.0/8 to any label "block private networks from wan block 127/8"
block drop in log quick on re0_vlan3 inet from 172.16.0.0/12 to any label "block private networks from wan block 172.16/12"
block drop in log quick on re0_vlan3 inet from 192.168.0.0/16 to any label "block private networks from wan block 192.168/16"
anchor "spoofing" all
anchor "loopback" all
pass in on lo0 all flags S/SA keep state label "pass loopback"
pass out on lo0 all flags S/SA keep state label "pass loopback"
anchor "firewallout" all
pass out all flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (re0_vlan5 88.114.48.109) inet from 88.114.48.110 to ! 88.114.48.108/30 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (re0_vlan3 88.114.55.133) inet from 88.114.55.134 to ! 88.114.55.132/30 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
anchor "anti-lockout" all
pass in quick on re0_vlan2 inet from 192.168.5.11 to any flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan2 inet from 192.168.5.0/24 to 192.168.100.111 flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan2 inet from 192.168.5.0/24 to 192.168.100.110 flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan2 route-to (re0_vlan3 88.114.55.133) inet proto udp from 192.168.5.0/24 to any port = domain keep state label "USER_RULE"
pass in quick on re0_vlan2 route-to (re0_vlan3 88.114.55.133) inet proto udp from 192.168.5.0/24 to any port = ntp keep state label "USER_RULE"
pass in quick on re0_vlan2 route-to (re0_vlan3 88.114.55.133) inet proto udp from 192.168.5.0/24 to any port = radius keep state label "USER_RULE"
pass in quick on re0_vlan2 route-to (re0_vlan3 88.114.55.133) inet proto udp from 192.168.5.0/24 to any port = radacct keep state label "USER_RULE"
pass in quick on re0_vlan10 inet from 192.168.100.0/24 to 192.168.5.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan10 inet from 192.168.100.0/24 to 192.168.100.100 flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan10 inet proto icmp from 192.168.100.0/24 to any keep state label "USER_RULE"
pass in quick on re0_vlan10 inet proto tcp from 192.168.100.111 to <vpns>flags S/SA keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
pass in quick on re0_vlan10 route-to (re0_vlan5 88.114.48.109) inet proto tcp from 192.168.100.111 to any port = smtp flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan10 inet proto tcp from 192.168.100.111 to <vpns>flags S/SA keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
pass in quick on re0_vlan10 inet proto udp from 192.168.100.111 to <vpns>keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
pass in quick on re0_vlan10 route-to (re0_vlan5 88.114.48.109) inet proto tcp from 192.168.100.111 to any port = domain flags S/SA keep state label "USER_RULE"
pass in quick on re0_vlan10 route-to (re0_vlan5 88.114.48.109) inet proto udp from 192.168.100.111 to any port = domain keep state label "USER_RULE"
pass in quick on re0_vlan10 inet from 192.168.100.0/24 to <vpns>flags S/SA keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
pass in quick on re0_vlan10 route-to (pppoe3 99.178.192.1) inet from 192.168.100.0/24 to any flags S/SA keep state label "USER_RULE"
anchor "packagelate" all
anchor "tftp-proxy/" all
anchor "limitingesr" all
anchor "miniupnpd" all
No queue in use</vpns></vpns></vpns></vpns></bogons></bogons></virusprot></sshlockout></snort2c></snort2c>

ivanmar

I can't belive I am the only with this problem. I use VLANs, but configuration is prety ordinary. I can see there are many ongoing changes in recent snapshots on pppoe interfaces…

gnhb

Hi,

Your problem is solved by this commit.
https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/58cbe016de8cd2e88b08529f6272da849251647a

Your PPPoE interface is last in your interface list and a code change on Jun3 caused the last interface in the list to not be recognized as an interface that really exists.

Look at the time stamp in the commit and select an update or a new build from the snapshot server that is newer.

GB

ivanmar

This was a big problem for me.
God bless you :)

gnhb

The code change is small. You can do it yourself if you do want to get a new build.

ivanmar

That did a trick even without restart. Thanks