MLPPP performance? (traffic through router slow) [NOT SOLVED] (Bounty?$$)

BBMitch · Nov 30, 2016, 2:09 AM

Using the latest release 2.2.6

After reading a few posts about trouble when the same gateway was used, I simplified my setup, but the same issue remains. I had single link ppp connections configured on the underlying interfaces to allow me backup ways of connecting to manage the router while I messed with my config. If this was the issue, I've removed all that. I now have a single wan, running MLPPP and it connects and routes traffic. No trouble right? But no… performance is horrible.

If I test using iperf, TCP or UDP, I get the effective upload and download of both connections I am linking with MLPPP.

BUT, If I test with say an FTP session behind the router, through it's NAT, I get the effective download of a single connection, and the upload is sporatic - it varies from 10% to 50% of a single connection.

Is this a known issue? Or is there something dumb I might be missing?

Thanks!

Just updated the subject to clarify I am having an issue. Thanks :-)

BBMitch · Nov 28, 2016, 7:25 PM

Adding additional background. The ISP uses cisco gear. They support two modes for MLPPP - one they call "Layer2" the other "Layer3". It's a Telus wholesale so I'm not actually sure where the auth and bonding parts are. I DO have some sample config scripts. But what they show and what pfsense does under the covers don't seem similar.

This config is for a Cisco 871k9 - what is significant to me is…

the way I set up seems to work UNTIL I try to route through the system.
this set up has a different login per connection which is then bundled together.
no idea how I would do that in pfsense?

N.B. to mask parts of the config, I used:
III to replace part of an IP address
PPP# to replace a password
UUU# to replace a username

! at the command prompt, create these vlans
vlan database
vlan 2 name ADSL2
vlan 3 name ADSL3
vlan 4 name ADSL4
exit
! go into the config prompt to program these lines into the k9
config t
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime
service timestamps log datetime localtime
no service password-encryption
!
!creating a command prompt or site name by replacing 970-MLPPP-K9 to new site name
hostname 970-MLPPP-k9
!
boot-start-marker
boot-end-marker
!
enable password cisco
!
no aaa new-model
!
resource policy
!
ip cef
!
! creating the 4 ADSL bonding tunnels
!
interface Tunnel10012
 ip address III.III.0.46 255.255.255.252
 ip load-sharing per-packet
 ip ospf mtu-ignore
 keepalive 5 3
 tunnel source Dialer1
 tunnel destination III.III.86.108
!
interface Tunnel20012
 ip address III.III.0.46 255.255.255.252
 ip load-sharing per-packet
 ip ospf mtu-ignore
 keepalive 5 3
 tunnel source Dialer2
 tunnel destination III.III.86.109
!
interface Tunnel30012
 ip address III.III.0.46 255.255.255.252
 ip load-sharing per-packet
 ip ospf mtu-ignore
 keepalive 5 3
 tunnel source Dialer3
 tunnel destination III.III.86.110
!
interface Tunnel40012
 ip address III.III.0.46 255.255.255.252
 ip load-sharing per-packet
 ip ospf mtu-ignore
 keepalive 5 3
 tunnel source Dialer4
 tunnel destination III.III.86.106
!
!
! setting up the k9 Ethernet ports and assign them to the vlans
interface FastEthernet0
 description ADSL2
 switchport access vlan 2
 no shut
!
interface FastEthernet1
 description ADSL3
 switchport access vlan 3
 no shut
!
interface FastEthernet2
 description ADSL4
 switchport access vlan 4
 no shut
!
interface FastEthernet3
 description LAN
 no shut
!
interface FastEthernet4
 description ADSL1
 no ip address
 ip virtual-reassembly
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 1
 no shut
!
interface Vlan1
description LAN
 ip address III.III.7.129 255.255.255.248
 ip tcp adjust-mss 1380
 no shut
!
interface Vlan2
 description ADSL2
 no ip address
 ip virtual-reassembly
 pppoe enable group global
 pppoe-client dial-pool-number 2
 no shut
!
interface Vlan3
 description ADSL3
 no ip address
 ip virtual-reassembly
 pppoe enable group global
 pppoe-client dial-pool-number 3
 no shut
!
interface Vlan4
 description ADSL4
 no ip address
 ip virtual-reassembly
 pppoe enable group global
 pppoe-client dial-pool-number 4
 no shut
!
interface Dialer1
 ip address negotiated
 ip mtu 1460
 ip virtual-reassembly
 encapsulation ppp
 dialer pool 1
 dialer-group 1
 ppp chap hostname UUU1@pppoe.net
 ppp chap password 0 PPP1 
 ppp authentication pap callin
 ppp pap sent-username UUU1@pppoe.net password 0 PPP1
!
interface Dialer2
 ip address negotiated
 ip mtu 1460
 ip virtual-reassembly
 encapsulation ppp
 dialer pool 2
 dialer-group 2
 ppp chap hostname UUU2@pppoe.net
 ppp chap password 0 PPP2
 ppp authentication pap callin
 ppp pap sent-username UUU2@pppoe.net password 0 PPP2
!
interface Dialer3
 ip address negotiated
 ip mtu 1460
 ip virtual-reassembly
 encapsulation ppp
 dialer pool 3
 dialer-group 3
 ppp chap hostname UUU3@pppoe.net
 ppp chap password 0 PPP3
 ppp authentication pap callin
 ppp pap sent-username UUU3@pppoe.net password 0 PPP4
!
interface Dialer4
 ip address negotiated
 ip mtu 1460
 ip virtual-reassembly
 encapsulation ppp
 dialer pool 4
 dialer-group 4
 ppp chap hostname UUU4@pppoe.net
 ppp chap password 0 PPP4
 ppp authentication pap callin
 ppp pap sent-username UUU4@pppoe.net password 0 PPP4
!
router ospf 100
 log-adjacency-changes
 redistribute connected subnets route-map RDSTB-CON-TO-OSPF
 redistribute static subnets route-map RDSTB-CON-TO-OSPF
 network 10.90.0.0 0.0.255.255 area 0
 network 10.91.0.0 0.0.255.255 area 0
 network 10.92.0.0 0.0.255.255 area 0
 network 10.93.0.0 0.0.255.255 area 0
 network 10.94.0.0 0.0.255.255 area 0
 distribute-list route-map OSPF-IN in
!
ip route 0.0.0.0 0.0.0.0 Dialer1 250
ip route III.III.86.106 255.255.255.255 Dialer4
ip route III.III.86.108 255.255.255.255 Dialer1
ip route III.III.86.109 255.255.255.255 Dialer2
ip route III.III.86.110 255.255.255.255 Dialer3
!
!
no ip http server
no ip http secure-server
!
!
ip prefix-list OSPF-IN seq 5 permit 0.0.0.0/0
ip prefix-list OSPF-IN seq 10 deny 0.0.0.0/0 le 32
!
ip prefix-list RDSTB-CON seq 4 permit 65.110.7.128/29
ip prefix-list RDSTB-CON seq 10 deny 0.0.0.0/0 le 32
!
!
snmp-server community cisco-k9 RO
snmp-server trap-source Vlan1
!
route-map OSPF-IN permit 5
 match ip address prefix-list OSPF-IN
!
route-map OSPF-IN deny 50
!
route-map RDSTB-CON-TO-OSPF permit 10
 match ip address prefix-list RDSTB-CON
!
route-map RDSTB-CON-TO-OSPF deny 100
!
!
control-plane
!
!
line con 0
 no modem enable
 transport preferred none
line aux 0
! creating a telnet login with default password cisco
line vty 0 4
 password k9pass
 login
 transport preferred none
!
scheduler max-task-time 5000
!
webvpn context Default_context
 ssl authenticate verify all
 !
 no inservice
!
! exit back to the command prompt after programmed the k9
end
write mem

When I look at the config pfsense generates, it seems to use the same usernames and passwords for all connections.

I can get online, but for some reason traffic nat'd through the pfsense either drops packets or ? and the performance goes to poop…

Does this config file provide any insight how I might have to adjust my config? Will keep digging.

Thanks!

BBMitch · Nov 28, 2016, 8:52 PM

The RAW services are matching 6Mb down / 1Mb up DSL - the sync rates are about 7Mb down, and 0.95Mb up. Latency is similar.

I tried editing the config and restarting the ppp connection manually to see if that had any effect - doesn't seem to.

Here's an example to clarify the speed difference I am seeing… This FTP was done on the pfsense box to a file located in /tmp (10MB)

ftp> get test.tmp
local: test.tmp remote: test.tmp
229 Entering Extended Passive Mode (|||32549|)
150 Opening BINARY mode data connection for test.tmp (10485760 bytes)
100% |***********************************| 10240 KiB  601.40 KiB/s    00:00 ETA
226 Transfer complete.
10485760 bytes received in 00:17 (599.71 KiB/s)
ftp> put test.tmp
local: test.tmp remote: test.tmp
229 Entering Extended Passive Mode (|||33341|)
150 Opening BINARY mode data connection for test.tmp
100% |***********************************| 10240 KiB  180.28 KiB/s    00:00 ETA
226 Transfer complete.
10485760 bytes sent in 00:57 (179.16 KiB/s)
ftp>

The upload demonstrates the benefits of the bonding in this case, but the download does not… the download is like the download of an unbonded service...

Here, I ran an iperf3 test - both directions behave like bonding is working:

# /usr/local/bin/iperf3 -4 -p 5201 -fm --client 65.110.9.139 --interval 1 --time 120
Connecting to host 65.110.9.139, port 5201
[  4] local 66.196.39.68 port 62255 connected to 65.110.9.139 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   831 KBytes  6.81 Mbits/sec    0   75.7 MBytes
{{SNIP}}
[  4] 119.00-120.00 sec  1.31 MBytes  11.0 Mbits/sec    0    153 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   153 MBytes  10.7 Mbits/sec   48             sender
[  4]   0.00-120.00 sec   153 MBytes  10.7 Mbits/sec                  receiver

iperf Done.
# /usr/local/bin/iperf3 -4 -p 5201 -fm --client 65.110.9.139 --interval 1 --reverse --time 120
Connecting to host 65.110.9.139, port 5201
Reverse mode, remote host 65.110.9.139 is sending
[  4] local 66.196.39.68 port 33740 connected to 65.110.9.139 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   179 KBytes  1.46 Mbits/sec
{{SNIP}}
[  4] 119.00-120.00 sec   179 KBytes  1.46 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  21.0 MBytes  1.47 Mbits/sec   14             sender
[  4]   0.00-120.00 sec  21.0 MBytes  1.47 Mbits/sec                  receiver

iperf Done.

Where it gets really bad though is if I try to work inside the firewall…

The downstream (FTP for example) never gets over 600KiB/s and the upstream averages around 45KiB/s

:(

I'm sure it's got to do something with how I'm configured. The cisco config seems to require OSPF config - is there something like that I should be doing on pfsense?

Thanks!

BBMitch · Nov 28, 2016, 8:51 PM

There is also some support / recommendations for an RB750 config:

!!!setup PPPoE username!!!
/interface pppoe-client add name=MLPPP user=UUU@pppoe.net  password=PPP interface=ether1,ether2,ether3,ether4 add-default-route=yes use-peer-dns=yes allow=chap disabled=no
!!!setup Lan Port!!!
/ip address add address=III.110.7.129/29 255.255.255.248 secondary interface=ether5 disabled=no
/ip address add address=III.110.7.129/29 255.255.255.255 secondary interface=ether5 disabled=no
/ip address add address=routedIP interface=ether5 disabled=no
/ip address add address=192.168.1.1/24  interface=ether5 disabled=no
!!!setup NAT!!!
/ip firewall nat add chain=srcnat src-address=192.168.1.0/24 out-interface=MLPPP action=masquerade disabled=no
!!! setup Password !!!
/password
k9pass
k9pass
!!! setup SNMP !!!
/snmp community add name=k9pass read-access=yes

I don't think the lines above containing "address=III.110.7.129/29" make any sense as the CIDR mask is wrong - but that's what was in the example. Seems to be adding an alias to the LAN for some reason?

But none of this makes any sense as to why the non-natted WAN traffic seems to work for the most part - and only routed / natted traffic seems to be slow?

I can't think of anything to try - hopefully someone sees my mistake / error?

Thanks!

Mitch

BBMitch · Nov 30, 2016, 2:10 AM

Willing to offer a bounty if someone can help resolve. And of course am willing / happy to have said solution / notes posted for all to see.

heper · Nov 30, 2016, 7:07 AM

there are very few people running/have knowledge about mlppp.
also: you run an unsupported version that is no longer used by the vast majority of the community…

(i know nothing about mlppp)

update to a current release, see if the problem still exists
bounty if it isn't urgent / pfSense commercial support if it is urgent

BBMitch · Nov 30, 2016, 8:55 AM

Weird - the unit says it is on the latest release… not sure why it isn't updating.
Sorry about that - I thought it was running the current stable!
I just checked again - the auto update url is set, and it checks and says it's:

Downloading new version information...done
Obtaining current version information...done

You are on the latest version.

Ok - now I see what happened… this unit has a 1GB CF. I guess when the updates switched to 2GB instead of reporting that it could NOT do an upgrade, it started reporting that it was already current. I'll look at a "forklift" upgrade to a new CF.

In my defense, my first port included the version number - which I stupidly thought was current:
Selfquoting "Using the latest release 2.2.6" :'(

I'll try a current release. :-X

Thanks!
m