Traffic inside a MPLS

Frankye

Hi everyone.
Apologies in advance if the question seems stupid, but I'm not very knowledgeable on internet routing and I definitely lack some theory basis.

I'm trying to tackle a main office <-> branch office setup with a MPLS in between them.

The setup looks like this:

             LAN
              |
         main pfsense
              |             _
www------main router         |
              |              |
       external router 1     |
              |              |
       external router 2     |
              |              | MPLS
       external router 3...  |
              |              |
         branch router       |
              |              |
         branch pfsense     _|
              |
             LAN

The main office router can route traffic inside the MPLS or the wider internet (the router has two different outgoing cables). The branch office one can only go through the MPLS.

However the ISP said that traffic from the branch office can enter the MPLS, reach the main office's router, and from there reach the internet.

It's not really working.
And I'm not even sure it's the two firewalls' fault. Said ISP already failed to deliver on other stuff.

Both sides' pfsense interfaces that point to the MPLS routers do not have a next hop defined, and no local address blocking. The traffic passes from one LAN to the other without problems.
I have even forced routing via firewall rules, but the side branch cannot access the internet across the MPLS and out the main branch's router.

I have some (sanitized, sorry, hope it's intellegible) traceroutes from a pc in the branch office's LAN (red lines for emphasis):

$ traceroute host.in.main.LAN
traceroute to host.in.main.LAN (x.x.x.88), 30 hops max, 60 byte packets
1 branch.mpls.router (x.x.x.2) 0.570 ms 0.530 ms 0.549 ms
2 mpls.router.5 (y.y.y.7) 7.347 ms 7.300 ms 7.305 ms
3 mpls.router.4 (y.y.y.229) 9.418 ms 9.371 ms 9.403 ms
4 mpls.router.3 (y.y.y.38) 9.305 ms 10.772 ms 10.207 ms
5 mpls.router.2 (y.y.y.97) 7.826 ms 8.154 ms 8.254 ms
6 mpls.router.1 (y.y.y.98) 8.981 ms 9.124 ms 9.044 ms
7 main.mpls.router (x.x.x.5) 9.058 ms 8.951 ms 8.888 ms
8 host.in.main.LAN (x.x.x.88) 9.211 ms 9.178 ms 9.118 ms

$ traceroute www.google.com
traceroute to www.google.com (216.58.198.4), 30 hops max, 60 byte packets
1 branch.mpls.router (x.x.x.2) 1.468 ms 1.379 ms 0.560 ms
2 mpls.router.5 (y.y.y.7) 7.334 ms 7.290 ms 7.225 ms
3 mpls.router.4 (y.y.y.229) 9.399 ms 9.317 ms 9.368 ms
4 mpls.router.3 (y.y.y.38) 9.305 ms 10.772 ms 10.207 ms
5 mpls.router.2 (y.y.y.97) 7.820 ms 8.140 ms 7.821 ms
6 mpls.router.1 (y.y.y.98) 9.032 ms 9.173 ms 9.101 ms
7 mpls.router.1 (y.y.y.98) 9.212 ms !H

Packet captures shows how traffic from inside the branch office's LAN do exit the firewall from the correct interface. For LAN to LAN traffic it's correctly received on the main office's firewall.
For traffic trying to reach the internet, the y.y.y.98 router always responds with an ICMP destination unreachable packet.

So, before I go on bashing my head against those routers: is it possible/likely this is not actually the two pfsense firewalls' fault, but instead that y.y.y.98 router is playing silly buggers?

Again, sorry if the question sounds stupid, but I very much recognize my ignorance on how a MPLS is supposed to work.

johnpoz

Unless the mpls network routed unknown traffic to your site, no your not going to be able to go out the internet via mpls.. Unless it allowed itself.

If you want branch site to go through the mpls network to get to the internet, you need that default route in the mpls network so when you try to go to google, it know to send it your site to get out.

Or! you run a proxy on your site, and this other site points at the proxy in their browser for internet - since the mpls knows how to get to your network.

Frankye

@johnpoz said in Traffic inside a MPLS:

you need that default route in the mpls network

I would guess that's something the ISP would need to add into the MPLS routing table, right?

@johnpoz said in Traffic inside a MPLS:

Or! you run a proxy on your site, and this other site points at the proxy in their browser for internet - since the mpls knows how to get to your network.

That's something I had not even considered, and a really great idea. Thanks!

johnpoz

Yeah you normally do not have access to manipulate routing inside the mpls network. But you could ask.

Proxy on your end, which you just run on pfsense would be easier way to go for sure ;)

Other solution would be to create a tunnel between their end your end where you could route internet through the tunnel. This removes any routing concerns inside the mpls path.. You could do openvpn from the branch pfsense to yours.