CsC
CsC (Carrier supporting Carrier) is defined in RFC 4364.
Control-Plane
- The Customer Carrier PEs run BGP VPNv4 in order to exchange VPN labels
- The Customer Carrier routers run IGP+LDP (or iBGP+Label) in order to exchange all their internal BGP next-hops and their labels
- The CsC-PEs and CsC-CEs run eBGP (or IGP) in order to exchange BGP next-hop prefixes
- The CsC-PEs and CsC-CEs run eBGP+Label (or IGP+LDP) in order to exchange labels for the BGP next-hop prefixes
- The Backbone Carrier routers run IGP+LDP in order to exchange all their internal BGP next-hops and their labels
The Backbone Carrier offers a MPLS VPN service to the Customer Carrier which in turn offers a MPLS VPN or Internet service to its customers.
The Backbone Carrier doesn't need to know the final customer prefixes.
Using IGP+LDP in CsC is not as risky as with Inter-AS MPLS VPN Option 3 because:
- Customer Carrier internal routes are put into a specific VRF in the Backbone Carrier
- No Backbone Carrier internal routes are distributed into the Customer Carrier network
You can have multiple Backbone Carriers, using Inter-AS MPLS L3VPN for interconnection.
By default a CsC-PE runs PHP towards the CsC-CE. If using an ipv4-labeled PE-CE session, you can change this behavior (in order to keep the QoS consistent across providers) by using the "neighbor x.x.x.x send-label explicit-null" on the CsC-CE.
IOS-XR supports only the use of Labeled BGP as a PE-CE protocol in CsC topologies. LDP (+IGP) is not supported.
CsC Load Balancing
Load balancing between CsC-PE and CsC-CE can be achieved with:
- directly connected loopback peering for one pair of PE/CE
- one eBGP session between neighbors
- multiple static routes for each other's loopback
- mpls forwarding on all directly connected physical interfaces
- eBGP multipath for multiple pairs of PEs/CEs
- maximum-paths under bgp & vrf address family on PE
- maximum-paths under bgp on CE
When using static routes, you also need to define the outgoing interface and the next-hop.
"mpls bgp forwarding" is not automatically added, because the BGP session is not between directly connected neighbors. You have to add it yourself.
Configuration
BC = Backbone Carrier (AS10)
CC = Customer Carrier (AS100)
C = Customer (AS200)
Backbone Carrier runs IS-IS or OSPF with MPLS/LDP in its core
Backbone Carrier (CsC-PE1) runs OSPF+LDP with Customer Carrier (CsC-CE1)
Backbone Carrier (CsC-PE2) runs eBGP+Label with Customer Carrier (CsC-CE2)
Customer Site 1 (C-CE1) runs OSPF with Customer Carrier (CC-PE1)
Customer Site 2 (C-CE2) runs ISIS with Customer Carrier (CC-PE2)
CC-PE (Customer Carrier PE serving the final customer site) and CsC-CE (Carrier supporting Carrier CE) functionalities can be collapsed into a single router.
CsC-PE1 and CsC-PE2 run iBGP VPNv4 in order to exchange Customer Carrier prefixes/labels
CsC-CE1 and CsC-CE2 run iBGP VPNv4 in order to exchange Customer prefixes/labels
IGP+LDP between CsC-PE1 and CsC-CE1
CsC-PE1 (IOS)
! for connectivity to BC core (IGP+LDP)
mpls ldp router-id Loopback0
mpls label protocol ldp
!
interface Ethernet0/2
description ** Link to BC core **
ip address x.x.x.x
mpls ip
!
router isis/ospf x
!
! for connectivity to CsC-CE1 (OSPF+LDP)
vrf definition CC-VPN
rd 10:X
route-target 10:X
!
address-family ipv4
exit-address-family
!
interface Ethernet1/0
description ** Link to CsC-CE1 **
vrf forwarding CC-VPN
ip address x.x.x.x
mpls ip
!
router ospf 10 vrf CC-VPN
redistribute bgp 10 subnets
network x.x.x.x area 0
!
! for connectivity to BC-PE2 (iBGP VPNv4)
router bgp 10
no bgp default ipv4-unicast
neighbor BC-PE2 remote-as 10
neighbor BC-PE2 update-source Loopback0
!
address-family vpnv4
neighbor BC-PE2 activate
neighbor BC-PE2 send-community extended
exit-address-family
!
address-family ipv4 vrf CC-VPN
redistribute ospf 10 vrf CC-VPN
exit-address-family
CsC-CE1 (IOS)
! for connectivity to CsC-PE1 (OSPF+LDP)
mpls ldp router-id Loopback0
mpls label protocol ldp
!
interface Ethernet1/0
description ** Link to CsC-PE1 **
ip address x.x.x.x
mpls ip
!
router ospf 10
network x.x.x.x area 0
!
CC-PE1 (IOS)
! for connectivity to C-CE1 (OSPF+VRF)
vrf definition C-VPN
rd 100:Y
route-target 100:Y
!
address-family ipv4
exit-address-family
!
interface Ethernet1/3
description ** Link to C-CE1 **
vrf forwarding C-VPN
ip address y.y.y.y
!
router ospf 200 vrf C-VPN
redistribute bgp 100 subnets
network y.y.y.y area 0
!
! for connectivity to CC-PE2 (iBGP VPNv4)
router bgp 100
no bgp default ipv4-unicast
neighbor CC-PE2 remote-as 100
neighbor CC-PE2 update-source Loopback0
!
address-family vpnv4
neighbor CC-PE2 activate
neighbor CC-PE2 send-community extended
exit-address-family
!
address-family ipv4 vrf C-VPN
redistribute ospf 200 vrf C-VPN
exit-address-family
!
BGP+Label between CsC-PE2 and CsC-CE2
CsC-PE2 (IOS-XR)
! for connectivity to BC core (IGP+LDP)
mpls ldp router-id Loopback0
mpls label protocol ldp
!
router isis/ospf x
!
mpls ldp
router-id x.x.x.x
interface x
!
! for connectivity to CsC-CE2 (eBGP+Label)
vrf CC-VPN
address-family ipv4 unicast
import route-target
10:X
export route-target
10:X
!
interface GigabitEthernet0/2/1/1
description ** Link to CsC-CE2 **
vrf CC-VPN
ipv4 address x.x.x.x
!
router static
vrf CC-VPN
address-family ipv4 unicast
CsC-CE2/32 GigabitEthernet0/2/1/1
!
router bgp 10
address-family ipv4 unicast
!
vrf CC-VPN
rd 10:X
address-family ipv4 unicast
network x.x.x.x
allocate-label all
!
neighbor CsC-CE2
remote-as 100
address-family ipv4 unicast
route-policy PASS-RPL in
route-policy PASS-RPL out
as-override
send-extended-community-ebgp
!
address-family ipv4 labeled-unicast
route-policy PASS-RPL in
route-policy PASS-RPL out
as-override
send-extended-community-ebgp
!
route-policy PASS-RPL
pass
end-policy
!
! for connectivity to BC-PE1 (iBGP VPNv4)
router bgp 10
address-family vpnv4 unicast
!
neighbor BC-PE1
remote-as 10
update-source Loopback0
address-family vpnv4 unicast
CsC-CE2 (IOS)
! for connectivity to CsC-PE2 (eBGP+Label)
interface Ethernet1/0
description ** Link to CsC-PE2 **
ip address x.x.x.x
mpls bgp forwarding
!
router bgp 100
no bgp default ipv4-unicast
neighbor CsC-PE2 remote-as 10
!
address-family ipv4
neighbor CsC-PE2 activate
neighbor CsC-PE2 send-label
exit-address-family
!
! for connectivity to C-CE2 (ISIS+VRF)
vrf definition C-VPN
rd 100:Y
route-target 100:Y
!
address-family ipv4
exit-address-family
!
interface Ethernet1/3
description ** Link to C-CE2 **
vrf forwarding C-VPN
ip address y.y.y.y
ip router isis 200
!
router isis 200
vrf C-VPN
redistribute bgp 100
!
! for connectivity to CC-PE1 (iBGP VPNv4)
router bgp 100
neighbor CC-PE1 remote-as 100
neighbor CC-PE1 update-source Loopback0
!
address-family vpnv4
neighbor CC-PE1 activate
neighbor CC-PE1 send-community extended
exit-address-family
!
address-family ipv4 vrf C-VPN
redistribute isis 200
exit-address-family
IOS-XR configuration is similar to IOS, with the major difference of using the labeled unicast address-family instead of the send-label keyword.
Don't forget to create a /32 static route for the CsC-PE/CE next-hop in IOS-XR when using eBGP+Label. Always verify the installation of labels for /32 next-hops.
Verification
- Customer Carrier PEs must have a BGP VPNv4 route and a label for the VPN prefix
- Customer Carrier routers must have a label for the VPN prefix's next-hop
- CsC-PEs must have a BGP VPNv4 route and a label for the VPN prefix's next-hop
- Backbone Carrier routers must have a label for the next-hop of VPN prefix's next-hop
Example
Assume the following network:
R1-R2-R3-R4-R5-R6-R7-R8-R9-R10
where
Customer Carrier Network
Backbone Carrier Network
Then the following would happen for a VPN packet originating at R1 and terminating at R10.
- R1 (1.1.1.1) (Customer Carrier PE router) - vrf VPN
- Transport label is 18, VPN label is 20
- next-hop is R10 (10.10.10.10)
- R2 (2.2.2.2) (Customer Carrier P router)
- Transport label is 20, VPN label is 20
- next-hop is R10 (10.10.10.10)
- R3 (3.3.3.3) (CsC-CE)
- Transport label is 26, VPN label is 20
- next-hop is R10 (10.10.10.10)
- R4 (4.4.4.4) (CsC-PE) - vrf CSC
- Transport label is 16/21, VPN label is 20
- next-hop is R7 (7.7.7.7)
- R5 (5.5.5.5) Backbone Carrier P router)
- Transport label is 16/21, VPN label is 20
- next-hop is R7 (7.7.7.7)
- R6 (6.6.6.6) (Backbone Carrier P router)
- Transport label is 21, VPN label is 20
- next-hop is R7 (7.7.7.7)
- R7 (7.7.7.7) (CsC-PE) - vrf CSC
- Transport label is 18, VPN label is 20
- next-hop is R10 (10.10.10.10)
- R8 (8.8.8.8) (CsC-CE)
- Transport label is 17, VPN label is 20
- next-hop is R10 (10.10.10.10)
- R9 (9.9.9.9) (Customer Carrier P router)
- Transport label is removed, VPN label is 20
- next-hop is R10 (10.10.10.10)
- R10 (10.10.10.10) (Customer Carrier PE router) - vrf VPN
- VPN label is removed, destination reached in next-hop
R1#trace vrf VPN 99.99.99.99
Type escape sequence to abort.
Tracing the route to 99.99.99.99
1 20.1.2.2 [MPLS: Labels 18/20 Exp 0] 10 msec 10 msec 8 msec
2 20.2.3.3 [MPLS: Labels 20/20 Exp 0] 7 msec 7 msec 7 msec
3 20.3.4.4 [MPLS: Labels 26/20 Exp 0] 8 msec 8 msec 7 msec
4 20.4.5.5 [MPLS: Labels 16/21/20 Exp 0] 7 msec 7 msec 8 msec
5 20.5.6.6 [MPLS: Labels 16/21/20 Exp 0] 6 msec 5 msec 5 msec
6 20.6.7.7 [MPLS: Labels 21/20 Exp 0] 3 msec 3 msec 3 msec
7 20.7.8.8 [MPLS: Labels 18/20 Exp 0] 1 msec 1 msec 1 msec
8 20.8.9.9 [MPLS: Labels 17/20 Exp 0] 1 msec 1 msec 1 msec
9 20.9.10.10 [MPLS: Label 20 Exp 0] 1 msec 1 msec 1 msec
10 30.10.10.99 1 msec 2 msec 2 msec
Verification in every hop
Customer Carrier PE router
R1#sh ip route vrf VPN 99.99.99.99
Routing Table: VPN
Routing entry for 99.99.99.99/32
Known via "bgp 100", distance 200, metric 1, type internal
Redistributing via rip
Advertised by rip metric transparent
Last update from 10.10.10.10 00:56:37 ago
Routing Descriptor Blocks:
* 10.10.10.10 (default), from 10.10.10.10, 00:56:37 ago
Route metric is 1, traffic share count is 1
AS Hops 0
MPLS label: 20
MPLS Flags: MPLS Required
R1#sh bgp vpnv4 unicast vrf VPN 99.99.99.99/32
BGP routing table entry for 100:1:99.99.99.99/32, version 11
Paths: (1 available, best #1, table VPN)
Not advertised to any peer
Local
10.10.10.10 (metric 20) from 10.10.10.10 (10.10.10.10)
Origin incomplete, metric 1, localpref 100, valid, internal, best
Extended Community: RT:100:1
mpls labels in/out nolabel/20
VPN label is 20
R1#sh mpls forwarding-table 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
23 18 10.10.10.10/32 0 Fa0/0.12 20.1.2.2
MAC/Encaps=18/22, MRU=1500, Label Stack{18}
CA0113DC0000CA03079400088100026C8847 00012000
No output feature configured
R1#sh ip cef vrf VPN 99.99.99.99 det99.99.99.99/32, epoch 0, flags rib defined all labels
recursive via 10.10.10.10 label 20
nexthop 20.1.2.2 FastEthernet0/0.12 label 18
Transport label is 18, VPN label is 20
From here you have 2 options:
- follow the next-hops
- follow the labels (i prefer this one)
Following the next-hops
Customer Carrier P router
R2#sh mpls forwarding-table 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
18 20 10.10.10.10/32 13707 Fa0/0.23 20.2.3.3
MAC/Encaps=18/22, MRU=1500, Label Stack{20}
CA0610240000CA0113DC00008100002E8847 00014000
No output feature configured
Transport label is 20, VPN label is 20
CsC-CE
R3#sh mpls forwarding-table 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
20 26 10.10.10.10/32 15048 Fa0/0.34 20.3.4.4
MAC/Encaps=18/22, MRU=1500, Label Stack{26}
CA0207940008CA0610240000810001A38847 0001A000
No output feature configured
Transport label is 26, VPN label is 20
CsC-PE
R4#sh mpls forwarding-table vrf CSC 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
26 21 10.10.10.10/32[V]16033 Fa0/0.45 20.4.5.5
MAC/Encaps=18/26, MRU=1496, Label Stack{16 21}
C20911080000CA0207940008810003338847 0001000000015000
VPN route: CSC
No output feature configured
2 Transport labels are used (use "detail" to see them)
R4#sh bgp vpnv4 unicast vrf CSC 10.10.10.10
BGP routing table entry for 200:1:10.10.10.10/32, version 12
Paths: (1 available, best #1, table CSC)
Advertised to update-groups:
3
100
7.7.7.7 (metric 4) from 7.7.7.7 (7.7.7.7)
Origin incomplete, metric 20, localpref 100, valid, internal, best
Extended Community: RT:200:1
mpls labels in/out 26/21
VPN label (21) for Backbone Carrier is actually Transport label (21) for Customer Carrier
R4#sh mpls forwarding-table 7.7.7.7 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
16 16 7.7.7.7/32 0 Fa0/0.45 20.4.5.5
MAC/Encaps=18/22, MRU=1500, Label Stack{16}
C20911080000CA0207940008810003338847 00010000
No output feature configured
Transport label is 16/21, VPN label is 20
Backbone Carrier P router
R5#sh mpls forwarding-table 7.7.7.7 detail
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 16 7.7.7.7/32 44218 Fa0/0.56 20.5.6.6
MAC/Encaps=18/22, MRU=1500, Tag Stack{16}
C20811080000C209110800008100004E8847 00010000
No output feature configured
Per-packet load-sharing
Transport label is 16/21, VPN label is 20
Backbone Carrier P router
R6#sh mpls forwarding-table 7.7.7.7 detail
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 Pop tag 7.7.7.7/32 42398 Fa0/0.67 20.6.7.7
MAC/Encaps=18/18, MRU=1504, Tag Stack{}
CA0415180000C20811080000810000118847
No output feature configured
Per-packet load-sharing
Transport label is 21, VPN label is 20
CsC-PE
R7#sh mpls forwarding-table vrf CSC 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
21 18 10.10.10.10/32[V]24156 Fa0/0.78 20.7.8.8
MAC/Encaps=18/22, MRU=1500, Label Stack{18}
CA0710240000CA04151800008100000D8847 00012000
VPN route: CSC
No output feature configured
Transport label is 18, VPN label is 20
CsC-CE
R8#sh mpls forwarding-table 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
18 17 10.10.10.10/32 24703 Fa0/0.89 20.8.9.9
MAC/Encaps=18/22, MRU=1500, Label Stack{17}
CA0013DC0000CA0710240000810000238847 00011000
No output feature configured
Transport label is 17, VPN label is 20
Customer Carrier P router
R9#sh mpls forwarding-table 10.10.10.10 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
17 Pop Label 10.10.10.10/32 23892 Fa0/0.910 20.9.10.10
MAC/Encaps=18/18, MRU=1504, Label Stack{}
CA0515180000CA0013DC0000810000198847
No output feature configured
Transport label is removed, VPN label is 20
Customer Carrier PE router
R10#sh mpls forwarding-table vrf VPN 99.99.99.99 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
20 No Label 99.99.99.99/32[V]1770 Fa0/0.1010 30.10.10.99
MAC/Encaps=18/18, MRU=1504, Label Stack{}
C20A0F840000CA05151800008100001D0800
VPN route: VPN
No output feature configured
VPN label is removed, destination reached
Following the labels
Customer Carrier P router
R2#sh mpls forwarding-table labels 18 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
18 20 10.10.10.10/32 13852 Fa0/0.23 20.4.6.4
MAC/Encaps=18/22, MRU=1500, Label Stack{20}
CA0610240000CA0113DC00008100002E8847 00014000
No output feature configured
Transport label is 20, VPN label is 20
CsC-CE
R3#sh mpls forwarding-table labels 20 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
20 26 10.10.10.10/32 15338 Fa0/0.34 20.4.19.19
MAC/Encaps=18/22, MRU=1500, Label Stack{26}
CA0207940008CA0610240000810001A38847 0001A000
No output feature configured
Transport label is 26, VPN label is 20
CsC-PE
R4#sh mpls forwarding-table labels 26 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
26 21 10.10.10.10/32[V]16645 Fa0/0.45 20.4.5.5
MAC/Encaps=18/26, MRU=1496, Label Stack{16 21}
C20911080000CA0207940008810003338847 0001000000015000
VPN route: CSC
No output feature configured
2 Transport labels are used (use "detail" to see them)
Transport label is 16/21, VPN label is 20
Backbone Carrier P router
R5#sh mpls forwarding-table labels 16 detail
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 16 7.7.7.7/32 30421 Fa0/0.56 20.5.6.6
MAC/Encaps=18/22, MRU=1500, Tag Stack{16}
C20811080000C209110800008100004E8847 00010000
No output feature configured
Per-packet load-sharing
Transport label is 16/21, VPN label is 20
Backbone Carrier P router
R6#sh mpls forwarding-table labels 16 detail
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 Pop tag 7.7.7.7/32 29337 Fa0/0.67 20.6.7.7
MAC/Encaps=18/18, MRU=1504, Tag Stack{}
CA0415180000C20811080000810000118847
No output feature configured
Per-packet load-sharing
Transport label is 21, VPN label is 20
CsC-PE
R7#sh mpls forwarding-table labels 21 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
21 18 10.10.10.10/32[V]17260 Fa0/0.78 20.7.8.8
MAC/Encaps=18/22, MRU=1500, Label Stack{18}
CA0710240000CA04151800008100000D8847 00012000
VPN route: CSC
No output feature configured
Transport label is 18, VPN label is 20
CsC-CE
R8#sh mpls forwarding-table labels 18 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
18 17 10.10.10.10/32 17726 Fa0/0.89 20.8.9.9
MAC/Encaps=18/22, MRU=1500, Label Stack{17}
CA0013DC0000CA0710240000810000238847 00011000
No output feature configured
Transport label is 17, VPN label is 20
Customer Carrier P router
R9#sh mpls forwarding-table labels 17 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
17 Pop Label 10.10.10.10/32 17200 Fa0/0.910 20.9.10.10
MAC/Encaps=18/18, MRU=1504, Label Stack{}
CA0515180000CA0013DC0000810000198847
No output feature configured
Transport label is removed, VPN label is 20
Customer Carrier PE router
R10#sh mpls forwarding-table labels 20 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
20 No Label 99.99.99.99/32[V]1770 Fa0/0.1010 30.10.10.99
MAC/Encaps=18/18, MRU=1504, Label Stack{}
C20A0F840000CA05151800008100001D0800
VPN route: VPN
No output feature configured
VPN label is removed, destination reached
