BGP for Cloud Engineers: Route Selection, RPKI, Direct Connect, ExpressRoute and Hybrid Routing (2026)

Most cloud engineers meet BGP the same way: a Direct Connect or ExpressRoute circuit is delivered, a provider hands over two /30 subnets and an ASN, and the only instruction is “bring up the session.” The session comes up, routes appear, and everything works until the day a failover takes four minutes instead of four seconds, or a typo in a prefix list sends production traffic out the wrong door. Border Gateway Protocol is not exotic, but it is a policy protocol, not a shortest-path protocol, and that single fact explains most hybrid-network outages.

This matters more in 2026 than it did five years ago. Hybrid estates now carry databases, identity traffic and AI data pipelines over private circuits, while the public internet’s routing security (RPKI, route-leak prevention) is finally moving from optional to expected. This article gives you the working model: how best-path selection really decides, how the three big clouds use BGP, how to do the failover arithmetic, and how to secure the edge.

What this covers: BGP fundamentals, the best-path algorithm with a worked example, communities and prepending, the documented behaviour of Direct Connect, ExpressRoute and Cloud Interconnect, convergence math with BFD, RPKI and route-leak defences, a runnable FRR configuration, and the failure modes that actually bite.

Context and Background

BGP version 4 is defined in RFC 4271. Autonomous systems (AS), each identified by an AS number, exchange reachability information for IP prefixes over TCP port 179. A session between two different ASes is external BGP (eBGP); a session inside one AS is internal BGP (iBGP). The distinction matters because the two behave differently: eBGP prepends the local AS number to the AS path and normally changes the next hop, while iBGP does neither and, by default, does not re-advertise routes learned from one iBGP peer to another. That last rule is why large networks use route reflectors or full meshes.

Four-byte AS numbers (RFC 6793) have been universal for years, and private ranges are carved out in RFC 6996: 64512 to 65534 for two-byte private ASNs and 4200000000 to 4294967294 for four-byte ones. Cloud connectivity leans heavily on private ASNs on the customer side, which is a quiet source of trouble we return to below.

BGP carries path attributes with each prefix. The well-known mandatory ones are ORIGIN, AS_PATH and NEXT_HOP. Optional but ubiquitous are LOCAL_PREF (iBGP only), MULTI_EXIT_DISC (MED), and COMMUNITIES (RFC 1997). Operators attach policy to these attributes. Nothing in the protocol tells a router which path is “best” in a physical sense; the operator’s configuration does.

The incumbents in cloud hybrid networking are AWS Direct Connect, Azure ExpressRoute and Google Cloud Interconnect. All three terminate your circuit on a provider-managed edge and speak BGP to you. All three also accept the same fallback: an IPsec VPN over the internet running BGP as well, which is why a clean mental model of BGP pays off across clouds. If you are designing the AWS side, the AWS VPC networking guide covering Transit Gateway, PrivateLink and hub-and-spoke design is the natural companion to this article.

Why do outages still happen? Because BGP defaults are permissive and the protocol trusts what it is told. RFC 8212 (2017) made the safe default explicit: an eBGP session with no configured import and export policy should accept and send nothing. Many implementations now comply. Legacy configurations and some appliances still do not, so “I brought the session up and routes flow” is not evidence of a safe configuration.

How BGP Decides: The Best-Path Reference Model

BGP chooses one best path per prefix from all candidate paths learned from all neighbours, using an ordered tie-break list. The decisive attributes, in order, are highest LOCAL_PREF, shortest AS_PATH, lowest MED, then eBGP over iBGP and lowest IGP cost. Crucially, the forwarding table (FIB) applies longest-prefix match before BGP’s best-path logic ever matters, so a more specific route always wins regardless of preference.

Figure 1: The BGP best-path decision sequence. Longest prefix match is applied in forwarding, before any attribute comparison; the remaining steps are an ordered tie-break.

Figure 1 shows the sequence a router walks. Vendors differ in small details (Cisco’s proprietary WEIGHT comes before everything, FRR and BIRD have their own optional steps), but the core order from the RFC 4271 decision process and common implementations is stable. The next three subsections explain why each of the first steps behaves the way it does in a cloud-hybrid design.

Longest prefix match beats everything

Suppose your data centre advertises 10.20.0.0/16 over a primary circuit and 10.20.4.0/24 over a backup. The /24 wins for traffic destined into that /24 even though the circuit is meant to be the backup, because forwarding looks up the most specific entry. This is not a BGP rule at all; it is how every IP forwarding plane works, and it is the most common cause of “failover traffic is on the wrong link.”

AWS makes this explicit. Its Direct Connect documentation lists longest prefix length as the first selection criterion for private and transit virtual interfaces, and recommends advertising more specific routes over multiple virtual interfaces for active/passive designs (AWS Direct Connect routing policies). The corollary is useful: to steer inbound traffic toward a preferred circuit when you cannot rely on communities, split a prefix into halves and advertise the halves on the preferred path while keeping the covering aggregate on both.

LOCAL_PREF is your steering wheel for outbound traffic

LOCAL_PREF is a number (higher wins) that travels only inside one AS. It is the first attribute operators reach for because it overrides everything below it, including a shorter AS path. If your router has two paths to the cloud and you set LOCAL_PREF 200 on the primary and 100 on the backup, the backup is used only when the primary disappears.

The catch for cloud engineers: you only control LOCAL_PREF inside your own AS. To influence what the cloud does with your routes you need something the cloud honours, which is why Direct Connect defines local-preference communities. AWS documents 7224:7100 (low), 7224:7200 (medium) and 7224:7300 (high) for private and transit virtual interfaces; they are mutually exclusive and are evaluated before AS_PATH, and when none is applied AWS defaults to medium for locations associated with the same Region (AWS documentation). Azure goes the other way: Microsoft states it does not honour community values you set on routes you advertise to it, so you steer with AS-path and prefix length instead.

AS_PATH length and prepending

Each eBGP hop prepends its AS number, so shorter paths are “closer.” Operators exploit this by prepending their own ASN several times on the path they want to be less attractive. Prepending is blunt: it is a hint, not an instruction, and it only helps if the receiver has not already decided on an earlier criterion such as LOCAL_PREF.

There is a cloud-specific trap. On a Direct Connect public virtual interface, if you use a private ASN, AWS replaces it with 7224 when re-advertising and your prepending is stripped, so it cannot influence routing outside AWS; with a public ASN, prepending works as normal (AWS documentation). Similarly, Microsoft removes private AS numbers from the AS path of prefixes received on Microsoft peering, so you cannot use them to influence routing there (Microsoft Learn: ExpressRoute routing requirements).

MED and the tie-breakers

MED (multi-exit discriminator) is a hint one AS gives a neighbouring AS about which of several entry points it prefers; lower is better. By default routers compare MED only between paths from the same neighbouring AS, which makes it unreliable across providers, and AWS states it ranks MED lowest and does not recommend relying on it. After MED come eBGP over iBGP, lowest IGP metric to the next hop, and finally stable tie-breakers such as route age and router ID. If you see traffic flipping between two equal paths after a flap, the tie-breaker (oldest route wins in many implementations) is the reason.

A worked best-path example

Take a router in your data centre (AS 65010) with three candidate paths to 172.31.0.0/16, a VPC range reachable via the cloud:

Path	Learned from	LOCAL_PREF	AS_PATH length	MED
A	Direct Connect A	200	2	0
B	Direct Connect B	200	1	50
C	Internet VPN	100	1	0

Path C loses immediately on LOCAL_PREF (100 < 200). A and B tie at 200, so AS path length decides: B has length 1, A has length 2, so B wins despite its higher MED, because MED is only reached if AS path lengths tie. This is the entire intuition for debugging: walk the list top to bottom and stop at the first difference. Most “why is it choosing that path” questions are answered in under a minute this way. If you want ECMP across A and B, you must make them equal on every step through to the IGP cost and enable multipath, which in FRR means maximum-paths and, when AS paths differ by content but not length, bgp bestpath as-path multipath-relax.

Communities, Prepending and Policy Design in Practice

A BGP community is a tag attached to a route so that neighbours can apply policy without knowing your internal topology. Standard communities (RFC 1997) are 32-bit values written ASN:value; large communities (RFC 8092) extend this to three 32-bit fields so that four-byte ASNs can be expressed unambiguously. Well-known communities such as NO_EXPORT and NO_ADVERTISE limit how far a route spreads.

The practical rule for hybrid networks: use communities to request behaviour from a provider, and use local policy to enforce your own. Providers publish what they honour; anything else is decoration. Treat the provider’s community table as part of the interface contract and version-control it with your router configuration.

What each cloud lets you control

The three providers have different philosophies, summarised below from their current documentation. Where a value was not stated in the pages consulted, that is marked rather than guessed.

Capability	AWS Direct Connect	Azure ExpressRoute	Google Cloud Interconnect
Provider-side ASN	7224 documented for public VIFs; private VIFs use the Amazon-side ASN you configure	12076 for private and Microsoft peering	Cloud Router ASN is configurable (check your attachment)
Inbound steering to cloud	Local-preference communities 7224:7100/7200/7300	Community values you set are not honoured; use prefix length and AS path	Route priority on the attachment, carried as MED (verify in current docs)
Prefix limit, you to cloud	100 IPv4 and 100 IPv6 per private or transit VIF by default, up to 1,000 each; 1,000 on public VIFs (fixed)	4,000 IPv4 and 100 IPv6 private peering; 10,000 IPv4 with premium add-on; 200 on Microsoft peering	Verify in current quotas
Cloud to you	Transit Gateway: 200 combined IPv4 and IPv6 prefixes by default	Regional communities such as 12076:50004 (East US, private)	Cloud Router advertises per configured mode
If limit exceeded	Session goes idle and shows DOWN	Session drops	Not confirmed this run

The numbers for AWS come from the Direct Connect quotas page and the routing page cited above, and the ExpressRoute numbers from Microsoft’s routing requirements. Quotas change; confirm them in Service Quotas before you design to a number.

The prefix-limit failure that looks like an outage

Both AWS and Azure enforce a hard prefix limit by dropping the BGP session, not by ignoring the excess. That is a deliberate protection, but it means a leak from your own side, for instance redistributing a full internal route table into BGP, turns a routing mistake into a total loss of connectivity on that virtual interface. A default private VIF accepts only 100 IPv4 prefixes from you. A data centre with 130 internal /24s that is advertised unsummarised will bring the session down the moment the 101st prefix lands.

The defence is summarisation and an explicit outbound prefix list. Advertise aggregates that your policy permits, filter everything else, and alarm when the count of advertised prefixes crosses, say, 70% of the limit. On AWS, prefix controls can raise the private and transit VIF allowance to 1,000 per address family; on ExpressRoute, the premium add-on lifts the IPv4 private peering limit to 10,000.

Cloud Edge Architecture: How Direct Connect, ExpressRoute and Cloud Interconnect Use BGP

For hybrid connectivity, the direct answer is: each cloud terminates a dedicated or partner circuit on a provider edge router that runs eBGP with your customer router, exchanges only the prefixes you and the cloud permit, and relies on a second diverse circuit plus BFD for fast failover. The design choice is not whether to use BGP but how to arrange diversity, preference and fall-back paths around it.

Figure 2: A resilient hybrid layout. Two circuits terminate in separate Direct Connect locations and attach through a Direct Connect gateway to a Transit Gateway; a BGP-over-VPN path acts as the last resort.

Figure 2 shows the AWS flavour of a pattern that translates across providers. Two circuits land in different physical facilities (not merely different ports on the same device), terminate on separate customer routers, and meet the cloud network at a gateway construct. An internet VPN, also running BGP, sits at lower preference as the last resort. The same shape works for ExpressRoute (two circuits in different peering locations into a virtual network gateway or Virtual WAN) and for Google (two Dedicated or Partner Interconnect attachments in different edge availability domains into a Cloud Router).

Active/active versus active/passive

With active/active you give both circuits equal preference and let ECMP spread flows. AWS’s guidance is to apply the same local-preference community to all connections, so a failure shifts traffic with ECMP across the remainder. With active/passive you mark the primary high (7224:7300) and the backup low (7224:7100). Active/active uses the capacity you pay for, but it hides a trap: if each circuit runs at 70% utilisation, losing one pushes the other to 140% and you have an outage by congestion rather than by routing. Size active/active so that either circuit alone can carry peak load, or treat it as active/passive honestly.

Azure adds a rule worth memorising. Microsoft relies on a redundant pair of BGP sessions per peering, requires you to advertise identical routes across multiple circuits you own (metrics need not match), and warns that you cannot make ExpressRoute a transit router between your own sites. If both of your circuits are connected to the same on-premises WAN and you advertise the same prefixes on each, the cloud may route site-to-site traffic through your own networks in ways you did not intend.

Direct Connect gateways, Transit Gateway and what gets advertised

The AWS side adds layers. A Direct Connect gateway is a global construct that can associate with up to 20 virtual private gateways or up to 6 Transit Gateways, and it holds up to 30 private or transit VIFs. When you use a transit VIF, AWS advertises to you only the prefixes you allow on the Transit Gateway association, capped at 200 by default. If your cloud estate has 300 VPC CIDRs, summarise them at the association or request a higher limit; otherwise prefixes will simply be missing from your side’s routing table, which presents as “some VPCs are unreachable from on-premises.”

Another documented rule: transitive routing between connections is not supported on public virtual interfaces. Do not assume AWS will carry traffic from one of your sites to another through a Direct Connect gateway. If you need site-to-site over the cloud, that is a Cloud WAN or Transit Gateway design with explicit configuration, not a side-effect of BGP. For a deeper look at how routes propagate inside the VPC layer, see the Transit Gateway and hub-and-spoke article.

Default routes and the Azure licensing surprise

ExpressRoute accepts a default route (0.0.0.0/0) only on private peering. When you advertise one, all traffic from the associated virtual networks heads to your network, including traffic that previously reached the internet directly. Microsoft documents a specific side effect: advertising a default route breaks Windows and VM licence activation (KMS), and the documented workaround is user-defined routes on the subnets that need direct internet access. Plan egress deliberately before you advertise a default route; forced tunnelling is a design decision, not a toggle.

IPv6 and dual stack

Both IPv4 and IPv6 are supported on the major clouds, but limits differ per address family and you must keep policy parity. ExpressRoute private peering allows 4,000 IPv4 but only 100 IPv6 prefixes from you. Google notes that IPv6 routes can be exchanged either over a BGP session carried on IPv6 or over IPv4 using multiprotocol BGP, and that dual-stack attachments can run separate IPv4 and IPv6 sessions at once. A frequent mistake is building meticulous IPv4 filters and leaving IPv6 with permissive defaults, which RFC 8212 behaviour at least partly guards against.

Failover Design and Convergence Arithmetic

Failover time is the sum of detection, propagation, route computation and forwarding-table programming. Detection dominates, and it is the one you control. Without BFD, a silent failure is found only when the BGP hold timer expires; with BFD, detection is bounded by interval times multiplier.

Figure 3: BFD detection versus BGP hold-timer detection for a silent link failure. BFD declares the neighbour down after a small number of missed packets, then BGP withdraws the routes and traffic moves to the backup.

The arithmetic

BGP itself uses KEEPALIVE messages and a hold time: if no message arrives within the hold time, the session is torn down. RFC 4271 suggests a 90 second hold time and one-third of that for keepalives, though implementations and providers differ. Microsoft states that the ExpressRoute edge (MSEE) typically uses 60 second keepalives and a 180 second hold time, so unaided detection can take up to three minutes; you may lower your side to a keepalive as low as 3 seconds and a hold time as low as 10, but Microsoft advises against aggressive BGP timers because BGP is process-intensive, and the lower value of the two peers wins (Microsoft Learn: Configure BFD).

BFD (RFC 5880) is a lightweight hello protocol designed to detect forwarding-path failure quickly. Detection time equals the negotiated transmit interval multiplied by the detect multiplier. On ExpressRoute the MSEE intervals are 300 ms (750 ms in certain scenarios), and a typical customer configuration uses a multiplier of 3, so detection is about 900 ms at 300 ms intervals. Microsoft adds that overall convergence between ExpressRoute virtual network gateways and the MSEEs can still take up to a minute after detection. Google Cloud Router’s BFD defaults are 1,000 ms transmit, 1,000 ms receive and a multiplier of 5, roughly 5 seconds, against about 60 seconds for BGP-only detection, and BFD there is off until enabled and works only on Dedicated or Partner Interconnect VLAN attachments with Dataplane version 2, not on HA VPN (Google Cloud BFD documentation).

For AWS, BFD must be enabled on the customer side to benefit; I could not re-verify the exact AWS default intervals from the live documentation during this run, so check the Direct Connect User Guide before quoting a number in a design review.

A rough end-to-end estimate for a link failure on Google Interconnect with default BFD: about 5 seconds of detection, plus a BGP withdrawal and re-selection that is typically sub-second inside a small network, plus FIB programming. Call it 5 to 8 seconds as an illustrative figure. Without BFD, the same event is closer to a minute. On a BGP-only ExpressRoute circuit with default timers, expect up to three minutes. These are planning figures; test on your own gear.

Do not enable aggressive timers blindly

BFD on a congested or CPU-starved router can flap, and a flap tears down the BGP session and reroutes traffic, which can be worse than the failure it was meant to detect. Practical guidance: start with the provider’s recommended values (300 ms by 3 on ExpressRoute, the defaults elsewhere), disable BFD echo mode where the provider says so (Microsoft recommends disabling it for ExpressRoute), and enable BGP graceful restart (RFC 4724) only after you understand its interaction with BFD, since the two can work against each other.

Also decide on the failure you are actually designing for. A BFD-detected link failure is different from a provider-side maintenance event. Planned maintenance should use graceful shutdown (RFC 8326), which signals neighbours to deprioritise routes before the session closes so traffic drains without loss.

Multi-region and DNS interplay

Routing converges in seconds, but applications converge on their own clocks: connection pools, DNS TTLs, database leaders. A perfect BGP failover into a region where the database primary has not moved is still an outage. Tie network failover to application failover deliberately. The DNS anycast and GSLB failover article and the multi-region active-active database architecture guide cover the layers above the network.

Securing the Edge: RPKI, Route Leaks, Hijacks and ASPA

BGP has no built-in way to check that an AS is allowed to originate a prefix, so anyone can announce anything and their neighbours may believe it. A hijack is a false origin or a more specific announcement of someone else’s space; a leak is a legitimate route propagated beyond its intended scope, such as a customer re-announcing one provider’s routes to another provider. Both make traffic flow where it should not, and both have been responsible for well-known multi-hour outages.

Figure 4: Route origin validation. A validator downloads signed ROAs from the five regional registries and feeds them to the router over the RPKI-to-Router protocol; each update is classified and handled by policy.

RPKI and route origin validation

The Resource Public Key Infrastructure ties address blocks to the holders of those blocks cryptographically. A holder issues a Route Origin Authorization (ROA) stating which AS may originate a prefix and up to what maximum length. Route origin validation (ROV), defined in RFC 6811, compares each received announcement to the validated ROAs and produces one of three states:

Valid: a covering ROA matches both the origin AS and the prefix length bound.
Invalid: a covering ROA exists, but the origin AS differs or the prefix is longer than the maximum length.
NotFound: no ROA covers the prefix.

The ROA object format is profiled in RFC 9582, and validators such as Routinator, rpki-client or FORT download and verify the signed objects from the five regional registries’ trust anchors. They feed the resulting validated payloads to routers over the RPKI-to-Router protocol (RFC 8210). The recommended policy is to reject Invalid routes and accept Valid and NotFound ones, because too many legitimate prefixes still lack ROAs for NotFound to be rejected.

I am deliberately not quoting a global adoption percentage here. The measurements I could confirm are old (a Cloudflare analysis published in December 2022 reported ROAs covering roughly 40% of routed prefix-origin pairs, with far lower enforcement coverage), and adoption has moved since. Check a live source such as Cloudflare’s RPKI portal before citing a figure.

What this means for a cloud customer

Your cloud provider’s border and your upstream carriers are the ones enforcing ROV against the public internet, and your job is to be a good origin. If you advertise your own public space to a cloud over a public VIF or Microsoft peering, create ROAs for it with the correct maximum length, keep them current when you re-home a prefix, and register the prefixes in a routing registry. AWS notes that for public virtual interfaces you must own the prefixes and register them with the regional registry, and Microsoft requires registration with an RIR or routing registry and may ask for a Letter of Authorization. A prefix without a matching ROA will usually still work today; one with a wrong ROA will be dropped by anyone doing ROV.

Two mistakes recur. Setting maximum length too long (a /16 ROA with maxLength 24) lets an attacker announce a forged /24 that remains “Valid,” so keep maxLength tight and issue separate ROAs for the specific lengths you actually announce. And forgetting to update the ROA before moving a prefix to a new origin AS makes your own migration look like a hijack and gets it filtered.

Route leaks and the Only-to-Customer attribute

ROV does nothing about leaks, because a leaked route carries the legitimate origin. RFC 9234 (May 2022) addresses this with BGP Roles and a new transitive attribute. Each eBGP session negotiates a role in the OPEN message: Provider, Customer, Route Server, RS-Client or Peer. Compatible roles must match (Provider with Customer, Peer with Peer) or the session is rejected with a Role Mismatch notification. A router then marks routes received from a Provider, Peer or Route Server with the Only to Customer (OTC) attribute, type code 35 containing an AS number, and refuses to forward OTC-marked routes to providers or peers. A leak is detected and stopped at the first hop that implements it.

For a cloud customer the immediate action is not a protocol rollout but old-fashioned hygiene described in RFC 7454 (BCP 194): outbound prefix lists that allow only what you own, maximum-prefix limits on every session, no transit between providers unless you are one, and filtering of bogons and over-specific routes. If you have two clouds or two carriers attached to the same router, the single most valuable policy is “routes learned from provider A are never exported to provider B.” That one line prevents you from accidentally becoming a transit network.

ASPA: where things stand

AS Provider Authorization (ASPA) extends RPKI so that an AS can publish which ASes are its authorised upstream providers, letting receivers check that an AS_PATH follows plausible customer-to-provider relationships. It targets leaks and forged-path attacks that ROV cannot see. As of my check on 2026-10-23, the verification procedure is still an Internet-Draft: draft-ietf-sidrops-aspa-verification-29, last updated 2026-10-16, marked as intended for Proposed Standard, with the working group state “WG Consensus: Waiting for Write-Up” and no RFC number assigned (IETF Datatracker). That means you can experiment where your router software supports it, but you should not design a hard dependency on it yet. Treat ASPA as an addition to, not a replacement for, ROV, RFC 9234 roles and prefix hygiene.

BGPsec: why it is rarely deployed

BGPsec (RFC 8205) signs the full path hop by hop and offers stronger guarantees, but it requires every AS on the path to participate, cannot be deployed incrementally with much benefit, and increases message size and processing. Most operators have chosen ROV plus operational filters plus, eventually, ASPA. That is the pragmatic route to adopt.

A Runnable Configuration: FRR as a Hybrid Edge

The configuration below uses FRR (Free Range Routing), an open-source routing suite that is widely used for lab work and cloud virtual appliances. It builds the design in Figures 1 to 4: an on-premises edge in AS 65010 with a primary and a backup session, explicit inbound and outbound policy, a prefix limit, BFD, ROV via an RTR cache, and a tag for the provider’s local-preference community. Addresses use documentation-style ranges and are illustrative; adapt them to your circuit.

! /etc/frr/frr.conf  (FRR 8.x or later)
frr defaults datacenter
hostname edge1
!
! Prefixes we are allowed to originate
ip prefix-list OUT-ALLOWED seq 10 permit 10.20.0.0/16
ip prefix-list OUT-ALLOWED seq 20 permit 10.21.0.0/16
!
! Never accept default or our own space back from the cloud
ip prefix-list IN-DENY seq 10 permit 0.0.0.0/0
ip prefix-list IN-DENY seq 20 permit 10.20.0.0/14 le 32
!
route-map CLOUD-OUT-PRIMARY permit 10
 match ip address prefix-list OUT-ALLOWED
 set community 7224:7300
route-map CLOUD-OUT-BACKUP permit 10
 match ip address prefix-list OUT-ALLOWED
 set community 7224:7100
 set as-path prepend 65010 65010
!
route-map CLOUD-IN deny 10
 match ip address prefix-list IN-DENY
route-map CLOUD-IN permit 20
 set local-preference 200
!
route-map ROV-IN deny 5
 match rpki invalid
route-map ROV-IN permit 10
!
rpki
 rpki polling_period 300
 rpki cache 192.0.2.50 3323 preference 1
exit
!
bfd
 peer 169.254.100.1
  receive-interval 300
  transmit-interval 300
  detect-multiplier 3
  no shutdown
 exit
exit
!
router bgp 65010
 bgp router-id 10.20.0.1
 no bgp ebgp-requires-policy
 bgp log-neighbor-changes
 neighbor PRIMARY peer-group
 neighbor PRIMARY remote-as 64512
 neighbor PRIMARY password CHANGE-ME
 neighbor PRIMARY bfd
 neighbor PRIMARY timers 10 30
 neighbor 169.254.100.1 peer-group PRIMARY
 neighbor BACKUP peer-group
 neighbor BACKUP remote-as 64512
 neighbor BACKUP bfd
 neighbor 169.254.101.1 peer-group BACKUP
 !
 address-family ipv4 unicast
  network 10.20.0.0/16
  network 10.21.0.0/16
  neighbor PRIMARY soft-reconfiguration inbound
  neighbor PRIMARY maximum-prefix 90 80
  neighbor PRIMARY route-map CLOUD-IN in
  neighbor PRIMARY route-map CLOUD-OUT-PRIMARY out
  neighbor BACKUP soft-reconfiguration inbound
  neighbor BACKUP maximum-prefix 90 80
  neighbor BACKUP route-map CLOUD-IN in
  neighbor BACKUP route-map CLOUD-OUT-BACKUP out
 exit-address-family
exit

Several lines deserve explanation. The no bgp ebgp-requires-policy line is present only so the file loads on lab builds; in production you want the default behaviour that mirrors RFC 8212, and you should delete that line once every neighbour has route-maps in both directions. The maximum-prefix 90 80 form tears down the session at 90 learned prefixes and warns at 80 percent; set it below the provider’s limit so that you drop the session on your own terms. The communities 7224:7300 and 7224:7100 follow the Direct Connect pattern described earlier and are only meaningful to AWS; if you adapt the file for another provider, replace or remove them.

The route-map ROV-IN is shown separately to demonstrate FRR’s match rpki clause. In real deployments you apply origin validation where you peer with the public internet, usually at an internet edge router, not on a private cloud circuit whose routes are controlled by contract. Combine it with the CLOUD-IN policy there by chaining the checks in one route-map. Validate the configuration by running show bgp summary, show bgp ipv4 unicast neighbors 169.254.100.1 advertised-routes, show bfd peers and show rpki prefix-table, and then pull a cable in a lab to measure what your failover actually costs.

What to measure after you apply it

Record four numbers before declaring the design done: time from link-down to BFD-down, time from BFD-down to the last route withdrawn from the FIB, time from FIB update to the first successful application request on the backup path, and recovery time when the primary returns. The last one matters because many designs fail back too eagerly: when a flapping primary returns, preemption can bounce traffic repeatedly. Use a hold-down timer or manual fail-back for circuits that flap.

Trade-offs, Gotchas, and What Goes Wrong

The recurring incidents in hybrid BGP are boring, which is exactly why they hurt. Each below has a mechanism, a symptom and a fix.

Asymmetric routing through stateful devices. Microsoft states symmetry is not required, so your forward and return paths may use different router pairs. If a stateful firewall sits behind only one of your circuits, return traffic arrives at a device that never saw the flow and drops it. Either make the policy symmetric with LOCAL_PREF and matching prepends, or place state on the shared side of the design.

Overlapping public routes. Advertising the same public prefix to the internet and over a cloud circuit creates asymmetry by construction. Microsoft explicitly warns against it, and AWS recommends that if you advertise the same prefix both ways, the more specific route should go over Direct Connect. Decide per prefix which path is authoritative.

Private ASN surprises. Private ASNs are fine on private virtual interfaces and many private peerings, but on public-facing attachments they are stripped or replaced (AWS substitutes 7224; Microsoft removes them from the AS path). Anything that relies on seeing your ASN, including prepending or ROA matching, then silently fails. Use a public ASN where outside parties must recognise you.

Prefix-limit tripwires. Covered earlier, but worth repeating as a failure mode: the limit is enforced by killing the session. Pair outbound summarisation with an alarm on advertised-prefix count and a runbook for raising the allowance, noting that some increases need a request through your account team.

Convergence hidden in the cloud control plane. Network failover can complete while the cloud’s own virtual-network route programming is still catching up; Microsoft notes up to a minute for gateway-to-edge failover even with BFD. Do not promise a sub-second recovery objective that the weakest layer cannot meet.

Active/active oversubscription. Already described: two circuits at 70% each means one failure pushes the survivor past capacity. Congestion drops packets without ever touching BGP, so no alarm fires in the routing system. Monitor circuit utilisation, not just session state.

Equal-cost multipath with unequal paths. ECMP hashes flows, so a long-lived elephant flow can saturate one path while the other is idle. Check hash fields and use more VIFs or LAGs if a single flow must exceed one circuit’s capacity.

Security tools that stop at the border. ROV protects you from some hijacks of the internet, but it does nothing for traffic on a private circuit with a provider you already trust, and nothing for leaks. Do not treat one control as a complete answer. Zero-trust identity, such as the approach in SPIFFE and SPIRE workload identity, is the right layer for authenticating workloads whatever path their packets took; the network cannot be the only authority.

Finally, a trade-off in philosophy: more automation means faster mistakes. Pushing router configuration from a pipeline is wise, but add a pre-deployment check that parses the prospective advertised-prefix set and refuses to deploy a change that would exceed the provider limit or advertise a default route. Test in a virtual lab first; FRR in containers is sufficient.

Practical Recommendations

Treat BGP as a contract between you and each provider and design for the second failure, not the first. Keep policy simple enough that a colleague can walk Figure 1 at 3 a.m., and keep one source of truth for communities, prefix lists and limits. Prefer boring and explicit over clever.

Start with diversity: two circuits in physically different facilities on separate customer routers, plus an internet VPN running BGP as a floor. Decide active/active or active/passive in writing and size capacity for the loss of one circuit. Then lock down policy: explicit import and export filters on every session, a maximum-prefix limit set below the provider’s, and no export of one provider’s routes to another.

Checklist:

Two diverse circuits plus a BGP-over-VPN fallback, each with a documented preference.
Explicit inbound and outbound prefix lists on every eBGP session (the RFC 8212 default on, not off).
Summarise to stay under provider limits (AWS private VIF default 100 IPv4 prefixes; ExpressRoute private peering 4,000) and alarm at 70% of the limit.
BFD enabled on your side wherever the provider supports it; verify the actual interval and multiplier from the provider’s current documentation.
Provider communities documented and versioned alongside router config; do not rely on communities Azure ignores.
ROAs published for every public prefix you originate, with tight maxLength, updated before any origin change.
ROV rejecting Invalid at the internet edge; track RFC 9234 roles and ASPA support in your router software.
Capacity planning that survives the loss of one circuit; alarms on utilisation as well as session state.
Failover test in a lab and in production on a schedule; record detection, withdrawal and application recovery times.
Graceful shutdown before planned maintenance, and a hold-down for fail-back.

Frequently Asked Questions

What is BGP and why do cloud providers use it for hybrid connectivity?

BGP is the path-vector protocol that exchanges IP reachability between autonomous systems, defined in RFC 4271. Cloud providers use it because it lets two independent organisations exchange routes under explicit policy without sharing an internal routing protocol. It also supports redundancy and dynamic failover: when a circuit dies, withdrawn routes shift traffic to the surviving path without manual changes. Direct Connect, ExpressRoute and Cloud Interconnect all rely on it.

How does BGP choose the best path?

Longest prefix match is applied during forwarding. Among equal-length candidates, BGP prefers the highest LOCAL_PREF, then the shortest AS_PATH, then lowest origin type and MED (compared only between paths from the same neighbouring AS), then eBGP over iBGP, lowest IGP cost to the next hop, and finally stable tie-breakers such as route age and router ID. Vendors add proprietary steps, such as Cisco WEIGHT, so check your platform.

How fast does BGP fail over on Direct Connect or ExpressRoute?

Without BFD, failure detection waits for the BGP hold timer; Microsoft says the ExpressRoute edge typically uses a 60 second keepalive and 180 second hold, so up to three minutes. With BFD at 300 ms and a multiplier of 3, detection is about 900 ms, though Microsoft notes overall convergence can still take up to a minute. For AWS, verify the BFD defaults in the current Direct Connect guide. Treat all figures as planning numbers and test your own circuits.

Does RPKI protect me from route leaks?

No. Route origin validation checks that the origin AS is authorised for a prefix, but a route leak keeps the legitimate origin, so it validates as Valid. Leak defence comes from RFC 9234 roles and the OTC attribute, from classic prefix filtering and maximum-prefix limits described in RFC 7454, and eventually from ASPA, which is still an Internet-Draft as of October 2026. Use RPKI alongside those, not instead of them.

Why did my BGP session drop when I advertised more routes?

Both AWS and Azure enforce prefix limits by dropping the session. A default Direct Connect private or transit virtual interface accepts 100 IPv4 prefixes from you (raisable to 1,000 with prefix controls), and ExpressRoute private peering accepts 4,000 IPv4 (10,000 with the premium add-on). Exceed the number and the session goes idle or drops. Summarise your advertisements, filter with an outbound prefix list, and set your own maximum-prefix threshold lower.

Can I use AS path prepending to steer traffic into AWS or Azure?

Sometimes. AWS evaluates local-preference communities before AS_PATH on private and transit VIFs, so a community is stronger than prepending, and prepending from a private ASN on a public VIF is stripped when AWS substitutes 7224. Azure removes private ASNs on Microsoft peering and ignores community values you send. Use more specific prefixes first, then communities where honoured, then prepending as a last, weaker nudge.