Layer-3 BGP Neighbor Discovery

The Data Center (DC) environment presents unusual problems of scale, e.g. O(10,000) forwarding devices, while its homogeneity presents opportunities for simple approaches. Layer-3 Discovery and Liveness (L3DL), , provides a solution at Layer-2. This document (set) provides a solition at Layer-3, attempting to be as similar as possible to L3DL. Layer-3 BGP Neighbor Discovery (L3BND) provides brutally simple mechanisms for devices to Discover each other's IP Addresses, Discover mutually supported layer-3 encapsulations, e.g. IP/MPLS, Discover Layer-3 IP and/or MPLS addressing of interfaces of the encapsulations, Provide for authenticity verification of protocol messages. In this document, the use case for L3BND is for point to point links in a datacenter Clos in order to exchange the data needed for bootstrapping BGP-based peering. Once IP connectivity has been leveraged to get layer-3 addressability and forwarding capabilities, normal IP forwarding and routing can take over. L3BND might be found to be more widely applicable to a range of routing and similar protocols which need Layer-3 neighbor discovery.

Even though it concentrates on the inter-device layer, this document relies heavily on routing terminology. The following attempts to clarify the use of some possibly confusing terms: Autonomous System Number , a BGP identifier for an originator of Layer-3 routes, particularly BGP announcements. A hierarchic subset of a crossbar switch topology commonly used in data centers . The L3BND content of a single Layer-3 Multicast UDP Datagram. Address Family Indicator and Subsequent Address Family Indicator (AFI/SAFI). I.e. classes of layer-2.5 and 3 addresses such as IPv4, IPv6, MPLS, etc. A logical connection between two logical ports on two devices. E.g. two VLANs between the same two ports are two links. Massive Data Center, commonly composed of thousands of Top of Rack Switches (TORs). Maximum Transmission Unit, the size in octets of the largest packet that can be sent on a medium, see 1.3.3. Protocol Data Unit, an L3DL application layer message. A PDU's content may need to be broken into multiple Datagrams to make it through MTU or other restrictions. An 32-bit identifier unique in the current routing domain, see . An established, via OPEN PDUs, session between two L3BND capable IP interfaces on a link, Top Of Rack switch, aggregates the servers in a rack and connects to aggregation layers of the Clos tree, AKA the Clos spine. Zero Touch Provisioning gives devices initial addresses, credentials, etc. on boot/restart.

L3BND is primarily designed for a Clos type datacenter scale and topology, but can accommodate richer topologies which contain potential cycles. While L3BND is designed for the MDC, there are no inherent reasons it could not run on a WAN. The authentication and authorization needed to run safely on a WAN need to be considered, and the appropriate level of security options chosen. The number of addresses of one Encapsulation type on an interface link may be quite large given a TOR with tens of servers, each server having a few hundred micro-services, resulting in an inordinate number of addresses. And highly automated micro-service migration can cause serious address prefix disaggregation, resulting in interfaces with thousands of disaggregated prefixes. Therefore the L3BND protocol is session oriented and uses incremental announcement and withdrawal with session restart, a la BGP ().

Devices discover each other's IP Addresses on logical links using Layer-3 Multicast UDP The protocol can up-level to TCP Further IP Address reachability may be exchanged IP Encapsulation data are exchanged A BGP-like upper layer protocol is assumed to use the identifiers and encapsulation data to discover and build a topology database using BGP

+-------------------+ +-------------------+ +-------------------+ | Device | | Device | | Device | | | | | | | |+-----------------+| |+-----------------+| |+-----------------+| || || || || || || || BGP <+---+> BGP <+---+> BGP || || || || || || || |+--------^--------+| |+--------^--------+| |+--------^--------+| | | | | | | | | | | | | | | | | | | |+--------+--------+| |+--------+--------+| |+--------+--------+| || Encapsulations || || Encapsulations || || Encapsulations || || & Addresses || || & Addresses || || & Addresses || |+--------^--------+| |+--------^--------+| |+--------^--------+| | | | | | | | | | | | | | | | | | | |+--------v--------+| |+--------v--------+| |+--------v--------+| || || || || || || ||Inter-Device PDUs<+---+>Inter-Device PDUs<+---+>Inter-Device PDUs|| || || || || || || |+-----------------+| |+-----------------+| |+-----------------+| +-------------------+ +-------------------+ +-------------------+ There are two protocols, the inter-device (left-right in the diagram) per-link layer-3 discovery and the API to the upper level BGP-like routing protocol (up-down in the above diagram): A Layer-3 Multicast UDP datagram is used broadcast an IP Address's willingness to explore higher level peering. This is responded to by a Unicast Layer-3 UDP datagram indicating willingness by a second IP Address on the link. The two devices than establish a TCP level session in which inter-device PDUs are used to exchange device and logical link identities and layer-2.5 (MPLS) and 3 identifiers (not payloads), e.g. more IP Addresses, loopback addresses, port identities, VLAN IDs, and Encapsulations. A L3BDL extension, describes the next upper layer L3BDL protocol to exchange BGP parameter information. The upper layer BGP family routing protocols cross all the devices, though they are not part of these L3BND protocols.

Two devices discover each other and their respective identities by one sending a Multicast UDP HELLO PDU (), and an other responding with a Unicast UDP HOWDY PDU. To assure discovery of new devices coming up on a multi-link topology, devices on such a topology, and only on a multi-link topology, send periodic HELLOs forever, see . Once a new device is recognized, both devices attempt to negotiate and establish a TCP session by sending unicast OPEN PDUs () to the source MAC addresses (plus VIDs if VLANs) of the received HELLOs. Once a session is established through the OPEN exchange, the Encapsulations () configured on an end point may be announced and modified. Note that these are only the encapsulation and addresses configured on the announcing interface; though a device's loopback and overlay interface(s) may also be announced. When two devices on a link have compatible Encapsulations and addresses, i.e. the same AFI/SAFI and the same subnet, the link is announced via the BGP-LS API.

The HELLO, , is a priming message sent on all configured logical links. It is a small L3BND Multicasy UDP PDU with the simple goal of discovering the IP identities of IP speaking link endpoint(s) reachable from an IP interface The HELLO PDU is either IPv4 or IPv6, which signals the protocol to be used for the rest of the session(s) between end-points. An interface on the link receiving the HELLO PDU and is willing to negotiate higher level protocols responds with a Unicast UDP HOWDY PDU , revealing it preferred IP address. Given the HELLO and HOWDY have revealed compatible IP addresses, the two endpoints establish a TCP session. This session may be secured with TLS, see . The OPEN, PDUs, used to details about the L3BND session, and the ACK/ERROR PDU, are mandatory; other PDUs are optional; though at least one encapsulation SHOULD be agreed at some point. The following is a ladder-style diagram of the L3BND protocol exchanges:

| HELLO | Logical Link Peer discovery |---------------------------->| | HOWDY | Mandatory |<----------------------------| | | | | | OPEN | IDs, security, etc. |---------------------------->| | ACK | |<----------------------------| | | | OPEN | Mandatory |<----------------------------| | ACK | |---------------------------->| | | | | | Interface IPv4 Addresses | Interface IPv4 Addresses |---------------------------->| Optional | ACK | |<----------------------------| | | | Interface IPv4 Addresses | |<----------------------------| | ACK | |---------------------------->| | | | | | Interface IPv6 Addresses | Interface IPv6 Addresses |---------------------------->| Optional | ACK | |<----------------------------| | | | Interface IPv6 Addresses | |<----------------------------| | ACK | |---------------------------->| | | | | | Interface MPLSv4 Labels | Interface MPLSv4 Labels |---------------------------->| Optional | ACK | |<----------------------------| | | | Interface MPLSv4 Labels | Interface MPLSv4 Labels |<----------------------------| Optional | ACK | |---------------------------->| | | | | | Interface MPLSv6 Labels | Interface MPLSv6 Labels |---------------------------->| Optional | ACK | |<----------------------------| | | | Interface MPLSv6 Labels | Interface MPLSv6 Labels |<----------------------------| Optional | ACK | |---------------------------->|

The basic L3BND application layer PDU is a typical TLV (Type Length Value) PDU. It includes a signature to provide optional integrity and authentication. It may be broken into multiple Datagrams, see xref target="transport".

L3BND discovers neighbors on logical links and establishes sessions between the two ends of all consenting discovered logical links. A logical link is described by a pair of Logical Link Endpoint Identifiers, LLEIs. An LLEI is a variable length descriptor which could be an ASN, a classic RouterID, a catenation of the two, an eight octet ISO System Identifier , or any other identifier unique to a single logical link endpoint in the topology. An L3BND deployment will choose and define an LLEI which suits its needs, simple or complex. Examples of two extremes follow: A simplistic view of a link between two devices is two ports, identified by unique MAC addresses, carrying a layer-3 protocol conversation. In this case, the MAC addresses might suffice for the LLEIs. Unfortunately, things can get more complex. Multiple VLANs can run between those two MAC addresses. In practice, many real devices use the same MAC address on multiple ports and/or sub-interfaces. Therefore, in the general circumstance, a fully described LLEI might be as follows:

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + System Identifier + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ifIndex | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ System Identifier, a la , is an eight octet identifier unique in the entire operational space. Routers and switches usually have internal MAC Addresses which can be padded with high order zeros and used if no System ID exists on the device. If no unique identifier is burned into a device, the local L3BND configuration SHOULD create and assign a unique one, likely by configuration. ifIndex is the SNMP identifier of the (sub-)interface, see . This uniquely identifies the port. For a layer-3 tagged sub-interface or a VLAN/SVI interface, IfIndex is that of the logical sub-interface, so no further disambiguation is needed. L3BND PDUs learned over VLAN-ports may be interpreted by upper layer-3 routing protocols as being learned on the corresponding layer-3 SVI interface for the VLAN. LLEIs are big-endian.

The HELLO PDU is unique in that it is encapsulated in a multicast UDP PDU. It solicits HOWDY, , response(s) from other peers on the link. See for why multicast is used. The destination Multicast Addressees to be used MUST be one of the following, see clause 9.2.2 of xref IEEE802-2014: Nearest Bridge = Propagation constrained to a single physical link; stopped by all types of bridges (including MPRs (media converters)). This SHOULD be used when the link is known to be a simple point to point link. When a switch receives a frame with a multicast destination MAC it does not recognize, it forwards to all ports. This destination MAC SHOULD be sent when the interface is known to be connected to a switch. See . This SHOULD be used when the link may be a multi-point link. All L3BND PDUs than HELLO AND HOWDY sent via TCP, as the peer's destination IP address is known after the HELLO/HOWDY exchange. When an interface is turned up on a device, it SHOULD issue a HELLO if it is to participate in L3BND sessions. If a constrained Nearest Bridge destination address has been configured for a point-to-point interface, see above, then the HELLO SHOULD NOT be repeated once a session has been created by an exchange of OPENs. If the configured destination address is one that is propagated by switches, the HELLO SHOULD be repeated at a configured interval, with a default of 60 seconds. This allows discovery by new devices which come up on the layer-2 mesh. In this multi-link scenario, the operator should be aware of the trade-off between timer tuning and network noise and adjust the inter-HELLO timer accordingly. The source IP address of the HELLO is the address soliciting peering, and HOWDY PDU responses MUST be unicast to it using the same AFI.

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 0 | Payload Length = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | +-+-+-+-+-+-+-+-+ If more than one device responds, one adjacency is formed for each unique source IP address. L3BND treats each adjacency as a separate logical link. When a HELLO is received from a source IP address (plus VID if VLAN) with which there is no established L3BND session, the receiver SHOULD respond by sending an HOWDY PDU to the source IP address (plus VID) of the HELLO. The two devices may then establish an L3BND TCP session. To ameliorate possible load spikes during bootstrap or event recovery, there SHOULD be a jittered delay between receipt of a HELLO and issue of the HOWDY. The default delay range SHOULD be zero to five seconds, and MUST be configurable. If a HELLO is received from an IP Address with which there is an established session, the HELLO should be dropped. The Payload Length is zero as there is no payload.

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 1 | Payload Length = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | +-+-+-+-+-+-+-+-+

Each device has learned the other's MAC Address from the HELLO exchange, see . Therefore the OPEN and all subsequent PDUs MUST BE unicast, as opposed to the HELLO's multicast frame.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 1 | Payload Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Nonce | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | LLEI Length | My LLEI | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | AttrCount | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Attribute List ... | Auth Type | Key Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Key ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Serial Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sig Type | Signature Length | Signature ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The Payload Length is the number of octets in all fields of the PDU from the Nonce through the Serial Number, not including the three final signature fields. The Nonce enables detection of a duplicate OPEN PDU. It SHOULD be either a random number or a high resolution timestamp. It is needed to prevent session closure due to a repeated OPEN caused by a race or a dropped or delayed ACK. My LLEI is the sender's LLEI, see . AttrCount is the number of attributes in the Attribute List. Attributes are single octets the semantics of which are operator-defined. A node may have zero or more operator-defined attributes, e.g.: spine, leaf, backbone, route reflector, arabica, ... Attribute syntax and semantics are local to an operator or datacenter; hence there is no global registry. Nodes exchange their attributes only in the OPEN PDU. Auth Type is the Signature algorithm suite, see . Key Length is a 16-bit field denoting the length in octets of the Key itself, not including the Auth Type or the Key Length. If the Auth Type is zero, then the Key Length MUST also be zero, and there MUST BE no Key data. The Key is specific to the operational environment. A failure to authenticate is a failure to start the L3BND session, an ERROR PDU MUST BE sent (Error Code 3), and HELLOs MUST be restarted. Although delay and jitter in responding with an OPEN were specified above, beware of load created by long strings of authentication failures and retries. A configurable failure count limit (default 8) SHOULD result in giving up on the connection attempt. The Serial Number is a monotonically increasing 32-bit value representing the sender's state at the time of sending the last PDU. It may be an integer, a timestamp, etc. If incrementing the Serial Number would cause it to be zero, it should be incremented again. On session restart (new OPEN), a receiver MAY send the last received Serial Number to tell the sender to only send data with a Serial Number greater (in the sense), or send a Serial Number of zero to request all data. The Serial Number supports session resumption in anticipation of peers having a very large amount of state they would prefer not to re-exchange because of some glitch. The Serial Number is not expected to wrap for a considerable time, e.g. days or weeks. But to address the rare case it does, on DNS Serial Number Arithmetic should be used as it is in the Transmission Sequence Number. This allows a sender of an OPEN to tell the receiver that the sender would like to resume a session and that the receiver only needs to send data starting with the PDU with the lowest Serial Number greater (in the sense) than the one sent in the OPEN. If the sender is not trying to resume a dropped session, the Serial Number MUST be zero. If the receiver of an OPEN PDU with a non-zero Serial Number can not resume from the requested point, it should return an ACK with an Error Code of 2, Session could not be continued. The sender of the failing OPEN PDU SHOULD then send an OPEN PDU with a Serial Number of zero. The Signature fields are described in and in an asymmetric key environment serve as a proof of possession of the signing auth data by the sender. Once two logical link endpoints know each other, and have ACKed each other's OPEN PDUs, Layer-2 KEEPALIVEs (see ) MAY be started to ensure Layer-2 liveness and keep the session semantics alive. The timing and acceptable drop of KEEPALIVE PDUs are discussed in . If a sender of OPEN does not receive an ACK of the OPEN PDU, then they MUST resend the same OPEN PDU, with the same Nonce. Resending an unacknowledged OPEN PDU, like other ACKed PDUs, SHOULD use exponential back-off, see . If a properly authenticated OPEN arrives at L3BND speaker A with a new Nonce from an LLEI, speaker B, with which A believes it already has an L3BND session (OPENs have already been exchanged), and the Serial Number in the OPEN PDU is non-zero, speaker A SHOULD establish a new sending session by sending an OPEN with the Serial Number being the same as that of A's last sent and ACKed PDU. A MUST resume sending encapsulations etc. subsequent to the requested Sequence Number. And B MUST retain all previously discovered encapsulation and other data received from A. If a properly authenticated OPEN arrives with a new Nonce from an LLEI with which the receiving logical link endpoint believes it already has an L3BND session (OPENs have already been exchanged), and the Serial Number in the OPEN is zero, then the receiver MUST assume that the sending LLEI or entire device has been reset. All Previously discovered encapsulation data MUST NOT be kept and MUST BE withdrawn via the BGP-LS API and the recipient MUST respond with a new OPEN.

The ACK PDU acknowledges receipt of a PDU and reports any error condition which might have been raised.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 3 | Payload Length = 5 ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | ACKed PDU | EType | Error Code | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Error Hint | Sig Type |Signature Leng.~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Signature ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The ACK acknowledges receipt of an OPEN, Encapsulation, VENDOR PDU, etc. The ACKed PDU is the PDU Type of the PDU being acknowledged, e.g., OPEN, one of the Encapsulations, etc. If there was an error processing the received PDU, then the EType is non-zero. If the EType is zero, Error Code and Error Hint MUST also be zero. A non-zero EType is the receiver's way of telling the PDU's sender that the receiver had problems processing the PDU. The Error Code and Error Hint will tell the sender more detail about the error. The decimal value of EType gives a strong hint how the receiver sending the ACK believes things should proceed: 0 - No Error, Error Code and Error Hint MUST be zero 1 - Warning, something not too serious happened, continue 2 - Session should not be continued, try to restart 3 - Restart is hopeless, call the operator 4-15 - Reserved The Error Codes, noting protocol failures, are listed in . Someone stuck in the 1990s might think the catenation of EType and Error Code as an echo of 0x1zzz, 0x2zzz, etc. They might be right; or not. The Error Hint, an arbitrary 16 bits, is any additional data the sender of the error PDU thinks will help the recipient or the debugger with the particular error. The Signature fields are described in .

If a PDU sender expects an ACK, e.g. for an OPEN, an Encapsulation, a VENDOR PDU, etc., and does not receive the ACK for a configurable time (default one second), and the interface is live at layer-2, the sender resends the PDU using exponential back-off, see . This cycle MAY be repeated a configurable number of times (default three) before it is considered a failure. The session MAY BE considered closed in this case of this ACK failure. If the link is broken at layer-2, retransmission MAY BE retried when the link is restored.

Once the devices know each other's LLEIs, know each other's upper layer (L2.5 and L3) identities, have means to ensure link state, etc., the L3BND session is considered established, and the devices SHOULD exchange L3 interface encapsulations, L3 addresses, and L2.5 labels. The Encapsulation types the peers exchange may be IPv4 (), IPv6 (), MPLS IPv4 (), MPLS IPv6 (), and/or possibly others not defined here. The sender of an Encapsulation PDU MUST NOT assume that the peer is capable of the same Encapsulation Type. An ACK () merely acknowledges receipt. Only if both peers have sent the same Encapsulation Type is it safe for Layer-3 protocols to assume that they are compatible for that type. A receiver of an encapsulation might recognize an addressing conflict, such as both ends of the link trying to use the same address. In this case, the receiver SHOULD respond with an error (Error Code 2) ACK. As there may be other usable addresses or encapsulations, this error might log and continue, letting an upper layer topology builder deal with what works. Further, to consider a logical link of a type to formally be established so that it may be pushed up to upper layer protocols, the addressing for the type must be compatible, e.g. on the same IP subnet.

The header for all encapsulation PDUs is as follows:

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type | Payload Length ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Serial Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Encapsulation List... | Sig Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Signature Length | Signature ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ An Encapsulation PDU describes zero or more addresses of the encapsulation type. The 24-bit Count is the number of Encapsulations in the Encapsulation list. The Serial Number is a monotonically increasing 32-bit value representing the sender's state in time. It may be an integer, a timestamp, etc. On session restart (new OPEN), a receiver MAY send the last received Session Number to tell the sender to only send newer data. If a sender has multiple links on the same interface, separate state: data, ACKs, etc. must be kept for each peer session. Over time, multiple Encapsulation PDUs may be sent for an interface as configuration changes. If the length of an Encapsulation PDU exceeds the Datagram size limit on media, the PDU is broken into multiple Datagrams. See . The Signature fields are described in . The Receiver MUST acknowledge the Encapsulation PDU with a Type=3, ACK PDU () with the Encapsulation Type being that of the encapsulation being announced, see . If the Sender does not receive an ACK in a configurable interval (default one second), and the interface is live at layer-2, they SHOULD retransmit. After a user configurable number of failures (default three), the L3BND session should be considered dead and the OPEN process SHOULD be restarted. If the link is broken at layer-2, retransmission MAY BE retried if data have not changed in the interim.

The Encapsulation Flags are a sequence of bit fields as follows:

0 1 2 3 4 ... 7 +------------+------------+------------+------------+------------+ | Ann/With | Primary | Under/Over | Loopback | Reserved ..| +------------+------------+------------+------------+------------+ Each encapsulation in an Encapsulation PDU of Type T may announce new and/or withdraw old encapsulations of Type T. It indicates this with the Ann/With Encapsulation Flag, Announce == 1, Withdraw == 0. Each Encapsulation interface address in an Encapsulation PDU is either a new encapsulation be announced (Ann/With == 1) (yes, a la BGP) or requests one be withdrawn (Ann/With == 0). Adding an encapsulation which already exists SHOULD raise an Announce/Withdraw Error (see ); the EType SHOULD be 2, suggesting a session restart (see so all encapsulations will be resent. If an LLEI has multiple addresses for an encapsulation type, one and only one address MAY be marked as primary (Primary Flag == 1) for that Encapsulation Type. An Encapsulation interface address in an Encapsulation PDU MAY be marked as a loopback, in which case the Loopback bit is set. Loopback addresses are generally not seen directly on an external interface. One or more loopback addresses MAY be exposed by configuration on one or more L3BND speaking external interfaces, e.g. for iBGP peering. They SHOULD be marked as such, Loopback Flag == 1. Each Encapsulation interface address in an Encapsulation PDU is that of the direct 'underlay interface (Under/Over == 1), or an 'overlay' address (Under/Over == 0), likely that of a VM or container guest bridged or configured on to the interface already having an underlay address.

The IPv4 Encapsulation describes a device's ability to exchange IPv4 packets on one or more subnets. It does so by stating the interface's addresses and the corresponding prefix lengths.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 4 | Payload Length ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Serial Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Encaps Flags | IPv4 Address ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | PrefixLen | more ... | Sig Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Signature Length | Signature ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The 24-bit Count is the sum of the number of IPv4 Encapsulations being announced and/or withdrawn.

The IPv6 Encapsulation describes a logical link's ability to exchange IPv6 packets on one or more subnets. It does so by stating the interface's addresses and the corresponding prefix lengths.

As an MPLS enabled interface may have a label stack, see , a variable length list of labels is needed. These are the labels the sender will accept for the prefix to which the list is attached.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Label Count | Label | Exp |S| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Label | Exp |S| more ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ A Label Count of zero is an implicit withdraw of all labels for that prefix on that interface.

The MPLS IPv4 Encapsulation describes a logical link's ability to exchange labeled IPv4 packets on one or more subnets. It does so by stating the interface's addresses the corresponding prefix lengths, and the corresponding labels which will be accepted for each address.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 6 | Payload Length ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Serial Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Encaps Flags | MPLS Label List ... | ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ IPv4 Address | PrefixLen | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | more ... | Sig Type | Signature Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Signature | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The 24-bit Count is the sum of the number of MPLSv4 Encapsulation being announced and/or withdrawn.

The MPLS IPv6 Encapsulation describes a logical link's ability to exchange labeled IPv6 packets on one or more subnets. It does so by stating the interface's addresses, the corresponding prefix lengths, and the corresponding labels which will be accepted for each address.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 7 | Payload Length ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Serial Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Encaps Flags | MPLS Label List ... | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + | | + + | IPv6 Address | + +-+-+-+-+-+-+-+-+ | | Prefix Len | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | more ... | Sig Type | Signature Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Signature ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The 24-bit Count is the sum of the number of MPLSv6 Encapsulations being announced and/or withdrawn.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 255| Payload Length ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Serial Number ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ent Type | Enterprise Data ... ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Sig Type | Signature Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Signature ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Vendors or enterprises may define TLVs beyond the scope of L3BND standards. This is done using a Private Enterprise Number followed by Enterprise Data in a format defined for that Enterprise Number and Ent Type. Ent Type allows a VENDOR PDU to be sub-typed in the event that the vendor/enterprise needs multiple PDU types. As with Encapsulation PDUs, a receiver of a VENDOR PDU MUST respond with an ACK or an ERROR PDU. Similarly, a VENDOR PDU MUST only be sent over an open session.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PDU Type = 2 | Payload Length = 0 ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ | Sig Type = 0 | Signature Length = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ L3BND devices SHOULD beacon frequent Layer-2 KEEPALIVE PDUs to ensure session continuity. The inter-KEEPALIVE interval is configurable, with a default of ten seconds. A receiver may choose to ignore KEEPALIVE PDUs. An operational deployment MUST BE configured whether to use KEEPALIVEs or not, either globally, or as finely as to per-link granularity. Disagreement MAY result in repeated session failure and reestablishment. KEEPALIVEs SHOULD be beaconed at a configured frequency. One per second is the default. Layer-3 liveness, such as BFD, may be more (or less) aggressive. When a sender transmits a PDU which is not a KEEPALIVE, the sender SHOULD reset the KEEPALIVE timer. I.e. sending any PDU acts as a keepalive. Once the last fragment has been sent, the KEEPALIVE timer SHOULD be restarted. Do not wait for the ACK. If a KEEPALIVE or other PDUs have not been received from a peer with which a receiver has an open session for a configurable time (default 30 seconds), the link SHOULD be presumed down. The devices MAY keep configuration state and restore it without retransmission if no data have changed. Otherwise, a new session SHOULD be established and new Encapsulation PDUs exchanged.

Layer-2 liveness may be continuously tested by KEEPALIVE PDUs, see . As layer-2.5 or layer-3 connectivity could still break, liveness above layer-2 MAY be frequently tested using BFD () or a similar technique. This protocol assumes that one or more Encapsulation addresses may be used to ping, run BFD, or whatever the operator configures.

Thus far, a one-hop point-to-point logical link discovery protocol has been defined. The devices know their unique LLEIs and know the unique peer LLEIs and Encapsulations on each logical link interface. Full topology discovery is not appropriate at the L3BND layer, so Dijkstra a la IS-IS etc. is assumed to be done by higher level protocols such as BGP-SPF. Therefore the LLEIs, link Encapsulations, and state changes are pushed North via a small subset of the BGP-LS API. The upper layer routing protocol(s), e.g. BGP-SPF, learn and maintain the topology, run Dijkstra, and build the routing database(s). For example, if a neighbor's IPv4 Encapsulation address changes, the devices seeing the change push that change Northbound.

BGP-LS defines BGP-like Datagrams describing logical link state (links, nodes, link prefixes, and many other things), and a new BGP path attribute providing Northbound transport, all of which can be ingested by upper layer protocols such as BGP-SPF; see Section 4 of . For IPv4 links, TLVs 259 and 260 are used. For IPv6 links, TLVs 261 and 262. If there are multiple addresses on a link, multiple TLV pairs are pushed North, having the same ID pairs.

The Northbound protocol needs a few minor extensions to BGP-LS. Luckily, others have needed the same extensions. Similarly to BGP-SPF, the BGP protocol is used in the Protocol-ID field specified in table 1 of . The local and remote node descriptors for all NLRI are the IDs described in . This is equivalent to an adjacency SID or a node SID if the address is a loopback address. Label Sub-TLVs from Section 2.1.1, are used to associate one or more MPLS Labels with a link.

This section explores some trade-offs taken and some considerations.

A device with multiple Layer-2 interfaces, traditionally called a switch, may be used to forward frames and therefore packets from multiple devices to one logical interface (LLEI), I, on an L3BND speaking device. Interface I could discover a peer J across the switch. Later, a prospective peer K could come up across the switch. If I was not still sending and listening for HELLOs, the potential peering with K could not be discovered. Therefore, on multi-link interfaces, L3BND MUST continue to send HELLOs as long as they are turned up.

Both HELLO and KEEPALIVE are periodic. KEEPALIVE might be eliminated in favor of keeping only HELLOs. But KEEPALIVEs are unicast, and thus less noisy on the network, especially if HELLO is configured to transit layer-2-only switches, see .

One can think of the protocol as an instance (i.e. state machine) which runs on each logical link of a device. As the upper routing layer must view VLAN topologies as separate graphs, L3BND treats VLAN ports as separate links. L3BND PDUs learned over VLAN-ports may be interpreted by upper layer-3 routing protocols as being learned on the corresponding layer-3 SVI interface for the VLAN. As Sub-Interfaces each have their own LLIEs, they act as separate interfaces, forming their own links.

An implementation SHOULD provide the ability to configure each logical interface as L3BND speaking or not. An implementation SHOULD provide the ability to configure whether HELLOs on an L3BND enabled interface send Nearest Bridge or the MAC which is propagated by switches from that interface; see . An implementation SHOULD provide the ability to distribute one or more loopback addresses or interfaces into L3BND on an external L3BND speaking interface. An implementation SHOULD provide the ability to distribute one or more overlay and/or underlay addresses or interfaces into L3BND on an external L3BND speaking interface. An implementation SHOULD provide the ability to configure one of the addresses of an encapsulation as primary on an L3BND speaking interface. If there is only one address for a particular encapsulation, the implementation MAY mark it as primary by default. An implementation MAY allow optional configuration which updates the local forwarding table with overlay and underlay data both learned from L3BND peers and configured locally.

The protocol as is MUST NOT be used outside a datacenter or similarly closed environment without authentication and authorization mechanisms such as . Many MDC operators have a strange belief that physical walls and firewalls provide sufficient security. This is not credible. All MDC protocols need to be examined for exposure and attack surface. In the case of L3BND, Authentication and Integrity as provided in is strongly recommended. It is generally unwise to assume that on the wire Layer-2 is secure. Strange/unauthorized devices may plug into a port. Mis-wiring is very common in datacenter installations. A poisoned laptop might be plugged into a device's port, form malicious sessions, etc. to divert, intercept, or drop traffic. Similarly, malicious nodes/devices could mis-announce addressing. If OPENs are not being authenticated, an attacker could forge an OPEN for an existing session and cause the session to be reset. For these reasons, the OPEN PDU's authentication data exchange SHOULD be used. If the KEEPALIVE PDU is not signed (as suggested in ) to save computation, then a MITM could fake a session being alive.

This document requests the IANA create a registry for L3BND PDU Type, which may range from 0 to 255. The name of the registry should be L3BND-PDU-Type. The policy for adding to the registry is RFC Required per , either standards track or experimental. The initial entries should be the following:

PDU Code PDU Name ---- ------------------- 0 HELLO 1 OPEN 2 KEEPALIVE 3 ACK 4 IPv4 Announcement 5 IPv6 Announcement 6 MPLS IPv4 Announcement 7 MPLS IPv6 Announcement 8-254 Reserved 255 VENDOR

This document requests the IANA create a registry for L3BND Signature Type, AKA Sig Type, which may range from 0 to 255. The name of the registry should be L3BND-Signature-Type. The policy for adding to the registry is RFC Required per , either standards track or experimental. The initial entries should be the following:

Number Name ------ ------------------- 0 Null 1-255 Reserved

This document requests the IANA create a registry for L3BND PL Flag Bits, which may range from 0 to 7. The name of the registry should be L3BND-PL-Flag-Bits. The policy for adding to the registry is RFC Required per , either standards track or experimental. The initial entries should be the following:

Bit Bit Name ---- ------------------- 0 Announce/Withdraw (ann == 0) 1 Primary 2 Underlay/Overlay (under == 0) 3 Loopback 4-7 Reserved

This document requests the IANA create a registry for L3BND Error Codes, a 16 bit integer. The name of the registry should be L3BND-Error-Codes. The policy for adding to the registry is RFC Required per , either standards track or experimental. The initial entries should be the following:

Error Code Error Name ---- ------------------- 0 No Error 1 Checksum Error 2 Logical Link Addressing Conflict 3 Authorization Failure 4 Announce/Withdraw Error

This document requires a new EtherType. This document requires a new multicast MAC address that will be broadcast through a switch.

The authors thank Cristel Pelsser for multiple reviews, Harsha Kovuru for comments during implementation, Jeff Haas for review and comments, Jörg Ott for an early but deep transport review, Joe Clarke for a useful review, John Scudder for deeply serious review and comments, Larry Kreeger for a lot of layer-2 clue, Martijn Schmidt for his contribution, Nalinaksh Pai for transport discussions, Neeraj Malhotra for review, Paul Congdon for Ethernet hints, Russ Housley for checksum discussion and sBox, and Steve Bellovin for checksum advice.