Network Working Group                                          L. Dunbar
Internet-Draft                                                 Futurewei
Intended status: Standards Track                                S. Hares
Expires: 23 July 2025                                             Huawei
                                                             K. Majumdar
                                                                  Oracle
                                                               R. Raszuk
                                                                  Arrcus
                                                      V. Kasiviswanathan
                                                                  Arista
                                                         19 January 2025


                  BGP UPDATE for SD-WAN Edge Discovery
                 draft-ietf-idr-sdwan-edge-discovery-21

Abstract

   The document describes the BGP mechanisms for SD-WAN edge node
   property discovery.  These mechanisms include a new tunnel type and
   subTLVs for the BGP Tunnel-Encapsulation Attribute [RFC9012] and set
   of NLRI (network layer reachability information) for Software-Defined
   Wide-Area Network (SD-WAN) underlay information.

   In the context of this document, BGP Route Reflector (RR) is the
   component of the SD-WAN Controller that receives the BGP UPDATE from
   SD-WAN edges and in turns propagates the information to the intended
   peers that are authorized to communicate via the SD-WAN overlay
   network.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119] [RFC8174]
   when, and only when, they appear in all capitals, as shown here.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.





Dunbar, et al.            Expires 23 July 2025                  [Page 1]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 23 July 2025.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   4
     1.1.  SD-WAN Secure L3VPNs  . . . . . . . . . . . . . . . . . .   4
     1.2.  SD-WAN Secure Links . . . . . . . . . . . . . . . . . . .   5
     1.3.  Conventions used in this document . . . . . . . . . . . .   5
   2.  Framework of BGP SD-WAN Edge Discovery  . . . . . . . . . . .   7
     2.1.  Supported Client Routes . . . . . . . . . . . . . . . . .   7
     2.2.  Topologies of SD-WAN Edge Discovery . . . . . . . . . . .   8
       2.2.1.  SD-WAN Secure L3VPN Example Topology  . . . . . . . .   8
       2.2.2.  SD-WAN Secure Links Example Topology  . . . . . . . .  11
       2.2.3.  RR connections to BGP Peers in SD-WAN . . . . . . . .  12
     2.3.  The Objectives of SD-WAN Edge Discovery . . . . . . . . .  13
     2.4.  Examples of SD-WAN Edge Discovery . . . . . . . . . . . .  14
     2.5.  Comparing SD-WAN Secure L3VPN with Pure IPsec VPN . . . .  15
     2.6.  SD-WAN Segmentation, Virtual Topology and Client VPN  . .  17
   3.  BGP SD-WAN Mechanisms . . . . . . . . . . . . . . . . . . . .  20
     3.1.  SD-WAN-Hybrid Tunnel Encoding . . . . . . . . . . . . . .  20
     3.2.  SD-WAN Underlay UPDATE  . . . . . . . . . . . . . . . . .  24
       3.2.1.  The NLRI for SD-WAN Underlay Tunnel Update  . . . . .  24
       3.2.2.  Validation of SD-WAN NLRI . . . . . . . . . . . . . .  26
       3.2.3.  BGP Path Attributes attached to SD-WAN NLRI . . . . .  26
     3.3.  IPsec SA Property Sub-TLVs  . . . . . . . . . . . . . . .  27
       3.3.1.  IPsec-SA-ID Sub-TLV . . . . . . . . . . . . . . . . .  28
       3.3.2.  IPsec SA Rekey Counter Sub-TLV  . . . . . . . . . . .  29
       3.3.3.  IPsec Public Key Sub-TLV  . . . . . . . . . . . . . .  31
       3.3.4.  IPsec SA Proposal Sub-TLV . . . . . . . . . . . . . .  33



Dunbar, et al.            Expires 23 July 2025                  [Page 2]

Internet-Draft            SDWAN Edge Discovery              January 2025


       3.3.5.  Simplified IPsec SA Sub-TLV . . . . . . . . . . . . .  34
       3.3.6.  Extended Port Attribute Sub-TLV . . . . . . . . . . .  36
     3.4.  Procedure for Client Routes with Hybrid SD-WAN Tunnel . .  42
       3.4.1.  SD-WAN Tunnel in Encapsulation Extended Community
               (Encap-EC)  . . . . . . . . . . . . . . . . . . . . .  43
       3.4.2.  SD-WAN Tunnel in Tunnel Encapsulation Path Attribute
               (TEA) . . . . . . . . . . . . . . . . . . . . . . . .  43
       3.4.3.  Multiple tunnels attached to One Client Route . . . .  44
       3.4.4.  SD-WAN VPN ID in the Client Route Update  . . . . . .  44
       3.4.5.  SD-WAN VPN ID in Data Plane . . . . . . . . . . . . .  44
     3.5.  Procedure for Underlay Routes with Hybrid SD-WAN Tunnel
           TLV . . . . . . . . . . . . . . . . . . . . . . . . . . .  45
       3.5.1.  Hybrid SD-WAN NLRI with a Encapsulation Extended
               Community . . . . . . . . . . . . . . . . . . . . . .  45
       3.5.2.  Underlay Route with a TEA . . . . . . . . . . . . . .  45
       3.5.3.  Underlay Routes with Port-Local-ID of zero  . . . . .  46
       3.5.4.  Multiple Tunnels attached to One Underlay Route . . .  47
     3.6.  Error handling  . . . . . . . . . . . . . . . . . . . . .  47
       3.6.1.  Error handling for the Tunnel Encapsulation
               Signaling . . . . . . . . . . . . . . . . . . . . . .  47
       3.6.2.  Error Handling for NLRI . . . . . . . . . . . . . . .  48
       3.6.3.  SDWAN NLRI and Tunnel Encapsulation Attribute . . . .  49
   4.  Manageability Considerations  . . . . . . . . . . . . . . . .  49
     4.1.  Detecting Misaligned Tunnels  . . . . . . . . . . . . . .  49
     4.2.  IPsec Attributes Mismatch . . . . . . . . . . . . . . . .  50
       4.2.1.  SD-WAN Hybrid Tunnel Mechanisms for Passing IPSec
               Security Association Info . . . . . . . . . . . . . .  50
       4.2.2.  Example creation of IPsec Security Association over
               SD-WAN Hybrid tunnel  . . . . . . . . . . . . . . . .  51
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .  52
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  53
     6.1.  Hybrid (SD-WAN) Overlay SAFI  . . . . . . . . . . . . . .  53
     6.2.  Tunnel Encapsulation Attribute Type . . . . . . . . . . .  53
     6.3.  Tunnel Encapsulation Attribute Sub-TLV Types  . . . . . .  54
     6.4.  Tunnel Encapsulation Attribute Type . . . . . . . . . . .  54
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  54
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .  54
     7.2.  Informative References  . . . . . . . . . . . . . . . . .  56
   Appendix A.  Acknowledgments  . . . . . . . . . . . . . . . . . .  57
   Contributors  . . . . . . . . . . . . . . . . . . . . . . . . . .  57
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  58










Dunbar, et al.            Expires 23 July 2025                  [Page 3]

Internet-Draft            SDWAN Edge Discovery              January 2025


1.  Introduction

   BGP [RFC4271] can be used as a control plane for a Software Defined
   Wide Area Network (SD-WAN) to support Secure VPNs or simply to
   support a set of Secure links in a network.  A brief definition of
   these two use cases is given in this section.  Section 2 describes
   the BGP framework in terms of NLRIs supported, example topologies,
   and objectives for the BGP mechanisms.  Section 3 describes the BGP
   mechanisms, and section 4 describes error handling for this
   mechanism.

   The BGP mechanisms for SD-WAN support requires a route reflector (RR)
   to have a secure connection to each BGP Peer participating in the BGP
   infrastructure support the SD-WAN.  The establishiment of a secure
   connection between each BGP Peer and the RR is outside the scope of
   this specification.

   This document describes the BGP mechanisms for an SD-WAN edge nodes
   to established a BGP peering with other SD-WAN edge nodes, and pass
   information in order to establish and update secure overlay tunnels
   [Net2Cloud].  These mechanisms include a new tunnel type and subTLVs
   for the BGP Tunnel-Encapsulation Attribute [RFC9012] and a set NLRIs
   for SD-WAN underlay information.

1.1.  SD-WAN Secure L3VPNs

   A SD-WAN network defined in [MEF70.1] and [MEF70.2], refers to a
   policy-driven network over multiple heterogeneous underlay networks
   to get better WAN bandwidth management, visibility, and control.
   This document refers to this network as a SD-WAN Secure L3VPNs.
   [SD-WAN-BGP-USAGE] defines the five requirements for BGP usage in an
   SD-WAN Secure L3VPN networks and 3 scenarios for deployment.  The
   five requirements defined are the following:

   *  Support for SD-WAN segmentation - that allows edge noded connected
      over tunnels to support VPNs.

      -  These VPNs can be MPLS VPNS ([RFC4364],[RFC4659] with VRFs,
         MPLS L2VPN [RFC4761], [RFC4762], L3VPN [RFC4364][RFC4659], or
         IPsec tunnels.

   *  Support for Client Services interfaces on edge nodes (e.g.  SD-WAN
      UNI [MEF70.1]),

   *  WD-WAN Traffic Segmentation,

   *  Zero Touch Provisioning, and




Dunbar, et al.            Expires 23 July 2025                  [Page 4]

Internet-Draft            SDWAN Edge Discovery              January 2025


   *  Constrained Propagation of SD-WAN Edge Properities by BGP
      infrastructure using a Route Reflector.

   [SD-WAN-BGP-USAGE] describes the three scenarios for SD-WAN networks
   comprised of a mixture of tunnels over private and public networks:

   *  Homogeneous Encrypted SD-WAN - where edge nodes encrypt data prior
      to entering the SD-WAN,

   *  Differential Encrypted SD-WAN - where edge nodes encrypt traffic
      traversing public networks, but do not encrypt traffic over
      private secure networks,

   *  Private VPN PE based SD-WAN - where an existing private VPN is
      expanded by adding extra ports over the public internet.

   [SD-WAN-BGP-USAGE] provides descriptions on the use of SD-WAN
   technology for L3VPNS, the provisioning of SD-WAN nodes, and example
   BGP topologie and SD-WAN forwarding mechanisms.  This document
   assumes the reader is familar with SD-WAN use case described in
   [SD-WAN-BGP-USAGE].

1.2.  SD-WAN Secure Links

   [RFC9012] defines a BGP mechanisms that links routes (prefix and Next
   Hop) to a specific tunnels using a specific encapsulation.  The SD-
   WAN Secure Links Topology uses a single Hybrid logical link on a SD-
   WAN Peer to represent multiple underlay topology links.  The SD-WAN
   peer distributes IPsec security association (IPsec-SA) related
   information regarding the hybrid link or individual underlay links.

   The traffic is routed via normal IP v4/v6 forwarding without any VPN
   addition.  The SD-WAN Secure Links provides some link security for
   some simple cases of the three scenarios from [SD-WAN-BGP-USAGE] that
   do not require L3VPN addresses (RD, prefix).

1.3.  Conventions used in this document

   The following acronyms and terms are used in this document:

   Authorized BGP peer:  Authorized BGP peer are Peers which a
      particular BGP Peer has been configured by local policy to connect
      to.

   Cloud DC:  Off-Premises Data Centers that usually host applications
      and workload owned by different organizations or tenants.

   Color-EC:  Color Extended Community defined in [RFC9012].



Dunbar, et al.            Expires 23 July 2025                  [Page 5]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Controller:  Used interchangeably with SD-WAN controller to manage
      SD-WAN overlay path creation/deletion and monitor the path
      conditions between sites.

   CPE (or C-PE):  Customer (Edge) Premises Equipment.  Please note that
      these two forms are equivalent in this document.  C-PE is used to
      emphasize the web of Provider Equipment devices.

   CPE-Based VPN:  Virtual Private Secure network formed among CPEs.
      This is to differentiate such VPNs from most commonly used PE-
      based VPNs discussed in [RFC4364].

   Encap-EC:  Encapsulation Extended Community defined in [RFC9012].

   IPsec-SA:  IPsec Security Association.

   MP-NLRI:  Multi-Protocol Network Layer Reachability Information
      (MP_REACH_NLRI) Path Attribute defined in [RFC4760].

   RR:  Route Reflector [RFC4456]

   RT-EC:  Route Target Extended Community [RFC4360]

   SA:  IPsec Security Association

   SD-WAN:  An overlay connectivity service that optimizes transport of
      IP Packets over one or more Underlay Connectivity Services by
      recognizing applications (Application Flows) and determining
      forwarding behavior by applying Policies to them.  [MEF-70.1][MEF-
      70.2]

   SD-WAN Endpoint:  can be the SD-WAN edge node address, a WAN port
      address (logical or physical) of a SD-WAN edge node, or a client
      port address.

   SD-WAN Hybrid tunnel:  A single logical tunnel that combines several
      links of different encapsulation iinto a single tunnel.  This
      logical tunnel may exist as part of a SD-WAN Secure L3VPN or
      simply be a SD-WAN secure link for a flat network.

   SD-WAN Secure L3VPN:  The mesh of SD-WAN secure tunnels that support
      the uses cases defined by [SD-WAN-BGP-USAGE] and [MEF-
      70.1][MEF70.2].  BGP transmits information about SD-WAN Hybrid
      tunnels in the SD-WAN Secure L3VPN network by sending L3VPN AFI/
      SAFI with a TEA with a tunnel type of SD WAN Hybrid Tunnel.
      Information about the IPsec Security Association (IPsec-SA) for
      the SD-WAN underlay for these SD WAN Hybrid tunnels is passed in
      the SD-WAN NLRI (AFI 1/74) with a TEA with SD-WAN Hybrid Tunnel.



Dunbar, et al.            Expires 23 July 2025                  [Page 6]

Internet-Draft            SDWAN Edge Discovery              January 2025


      This document defines the BGP mechanisms for the SD-WAN Secure
      L3VPN, but the [SD-WAN-BGP-USAGE] defines the requirements,
      scenarios, provisioning model, examples of normal BGP flows, and
      SD-WAN forwarding.

   SD-WAN Secure Links:  A mesh of SD-WAN Hybrid tunnels may connect
      several SD-WAN edge nodes in a flat (non-VPN) network.  The SD-WAN
      Secure Links use case does not support VPN identification, and
      applies only a simple creation of secure link with support for
      passing IPsec-SA information regarding the SD-WAN Hybrid Tunnel.
      BGP sends a Unicast v4/v6 NLRI (AFI/SAFI 1/1 and 2/1) with a TEA
      for a SD-WAN Hybrid tunnel to announce the existance of the link.
      Information about the IPsec Security Association (IPsec-SA) for
      the SD-WAN underlay for these SD WAN Hybrid tunnels is passed in
      the SD-WAN NLRIs (AFI/SAFI 1/74 and 2/74) with a TEA with SD-WAN
      Hybrid Tunnel

   TEA:  Tunnel Encapsulation Path Attribute [RFC9012]

   VPN:  Virtual Private Network

   VRF:  VPN Routing and Forwarding instance

   WAN:  Wide Area Network

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC8174] when, and only when, they appear in all capitals, as
   shown here.

2.  Framework of BGP SD-WAN Edge Discovery

   This section provides a framework to describe the overall solution
   parts based on the reference diagram shown in Figure 1.  The
   framework covers the following: client routes supported, example
   topologies, objectives of the SD-WAN Edge discovery, comparison of
   SD-WAN Secure VPNs with Pure IPsec VPNS, and what SD-WAN segmentation
   means in this SD-WAN context.

2.1.  Supported Client Routes

   Tunnel Encapsulation may be attached to the prefixes from the Unicast
   v4 and v6 ((AFI/SAFI 1/1 and 2/1), and L3VPNs for v4 and v6 (AFI/SAFI
   1/128 and 2/128) (per [RFC4364], [RFC4659]).






Dunbar, et al.            Expires 23 July 2025                  [Page 7]

Internet-Draft            SDWAN Edge Discovery              January 2025


   The document defines SD-WAN Secure L3VPN as the SD-WAN Secure VPN
   defined in [SD-WAN-BGP-USAGE], [MEF-70.1], and [MEF70.2].  The SD-WAN
   Secure L3VPN requires the L3VPN for IPv4 [RFC4364] and IPv6 [RFC4659]
   to be expanded to support SD-WAN Hybrid Tunnels and the passages of
   IPsec-SA information for the underlay tunnels that support SD-WAN
   Secure L3VPN.

   This document defines the SD-WAN Secure Links as links in network
   that uses SD-WAN tunnels to create secure Hybrid SD-WAN tunnels
   between SD-WAN End Nodes.  The SD-WAN Secure Links application does
   not support identification of a VPN via L3VPN NLRI.  The SD-WAN end
   node uses BGP to pass Unicast v4/v6 prefixes ((AFI/SAFI 1/1 and 2/1)
   routes with TEA with a SD-WAN Hybrid tunnel TLV.  The SD-WAN secure
   links application also passes the IPSec-SA information for the
   underlay tunnels in BGP.

   This document specifies the BGP mechanism for SD-WAN Secure L3VPN and
   SD-WAN Secure Links.

2.2.  Topologies of SD-WAN Edge Discovery

   This section describes how the topologies for SD-WAN Secure L3VPN and
   SD-WAN Secure Links can be served by the BGP SD-WAN logical Tunnel
   links.

2.2.1.  SD-WAN Secure L3VPN Example Topology

   This section summarizes the BGP requirements from the following three
   scenarios from [SD-WAN-BGP-USAGE] can be handled by a BGP control
   plane using BGP Tunnel-Encapsulation attribute [RFC9012]:

   *  Homogeneous Encrypted SD-WAN,

   *  Differential Encrypted SD-WAN, and

   *  Private VPN PE based SD-WAN.

   Based on [SD-WAN-BGP-USAGE], it is easy to see how Figure 1 matches
   scenario 1 (Homogeneneous Encrypted SD-WAN) and scenario 2
   (Differential Encrypted SD-WAN).  Recasting Figure 1 as a logical BGP
   peering topology results in a BGP topology between PEs and CN (CE at
   customer site) results in Figure 2.  Scenario 3 (Private VPNs based
   on SD-WAN) from [SD-WAN-BGP-USAGE] aligns with the Figure 2.








Dunbar, et al.            Expires 23 July 2025                  [Page 8]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Hybrid SD-WAN tunnel infrastructure requires that the PEs (C-PE1,
   C-PE2, C-PE3, and C-PE4) have an existing topology that the Hybrid
   SD-WAN tunnel overlays.  These PEs use local policy to sending the
   appropriate traffic across the appropriate network based on local
   policy.

  Hybrid SD-WAN Secure L3VPN:  Sample Topology



       BGP over secure link       +---+  BGP over secure link
                     +------------|RR |---------+
                    / BGP Peer    +-+-+ BGP Peer \
                   /                              \
                  /                                \
          +---------+                              +-------+
   +---+  | C-PE1   |--P1--( MPLS L3VPN Net1 )--P1-| C-PE2 | +---+
   |CN3|--|         |--P2--( MPLS L2VPN Net2 )--P2-|       |-|CN3|
   +---+  |         |                              |       |
   +---+  |         |                              |       | +---+
   |CN2|--| Addr    |--P3 --( WAN L3 Net3)------P3-|       |-|CN1|
   +---+  |         |--P4---(L2 direct link)----P4-|       | +---+
      |   +---------+                              +-------+    |
      \                                                         /
       \  +---------+                              +-------+  /
        +-|         |                              |       |-+
   +---+  | C-PE3   |--P1---( MPLS L3VPN Net1 )--P1| C-PE4 | +---+
   |CN2|--|         |--P2---( MPLS L2VPN Net2 )--P2|       |-|CN1|
   +---+  |         |                              |       | +---+
   +---+  |         |                              |       | +---+
   |CN1|--| Addr    |--P3 --( WAN L3 Net3)------P3-|       |-|CN2|
   +---+  |         |--P4---(L2 direct link)----P4-|       | +---+
          +---------+                              +-------+
                 \                                 /
                  \                               /
                   \ BGP Peer    +-+-+ BGP Peer  /
                    +------------|RR |----------+
       BGP over secure link      +---+  BGP over secure link

           Please note that RR may be one RR, but for clarity of diagram
           the RR has been displayed as two parts.


                 Figure 1: Hybrid SD-WAN Secure L3VPN

  Hybrid SD-WAN BGP Mesh :  Logical BGP Topology for SD-WAN





Dunbar, et al.            Expires 23 July 2025                  [Page 9]

Internet-Draft            SDWAN Edge Discovery              January 2025


                      Logical topology for BGP peers

                                  +----+
                        +---------| RR |---------+
                       /          +--+-+          \
                      /                            \
               +------+                         +------+
        +---+  |      |------SDWAN hybrid-------|      |  +---+
        |CN3|--|CE-PE1|                         |CE-PE2|--|CN3|
        +---+  |      |           /-------------|      |  +---+
        +---+  |      |---\      /              |      |  +---+
        |CN2|--|      |    \    SDWAN Hybrid    |      |--|CN1|
        +---+  |      |     \ /                 |      |  +---+
               +------+     /\                  +------+
                           /  \
               +------+   /    \                +------+
        +---+  |      |--/    SDWAN Hybrid      |      |  +---+
        |CN2|--|CE-PE |          \------------- |CE-PE4|--|CN1|
        +---+  |      |                         |      |  +---+
        +---+  |      |                         |      |  +---+
        |CN1|--|      |                         |      |--|CN2|
        +---+  |      |------SDWAN hybrid-------|      |  +---+
               +------+                         +------+
                    \                                /
                     \                              /
                      \ BGP Peer  +-+--+ BGP Peer  /
                       +----------| RR |----------+
                                  +----+

          Please note that RR may be one RR, but for clarity of diagram
              the RR has been displayed as two parts.
              The BGP connections to the RR are over secure links or
              secure transports.

                      Figure 2: Hybrid SD-WAN BGP Mesh

   Note that the Figure 1 SD-WAN node BGP peers (C-PE) are connected in
   the underlay by both trusted VPNs and untrusted public networks.  For
   trusted VPNs, IPsec Security associations may not be set-up.  In some
   circumstances, some SD-WAN node peers may be connected only by
   untrusted public networks.  For the traffic over untrusted networks,
   IPsec Security Associations (IPsec SA) must be established and
   maintained.  If an edge node has network ports behind a NAT, the NAT
   properties need to be discovered by the authorized SD-WAN peers.







Dunbar, et al.            Expires 23 July 2025                 [Page 10]

Internet-Draft            SDWAN Edge Discovery              January 2025


2.2.2.  SD-WAN Secure Links Example Topology

   Suppose the SD-WAN network topology from Figure 1 removes the L3VPN
   links.  Instead the links are simply a combination of L3 WAN Networks
   (unsecured) and physically secure direct L2 and L3 links.  The
   topology in Figure 1 becomes the underlay physical topology in
   Figure 3.  The unsecured links need IPSec encryptions so the IPsec
   links must be configured.  An SD-WAN Hybrid tunnel allows connections
   between the C-PEs (C-PE1, C-PE2, C-PE3, and C-PE) to be a single
   logical link.

   Therefore, the logical topology in Figure 3, reduces to the SD-WAN
   logical topology shown in Figure 2.

  Hybrid SD-WAN Secure Links:  Sample Topology




































Dunbar, et al.            Expires 23 July 2025                 [Page 11]

Internet-Draft            SDWAN Edge Discovery              January 2025


       BGP over secure link       +---+  BGP over secure link
                     +------------|RR |---------+
                    / BGP Peer    +-+-+ BGP Peer \
                   /                              \
                  /                                \
          +---------+                              +-------+
   +---+  | C-PE1   |--P1--( WAN L3 Net1)-------P1-| C-PE2 | +---+
   |CN3|--|         |--P2--( L3 Direct Link)----P2-|       |-|CN3|
   +---+  |         |                              |       |
   +---+  |         |                              |       | +---+
   |CN2|--| Addr    |--P3 --( WAN L3 Net3)------P3-|       |-|CN1|
   +---+  |         |--P4---(L2 direct link)----P4-|       | +---+
      |   +---------+                              +-------+    |
      \                                                         /
       \  +---------+                              +-------+  /
        +-|         |                              |       |-+
   +---+  | C-PE3   |--P1--( WAN L3 Net1 )------P1-| C-PE4 | +---+
   |CN2|--|         |--P2--( L3 Direct Link )---P2-|       |-|CN1|
   +---+  |         |                              |       | +---+
   +---+  |         |                              |       | +---+
   |CN1|--| Addr    |--P3 --( WAN L3 Net3)------P3-|       |-|CN2|
   +---+  |         |--P4---(L2 direct link)----P4-|       | +---+
          +---------+                              +-------+
                 \                                 /
                  \                               /
                   \ BGP Peer    +-+-+ BGP Peer  /
                    +------------|RR |----------+
       BGP over secure link      +---+  BGP over secure link

           Please note that RR may be one RR, but for clarity of diagram
           the RR has been displayed as two parts.

                 Figure 3: Hybrid SD-WAN Secure Links

2.2.3.  RR connections to BGP Peers in SD-WAN

   For both the SD-WAN Secure L3VPN network and the SD-WAN Secure Links,
   the BGP Peers are assumed to be connected to a Route Reflector via
   secure link.  For an SD-WAN deployment with multiple RRs, it is
   assumed that there are secure connections among those RRs.

   How secure connections are established between the BGP Peer and the
   RR, or between the multiple RRs is out of the scope of this document.
   The existing BGP UPDATE propagation mechanisms control the edge
   properties propagation among the RRs.






Dunbar, et al.            Expires 23 July 2025                 [Page 12]

Internet-Draft            SDWAN Edge Discovery              January 2025


   For some environments where the communication to RR is highly
   secured, [RFC9016] IKE-less can be deployed to simplify IPsec SA
   establishment among edge nodes.

2.3.  The Objectives of SD-WAN Edge Discovery

   The objectives of SD-WAN edge discovery for an SD-WAN edge node are
   to discover its authorized BGP peers and each peer's associated
   properties in order to establish secure overlay tunnels [Net2Cloud].
   The attributes to be propagated for the SD-WAN Secure L3VPN case are:

   *  the SD-WAN (client) VPN information,

   *  the attached client routes per VPN, and

   *  the properties of the underlay networks over which the client
      routes can be carried.

   Like any VPN networks, the attached client routes belonging to
   specific SD-WAN VPNs can only be exchanged with the SD-WAN peer nodes
   authorized to communicate.

   The attributes to be propagated for the SD-WAN Secure L3 Links are:

   *  the attached client routes and

   *  the properties of the underlay networks over which the client
      routes can be carried.

   The properities of the underlay network may include the following:
   IPsec-SA information, information needed for NAT transversal with
   IPsec, and link speeds.

   Some SD-WAN peers are connected by both trusted VPNs and untrusted
   public networks.  Some SD-WAN peers are connected only by untrusted
   public networks.  For the traffic over untrusted networks, IPsec
   Security Associations (IPsec SA) must be established and maintained.
   For the trusted VPNs, IPsec Security associations may not be set-up.
   If an edge node has network ports behind a NAT, the NAT properties
   need to be discovered by the authorized SD-WAN peers.











Dunbar, et al.            Expires 23 July 2025                 [Page 13]

Internet-Draft            SDWAN Edge Discovery              January 2025


2.4.  Examples of SD-WAN Edge Discovery

   Suppose the environment is Figure 1 with the logical SD-WAN Hybrid
   link topology of Figure 2.  These topologies are set-up via
   configuration with IPsec-SA pre-configured.  Suppose that C-PE1 and
   C-PE2 have 10 pre-shared keys to use on IPsec links.  Currently P3 is
   using IPSec-SA ID-1.  C-PE1 wants to receive traffic flowing from
   C-PE2 over the Hybrid SD-WAN links.

   Step 1: Send Client Route with TEA to RR:  Refering to the Figure 2,
      the C-PE1 routers send customer routes (L3VPN v4 route) to the RR
      with

      *  NextHop set to self (C-PE1), and

      *  attaching a Tunnel Encapsulation attribute specifying a Hybrid
         SD-WAN tunnel to the RR over a secure connection

   Step 2: RR forwards the Client route to C-PEs:  Based on RR policy,
      the RR forwards routes to other BGP peers that support SD-WAN
      discovery for that AFI/SAFI (L3VPN v4).  Using the Figure 2
      example with the L3VPN for v4, RR would forward client routes with
      TEA of SD-WAN tunnel TLV and Next-Hop of C-PE1.

   Step 3: Remote C-PE checks link:  Prior to CE-P2 installing the L3VPN
      Unicast Routes (1/128), the C-PE2 MUST verify at least one of the
      underlay connections exist between C-PE1 and C-PE2.  Local policy
      will define which links (or links) the traffic for this route goes
      over between C-PE1 and C-PE2.

   Suppose for some reason the L2 link between C-PE1 and C-PE2 has
   encounter attacks, and the IPsec encryption must now run on links P3
   and P4.  C-PE1 detects the problem and to change the encryption on P3
   and add encryption on P4.  C-PE1 and C-PE2 must agree upon the next
   encryption on the list, and will send the IPsec-SA information in-
   band via BGP.

   Step 1: CE-P1 Send Underlay Route with TEA to RR:  C-PE1 sends two
      SD-WAN NLRIs in an BGP Update with a TEA with SD-WAN Hybrid Tunnel
      TLV with an IPsec-SA-ID TLV with 4 IPsec SA Identifiers (4, 5, 6,
      7).  The first SD-WAN NLRI contains Port P3's information (Port-
      local-ID P3, SD-WAN Color Gold (1), and SD-WAN Node-ID CE-PE1).
      The second SD-WAN NLRI contains Port P4's information (Port-local-
      id P4, SD-WAN Color Gold (1), and SD-WAN Node-ID CE-PE1).

   Step 2: RR forwards Underlay route to C-PEs:  The RR forwards the 2
      underlay routes to C-PEs (C-PE1, C-PE2, C-PE3, C-PE4).




Dunbar, et al.            Expires 23 July 2025                 [Page 14]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Step 3: C-PE2 receives the route and processes IPsec-SA  After
      receiving C-PE1 update, C-PE2 process the update, and switches the
      IPsec-SA on Port 3 to IPsec-SA 4, and the IPsec-SA on P4 to IPsec-
      SA 4.

   This simple examples shows the value of rotating the pre-shared keys.
   Future IPsec SA can also be set-up, negotiated, or rekeyed in the
   same manner.

   The following question may occur to the observant reader:

   *  Why is IPsec related information passed on a different AFI/SAFI?

   *  Why do you do to support IPsec combined with NATs?

   *  Is this the best way to support NATs?

   The BGP SD-WAN Nodes can attach the TEA with a Hybrid SD-WAN TLV
   attached to the client routes (AFI/SAFI 1/1, 2/1, 1/128, 2/128) or
   attached to the SD-WAN NLRI (AFI/SAFI 1/74 or 2/74).  The SD-WAN NLRI
   allows the passing of IPsec information on a unique AFI/SAFI.  How
   implementation prioritize the processing of client routes versus
   underlay routes (SD-WAN NLRI) is an implementation matter, and out of
   scope for this document.

   For NATs, the Extended Port Sub-TLV (see section 3.4) represents the
   information the first deployment thought it needed to identify the
   NAT port for a IPsec Security Association.  The deployments of this
   SD-WAN technology found the amount of data gave enough information,
   but suggested future work might be able to send less information.
   However, those revisions are outside the scope of this document.

2.5.  Comparing SD-WAN Secure L3VPN with Pure IPsec VPN

   This section describes why this document chose to pass IPSec-SA
   information via a new BGP AFI/SAFI with a TEA [RFC9012] instead of
   using traditional IPsec mechanisms.

   A pure IPsec VPN has IPsec tunnels connecting all edge nodes over
   public networks.  Therefore, it requires stringent authentication and
   authorization (i.e., IKE Phase 1 [RFC7296]) before other properties
   of IPsec SA can be exchanged.  The IPsec Security Association (SA)
   between two untrusted nodes typically requires the following
   configurations and message exchanges:

      IPsec IKEV2:  Messages are sent to authenticate with each other.

      Establish IPsec SA:  Establishing the IPsec SA requires the



Dunbar, et al.            Expires 23 July 2025                 [Page 15]

Internet-Draft            SDWAN Edge Discovery              January 2025


         following set-up

         -  Local key configuration,

         -  Setting the Remote Peer address,

         -  Information from IKEv2 Proposal directly sent to peer
            (encryption method, integrity sha512, etc.), and

         -  Transform set.

      Attached client prefixes discovery achieved by:
         -  Running routing protocol within each IPsec SA.

         -  If multiple IPsec SAs between two peer nodes are established
            to achieve load sharing, each IPsec tunnel needs to run its
            own routing protocol to exchange client routes attached to
            the edges.

      Set-up Access List or Traffic Selector:  Access Lists permit
         specific Local-IP1, Remote-IP2, and other IPsec parameters.

   In a BGP-controlled SD-WAN network over hybrid MPLS VPN and public
   internet underlay networks, all edge nodes and RRs are already
   connected by private secure paths.  The RRs have the policies to
   manage the authentication of all peer nodes.  More importantly, when
   an edge node needs to establish multiple IPsec tunnels to many edge
   nodes, all the management information can be multiplexed into the
   secure management tunnel between RR and the edge node operating as a
   BGP peer.  Therefore, the amount of authentication in a BGP-
   Controlled SD-WAN network can be significantly reduced.

   Client VPNs are configured via VRFs, just like the configuration of
   the existing MPLS VPN.  The IPsec equivalent traffic selectors for
   local and remote routes are achieved by importing/exporting VPN Route
   Targets.  The binding of client routes to IPsec SA is dictated by
   policies.  As a result, the IPsec configuration for a BGP controlled
   SD-WAN (with mixed MPLS VPN) can be simplified in the following
   manner:

   *  The SD-WAN controller has the authority to authenticate edges and
      peers so the Remote Peer association is controlled by the SD-WAN
      Controller (RR).

   *  The IKEv2 proposals (including the IPsec Transform set) can be
      sent directly to peers, or incorporated in a BGP UPDATE.





Dunbar, et al.            Expires 23 July 2025                 [Page 16]

Internet-Draft            SDWAN Edge Discovery              January 2025


   *  The BGP UPDATE announces the client route reachability through the
      SDWAN hybrid tunnels.  A SDWAN hybrid tunnel combines several
      other tunnels into a single logical tunnel.  The SD-WAN Hybrid
      tunnel implementations insure that all tunnels within are either
      running over secure network links or secured by IPsec.

   *  Importing/exporting Route Targets under each client VPN (VRF)
      achieves the traffic selection (or permission) among clients'
      routes attached to multiple edge nodes.

   Note: The web of SDWAN hybrid tunnels in a network is denoted in this
   document as an SD-WAN underlay.  BGP passes information about the
   SDWAN hybrid tunnels between BGP peers by passing an SD-WAN Underlay
   NLRI paired with the tunnel encapsulation attribute (TEA) with an
   SDWAN tunnel type SD-WAN-Hybrid TLV.

   Also, note that with this method there is no need to run multiple
   routing protocols in each IPsec tunnel.

2.6.  SD-WAN Segmentation, Virtual Topology and Client VPN

   In SD-WAN Secure L3VPN deployments, SD-WAN Segmentation is a
   frequently used term which refers to partitioning a network into
   multiple subnetworks, just like MPLS VPNs.  SD-WAN Segmentation is
   achieved by creating SD-WAN virtual topologies and SD-WAN VPNs.

   An SD-WAN virtual topology consists of a set of edge nodes and the
   tunnels (a.k.a. underlay paths) interconnecting those edge nodes.
   These tunnels forming the underlay paths can be IPsec tunnels, or
   MPLS VPN tunnels, or other tunnels.  Figure 4 (top) illustrates an
   SD-WAN L3VPN underlay topology and Figure 4 (bottom) show the same
   topology as the virtual topology based on SD-WAN Links.

   An SD-WAN VPN is configured in the same way as the VRFs of an MPLS
   VPN.  One SD-WAN client VPN can be mapped to multiple SD-WAN virtual
   topologies.  A Route Target is used to differentiate between the SD-
   WAN VPNs.  For example, in figure 4 below, the Payment-Flow on C-PE2
   is only mapped to the virtual topology of C-PEs to/from Payment
   Gateway, whereas other flows can be mapped to a multipoint-to-
   multipoint virtual topology.

   SD-WAN Controller for the SD-WAN Secure L3VPN governs the policies of
   mapping a client VPN to SD-WAN virtual topologies.








Dunbar, et al.            Expires 23 July 2025                 [Page 17]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Each SD-WAN edge node may need to support multiple VPNs.  Route
   Target is used to differentiate the SD-WAN VPNs.  For example, in the
   picture below, the Payment-Flow on C-PE2 is only mapped to the
   virtual topology of C-PEs to/from Payment Gateway, whereas other
   flows can be mapped to a multipoint-to-multipoint virtual topology.














































Dunbar, et al.            Expires 23 July 2025                 [Page 18]

Internet-Draft            SDWAN Edge Discovery              January 2025


        BGP over secure link       +---+  BGP over secure link
                      +------------|RR |---------+
                     / BGP Peer    +-+-+ BGP Peer \
                    /                              \
                   /                                \
           +---------+                              +-------+
    +---+  | C-PE1   |--P1--( MPLS L3VPN Net1 )--P1-| C-PE2 | +---+
    |CN3|--|         |--P2--( MPLS L2VPN Net2 )--P2-|       |-|CN3|
    +---+  |         |                              |       |
    +---+  |         |                              |       | +---+
    |CN2|--| Addr    |--P3 --( WAN L3 Net3)------P3-|       |-|CN1|
    +---+  |         |--P4---(L2 direct link)----P4-|       | +---+
           +---------+                              +-------+
               |
               | P5 (L3 Direct link)
               |
               |
                           P1
               |
             +-------+
             |Payment|
                     |gateway|
             +-------+

           Tunnel exist C-PE2 port P4 on L2 direct link
           with encryption to P1 port on Payment gateway.


   Virtual topology
           +---------+                 +-------+
    +---+  | C-PE1   |-----------------+ C-PE2 |
    |CN3|--|         |                 |       |-|CN3|
    +---+  |         |                 |       |
           +---------+                 +-------+
                \                      /
                 \                    /
                  \                  /
                   \               /
                     \       +----+----+
                      +------| payment |
                             | Gateway |
                             +---------+

                 Figure 4: SD-WAN Virtual Topology and VPN







Dunbar, et al.            Expires 23 July 2025                 [Page 19]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.  BGP SD-WAN Mechanisms

   The BGP mechanisms have two functions:

   Pass Client routes with Hybrid SD-WAN tunnel:  A BGP peer supporting
      SD-WAN re-announces the routes passed by client routers with a
      next hop self and an BGP attribute indicating the SD-WAN Hybrid
      Tunnel.  Clients routes may be one of the following AFI/SAFIs:
      Unicast IPv4/IPv6(1/1, 2/1) and L3VPN IPv4/IPv6 (1/128, 2/128).
      The term "next hop self" means the routers sets the next Hop
      Address to an address indicating the BGP Peer.  The following two
      BGP Path attributes can pass the Tunnel indication: the
      Encapsulation Extended Community (Encap-EC) and the Tunnel
      Encapsulation attribute (TEA).

   Pass Underlay Routes (SD-WAN NLRIs) with Hybrid SD-WAN TEA:  A BGP
      peer sends the SD-WAN NLRI v4/v6 (AFI/SAFI 1/74 or 2/74) with Next
      Hop set to Next-Hop-self and an BGP attribute indicating the
      Hybrid Tunnel.  The SD-WAN NLRI identifies the port or (ports)
      within the Hybrid SD-WAN Tunnel that the BGP peer is sending
      encapsulation or IPsec-SA related information via the Hybrid SD-
      WAN TEA.  The Hybrid SD-WAN TEA contains the IPsec-SA and
      optionally NAT information.

   This section describes the Hybrid SD-WAN tunnel, the SD-WAN NLRIs,
   the new subTLVs for SD-WAN Tunnel IPSec-SA, subTLVs for Port
   attributes, the procedures for the client routes, the procedures for
   underlay routes, error handling, and considerations for managing SD-
   WAN technologies.

3.1.  SD-WAN-Hybrid Tunnel Encoding

  Name:  SD-WAN Hybrid Tunnel

  Code:  25 (IANA assigned)

  Description:  The SD-WAN-Hybrid tunnel identifies a hybrid tunnel
     that overlays a virtual path over a set of links between two BGP
     Peers comprised of various technologies (e.g.  MPLS, L2 direct
     link, or L3 through Public Internet, and others).  These links
     between the two BGP peers are denoted as underlay links.  The
     underlay links may or may not need encryption.  If some of these
     underlay links need encryption, the SD-WAN Hybrid Tunnel provides
     a mechanism to pass this information via BGP.  Passage of the
     IPsec-SA information is optional.

  Encoding:  Per [RFC9012], the following two BGP attributes that may




Dunbar, et al.            Expires 23 July 2025                 [Page 20]

Internet-Draft            SDWAN Edge Discovery              January 2025


  encode a Tunnel Encapsulation attribute information: the Tunnel
  Encapsulation Attribute, and the Encapsulation Extended Community
  (Encap-EC) as a "barebones" tunnel identification.  The encoding
  for the SD-WAN Hybrid tunnel is described for both BGP attributes.

  SD-WAN Encoding in Encap-EC

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | 0x03 (1 octet)| 0x0c (1 octet)|       Reserved (2 octets)     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |     Reserved (2 octets)       |    Tunnel Type=25  (2 octets) |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Encapsulation Extended Community


        Figure 5: SD-WAN Hybrid tunnel encoding in Encap-EC

  The NextHop Field in the BGP update is the tunnel egress
  Endpoint, and this should be set to the BGP Peer Address for
  the SD-WAN Peer.

  SD-WAN Encoding in TEA

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Tunnel-Type=25(SD-WAN-Hybrid )| Length (2 Octets)             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Sub-TLVs                            |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


                 Figure 6: SD-WAN Hybrid Value Field

  Valid SubTLVs for Encap-EC form of Hybrid SD-WAN Tunnel:  The Encap-
     EC format (barebones) of the tunnel encapsulation attribute cannot
     attach any subTLVs.

  Valid SubTLVs for the TEA form of the Hybrid SD-WAN Tunnel:  The
     valid SubTLV for the TEA form of the Hybrid SD-WAN Tunnel TLV
     depends whether the TEA is attached to a client route or an
     underlay route.  Table 1 provides a list of valid subTLVs per NLRI
     type (client or underlay).  The valid subTLVs specified [RFC9012]
     are Color (3) and Tunnel Egress End Point (6).  The valid SubTLV



Dunbar, et al.            Expires 23 July 2025                 [Page 21]

Internet-Draft            SDWAN Edge Discovery              January 2025


     specified in this document are: Extended Port Attributes, IPsec
     SA-ID, IPSec SA Rekey, IPsec Public Key, IPsec SA Proposal SubTLV,
     and Simplified IPsec SA SubTLV.

  Validation Procedure:  The validation procedure for the SD-WAN tunnel
     TLV has the following components:

        1) validation of tunnel TLV encoding,

        2) Check that SubTLVs are valid for NLRI (see Table 1), and

        3) Egress tunnel End Point Check.

     Prior to installing a route with a SD-WAN tunnel as an active
     route, the BGP peer installing the route MUST also validate that
     the SD-WAN tunnel and underlay links are active.



































Dunbar, et al.            Expires 23 July 2025                 [Page 22]

Internet-Draft            SDWAN Edge Discovery              January 2025


                  Table 1

   Client Routes AFI/SAFI = 1/1, 2/1, 1/128, 2/128
   Underlay Routes AFI/SAFI = 1/74 and 2/74

   SubTLV            Code  Client Routes  Underlay Routes
   ------            ----  -------------  ---------------
   Encapsulation        1   not valid     not valid
   Protocol             2   not valid     not valid
   Color                3   valid *1      not valid *2
   Load-Balancing Bk    5   not valid     not valid *2
   Tunnel Egress EP     6   required      required
   DS Field             7   not valid     not valid *2
   UDP Dest. Port       8   not valid     not valid *2
   Embeded Label H.     9   not valid     not valid *2
   MPLS label Stack    10   not valid     not valid *2
   Prefix-SID          11   not valid     not valid *2
   Preference          12   not valid     not valid *2
   Binding SID         13   not valid     not valid *2
   ENLP                14   not valid     not valid *2
   Priority            15   not valid     not valid *2
   SPI/SI              16   not valid     not valid *2
   SRv6 Binding SID    20   not valid     not valid *2
   IPsec-SA-ID         64   valid *3      valid *3
   Extended Port       65   not valid     valid *4
   Underlay ISP        66   not valid     valid *4
   IPsec SA Nonce      67   valid *3      valid *3
   IPsec Public Key    68   valid *3      valid *3
   IPsec SA Proposal   69   valid *3      valid *3
   Simplified IPSec SA 70   valid *3      valid *3




                       Figure 7: Table 1: SubTLV list

   Notes

   *  *1 - For client traffic, the validation procedures for the Color
      subTLV follows the validation procedures of [RFC9012].  Precedence
      between Color Extended Community (Color-EC) and Color SubTLV is
      first Color-EC, and then Color SubTLV.

   *  *2 - In the future, the SD-WAN TEA could specify more details on
      Underlay links beyond protocol.  However, those future extensions
      are outside the scope of this document.





Dunbar, et al.            Expires 23 July 2025                 [Page 23]

Internet-Draft            SDWAN Edge Discovery              January 2025


   *  *3 - Encodings and validation of syntax are defined in section
      3.3.  Section 3.5 provides content processing and validation
      procedures for client routes, and section 3.6 provides content
      processing and validation procedures for underlay routes

   *  *4 - Encoding and mechanisms are defined 3.3.6.  Section 3.6
      provides error handling procedures in case of SubTLV being
      MALFORMED or included with the wrong NLRI.

3.2.  SD-WAN Underlay UPDATE

   The Edge BGP Peer using BGP SD-WAN discovery sends the hybrid SD-WAN
   NLRI with the SD-WAN Hybrid tunnel attribute to advertise the
   detailed properties associated with the public facing WAN ports and
   IPsec tunnels.  The Edge BGP Peer sends this information to its
   designated RR via the secure connection.  Each BGP Update message
   with a SD-WAN Underlay NLRI MUST contain a Tunnel Encapsulation
   Attribute (TEA) for a Hybrid Tunnel type.  The TEA can include
   SubTLVs for Extended Port attribute (see section 7) or IP Sec
   information (see section 8).  The IPsec information subTLVs include:
   IPsec-SA-ID, IPsec SA Nonce, IPsec Public Key, IPsec SA Proposal, and
   Simplified IPsec SA.

3.2.1.  The NLRI for SD-WAN Underlay Tunnel Update

   A new NLRI SAFI (SD-WAN SAFI=74) is introduced within the
   MP_REACH_NLRI Path Attribute of [RFC4760] for advertising the
   detailed properties of the SD-WAN tunnels terminated at the edge
   node:


   +------------------+
   |    Route Type    | 2 octets
   +------------------+
   |     Length       | 2 octets
   +------------------+
   |   Type Specific  |
   ~ Value (Variable) ~
   |                  |
   +------------------+

                       Figure 8: SD-WAN NLRI Encoding

   where:

   Route (NLRI) Type:  2 octet value to define the encoding of the rest
      of the SD-WAN the NLRI.  The SD-WAN Secure Links use case may have
      many different formats, the route type was assigned two octets.



Dunbar, et al.            Expires 23 July 2025                 [Page 24]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Length:  2 octets of length expressed in bits as defined in
      [RFC4760].


   This document defines the following SD-WAN Route type:

   NLRI Route-Type = 1:  For advertising the detailed properties of the
      SD-WAN tunnels terminated at the edge, where the transport network
      port can be uniquely identified by a tuple of three values (Port-
      Local-ID, SD-WAN-Color, SD-WAN-Node-ID).  The SD-WAN NLRI Route-
      Type =1 has the following encoding:


           +------------------+
           |  Route Type = 1  | 2 octets
           +------------------+
           |     Length       | 2 octets
           +------------------+
           |   Port-Local-ID  | 4 octets
           +------------------+
           |   SD-WAN-Color   | 4 octets
           +------------------+
           |  SD-WAN-Node-ID  | 4 or 16 octets
           +------------------+

                      Figure 9: SD-WAN NLRI Route Type 1

   Port-local-ID:  SD-WAN edge node Port identifier, which is locally
      significant.  If the SD-WAN NLRI applies to multiple WAN ports,
      this field is zero.

   SD-WAN-Color:  represents a group of tunnels, which correlate with
      the Color-Extended-community included in the client routes UPDATE.
      When a client route can be reached by multiple SD-WAN edges co-
      located at one site, the SD-WAN-Color can represent a group of
      tunnels terminated at those SD-WAN edges co-located at the site.
      If an SD-WAN-Color represents all the tunnels at a site, then the
      SD-WAN-Color effectively represents the site.

   SD-WAN Node ID:  The node's IPv4 or IPv6 address.  For IPv4 SD-WAN
      NLRI (1/74) the length is 4.  For IPv6 SD-WAN NLRI (1/76), the
      length is 16.

   Route Type values outside of 1 are out of scope for this document.







Dunbar, et al.            Expires 23 July 2025                 [Page 25]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.2.2.  Validation of SD-WAN NLRI

   The SD-WAN Node-ID field carries the Tunnel Endpoint.  The Tunnel
   Egress Endpoint in the Tunnel Encapsulation Attribute (TEA) is not
   used to validate the tunnel indicated by the SD-WAN NLRI.  The Next
   Hop within the UPDATE message MAY be tested by local policy for
   validity as a BGP Peer source, but the validation of the tunnel
   endpoint depends solely on the SD-WAN Node-ID.

3.2.3.  BGP Path Attributes attached to SD-WAN NLRI

   The BGP Path Attributes for the SD-WAN NLRIs which are required to be
   supported are:

   *  Origin, AS_PATH, Next_HOP, MULTI_EXIT_DISC, LOCAL_PREF
      ([RFC4271]),

   *  Originator_ID and Cluster-LIST ([RFC4456]),

   *  Tunnel Encapsulation Attribute ([RFC9012]),

   *  Extended Communities ([RFC4360]).

   Per [RFC9012], the Encapsulation Extended Community and the Color
   Extended Community must be supported.

   The following optional BGP attributes MAY be attached to an BGP
   UDPATE with the Hybrid SD-WAN NLRI in the MP-REACH-NLRI or
   MP_UNREACH_NLRI:

   *  Community ([RFC1997]),

   *  Large Community ([RFC8092]),

   *  IPv6 Address Specific Extended Community ([RFC5701]),

   *  AS4_Path and AS4_Aggregator ([RFC6793]) and

   These attributes are extensions of required AS_PATH and BGP Community
   support that may be used to set in local policy evaluations for the
   Hybrid SD-WAN Underlay.  However, the actions of local Policy
   regarding these BGP attributes is outside scope of this document.

   All other optional BGP attributes MAY also be attached to the NLRI,
   but the meaning of these attributes when attached to the NLRI is
   outside the scope of this document.





Dunbar, et al.            Expires 23 July 2025                 [Page 26]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.3.  IPsec SA Property Sub-TLVs

   This section describes the SubTLVs that pass data regarding IPsec
   parameters for the Hybrid SD-WAN tunnel.  During set-up of the Hybrid
   SD-WAN tunnels, the IPsec parameters need to be securely passed to
   set-up secure association.  For hybrid SD-WAN tunnels, the IPsec
   security association for IPsec links may change to different security
   associations over time.

   The subTLVs related to IPsec supported by the TEA TLV for the SD-WAN
   Hybrid Tunnel type are: IPSec-SA-ID, IPsec SA Nounce, IPsec Public
   Key, IPsec Proposal, and Simplified IPSec SA.  The IPSec-SA-ID SubTLV
   provides a way to indicate the IPsec SA Identifiers (section 3.3.1)
   for pre-configured security association.  The other four SubTLVs
   provide different ways to pass details regarding IPsec security
   associations.  The IPsec SA Nounce passes Nounce and rekey counters
   for a Secure Association identified by IPsec SA Identifier (see
   section 3.3.2).  The IPSec Public Key SubTLV passes IPsec Public Key
   data with a time duration (see section 3.3.3).  The IPsec Proposal
   SubTLV provides Transform attributes and Transform IDs (see section
   3.3.4).  The Simplified IP SEC SA passes the information that
   identifies configuration for 2 keys (see section 3.3.5).  The NAT-
   related portion infromation is carried in the Extended Port SubTLV
   (see section 3.3.6)

   If any SubTLV is MALFORMED, the implementation MUST follow the
   procedure in [RFC9012] in section 13.

   For a quick rotation between security associations, the SDWAN NLRI
   (port-id, color, node) can quickly distribute a switch to a set of
   new security association using the BGP Update message.  In this case,
   the BGP UPDATE message would like Figure 10


       SDWAN NLRI
              Route-type: 1
              length: 12
              port-id - 0.0.0.0
              SD-WAN-Color - 0.0.0.1
              node-id - 2.2.2.2
       TEA:
        Tunnel TLV: (type: SD-WAN Hybrid)
              Tunnel Egress Endpoint SubTLV: 2.2.2.2
              IPsec-SA-ID SubTLV: 20, 30

              Figure 10: SD-WAN NLRI IPsec rotation in attack





Dunbar, et al.            Expires 23 July 2025                 [Page 27]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.3.1.  IPsec-SA-ID Sub-TLV

   SubTLV Name:  IPsec-SA-ID - Send IPsec SA-IDs

   SubTLV Code:  64 (IANA assigned)

   SubTLV Description:  IPsec-SA-ID Sub-TLV within the Hybrid Underlay
      Tunnel UPDATE indicates one or more pre-established IPsec SAs by
      using their identifiers, instead of listing all the detailed
      attributes of the IPsec SAs.  Using an IPsec-SA-ID Sub-TLV not
      only greatly reduces the size of BGP UPDATE messages, but also
      allows the pairwise IPsec rekeying process to be performed
      independently.

   SubTLV Encoding:

       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |IPSec-SA-ID Sub|   Length      |  Reserved                     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPsec SA Identifier #1                   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPsec SA Identifier #2                   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      ~                                                               ~
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPsec SA Identifier #n                   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                        Figure 11: IPsec-SA-ID Sub-TLV

      where:

      *  IPSec-SA-ID Sub-Type (8 bits): 64(IANA Assigned).

      *  length (8 bits): Specifies the total length in octets of the
         value field (not including the Type and Length fields).  For
         the IPSec-SA-ID Sub-Type, the length should be the 2 + 4
         *(number of IPsec SA IDs) .

      *  Reserved: Reserved for future use.  In this version of the
         document, the Reserved field MUST be set to zero and MUST be
         ignored upon receipt.  Received values MUST be propagated
         without change.





Dunbar, et al.            Expires 23 July 2025                 [Page 28]

Internet-Draft            SDWAN Edge Discovery              January 2025


      *  Value field: The value field consists of a sequence of IPsec SA
         Identifiers, each 4 octets long.  As shown in figure above, n
         IPsec SAs are attached in the IPsec-SA-ID Sub-TLV:

         -  IPSec SA Identifier #1: A 4 octet identifier for a pre-
            established IP security association.

         -  IPSec SA Identifier #2: A 4 octet identifier for a pre-
            established IP security association.

         -  IPSec SA Identifier #n: A 4 octet identifier for a pre-
            established IP security association.

   SubTLV Error Handling:  A IPsec-SA-ID Sub-TLV is a MALFORMED if the
      fields do not fit the limits specified above.  Per [RFC9012] a
      MALFORMED Sub-TLV is ignored.  The procedures for content check
      for the IPsec-SA-ID Sub-TLV is specified in section 3.4 for client
      routes, and section 3.5 for underlay routes.

   Tunnel End Point Validation:  The IPsec-SA-ID Sub-TLV does not help
      validate the Tunnel Egress EndPoint.  However, the IPSec-SA
      provides the security information associated with set of routes
      sent from a peer which must be correctly handled upon reception of
      the data.

3.3.2.  IPsec SA Rekey Counter Sub-TLV

SubTLV Name:  IPsec SA ReKey Counter - Rekey Counter for a IPsec SA

SubTLV Code:  67 (IANA assigned)

SubTLV Description:  The IPsec SA Rekey Counter Sub-TLV provides the
   rekey counter for a security association (identified by IPsec SA
   Identifier).

SubTLV Encoding:  The encoding is shown in the figure below:















Dunbar, et al.            Expires 23 July 2025                 [Page 29]

Internet-Draft            SDWAN Edge Discovery              January 2025


        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |IPSec-SA-ID Sub|   Length      |  Reserved                     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   ID Length   |       Nonce Length            |I|   Flags     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                             Rekey                             |
       |                            Counter                            |
       +---------------------------------------------------------------+
       |                IPsec SA Identifier                            |
       +---------------------------------------------------------------+
       |                                                               |
       ~                          Nonce Data                           ~
       |                                                               |
       +---------------------------------------------------------------+

      Figure 12: IPsec SA Rekey Counter Sub-TLV Value Field

where:

   IPSec-SA-ID Sub-Type (8 bits): 67 (IANA assigned)

   length (8 bits): Specifies the total length in octets of the
   value field (not including the Type and Length fields).  The
   total length is variable with the value equal to 16 plus Nonce
   length.

   Reserved: Reserved for future use.  In this version of the
   document, the Reserved field MUST be set to zero and MUST be
   ignored upon receipt.  Received values MUST be propagated
   without change.

   ID Length (8 bits): indicates the length in octets of SA-
   Identifer.  This length should be 4 octets.

   Nonce Length (16 bits): indicates the length in octets of the
   Nonce Data.

   I Flag: when set to 1, the I-flag indicates that the
   communication being established is new. when set to 0, it
   signals that the communication is a continuation of an existing
   session.

   Flags (7 bits): Reserved for future use.  In this version of
   the document, the Reserved field MUST be set to zero and MUST
   be ignored upon receipt.  Received values MUST be propagated
   without change.



Dunbar, et al.            Expires 23 July 2025                 [Page 30]

Internet-Draft            SDWAN Edge Discovery              January 2025


   Rekey Counter (64 bits): the number of key updates or rekeys
   that have occurred.  Each time a key is rotated or replaced,
   the ReKey Counter is incremented.

   IPsec SA Identifier (IPSec-SA-ID): a specific IPsec SAs
   terminated at the edge.  The length of this field is specified
   in ID Length.  For this specification, the length is 4.  Other
   lengths are outside of the scope of this document.

   Nonce Data: a random or pseudo-random number for preventing
   replay attacks.  Its length is a multiple of 32 bits[RFC7296].

SubTLV Error Handling:  A IPsec SA Rekey Counter Sub-TLV s a
   MALFORMED if the fields do not fit the limits specified above.
   Per [RFC9012] a MALFORMED SubTLV is ignored.  The procedures for
   content checks for this Sub-tLV are specified in section 3.4 for
   client routes, and section 3.5 for underlay routes.  Please note
   that:

      The IPsec-SA-ID may also refer to the values carried in the
      same TEA in the same Tunnel TLV (type SD-WAN Hybrid) as the
      IPsec SA Rekey Counter Sub-TLV in either the IPsec Public Key
      Sub-TLV or the IPsec SA Proposal Sub-TLV.  The IPsec SA Rekey
      Counter, IPsec Public Key, and IPsec SA Proposal Sub-TLVs work
      together to create security associations.

      The IPsec-SA-ID may refer to an IPsec SA-ID in another SD-WAN
      Hybrid Tunnel TLV in the same TEA as the IPsec SA Rekey Counter
      sub-TLV,

      The IPsec SA-ID may refer to a preconfigured IPsec SA-ID
      preconfigured associated with this SD-WAN Hybrid Tunnel
      Endpoint.

Tunnel End Point Validation:  The IPsec SA Rekey SubTLV does not help
   validate the Tunnel Egress EndPoint.

3.3.3.  IPsec Public Key Sub-TLV

SubTLV Name:  IPsec Public Key - IPsec Public Key information

SubTLV Code:  68 (IANA assigned)

SubTLV Description:  The IPsec Public Key Sub-TLV provides the Public
   Key exchange information and the life span for the Diffie-Hellman
   Key.

SubTLV Encoding:  The encoding is shown in the figure below:



Dunbar, et al.            Expires 23 July 2025                 [Page 31]

Internet-Draft            SDWAN Edge Discovery              January 2025


        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |IPSec-SA-ID Sub|   Length      |  Reserved                     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   Diffie-Hellman Group Num    |          Reserved             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       ~                       Key Exchange Data                       ~
       |                                                               |
       +---------------------------------------------------------------+
       |                            Duration                           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Figure 13: IPsec SA Public Key Sub-TLV Value Field

where:

   IPSec-SA-ID Sub-Type (8 bits): 68 (IANA assigned)

   length (8 bits): Specifies the total length in octets of the
   value field (not including the Type and Length fields).  The
   total length is variable with the length being 14 + the Key
   Exchange Data length.

   Diffie-Hellman Group Num (16-bits): identifies the Diffie-
   Hellman group used to compute the Key Exchange Data.  Details
   on Diffie-Hellman group numbers can be found in Appendix B of
   IKEv2 [RFC7296] and [RFC5114].

   The Key Exchange data: This refers to a copy of the sender's
   Diffie-Hellman public value.  The length of the Diffie-Hellman
   public value is defined for MODP groups in [RFC 7296] and for
   ECP groups in [RFC 5903].

   Duration (32 bits): a 4-octet value specifying the life span of
   the Diffie-Hellman key in seconds.

SubTLV Error Handling:  A IPsec Public Key SubTLV is a MALFORMED
   SubTLV if the fields do not fit the limits specified above.  Per
   [RFC9012] a MALFORMED SubTLV is ignored.  The procedures for
   content checks for the IPsec Public is specified in section 3.4
   for client routes, and section 3.5 for underlay routes.

Tunnel End Point Validation:  The IPsec SA Rekey SubTLV does not help
   validate the Tunnel Egress EndPoint.





Dunbar, et al.            Expires 23 July 2025                 [Page 32]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.3.4.  IPsec SA Proposal Sub-TLV

   SubTLV Name:  IPsec SA Proposal - Indicates number of IPsec
      Transforms

   SubTLV Code:  69 (IANA assigned)

   SubTLV Description:  The IPsec SA Proposal Sub-TLV is to indicate the
      number of Transform Sub-TLVs.

   SubTLV Encoding:  The encoding is shown in the figure below:


       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |IPSec-SA-ID Sub|   Length      |       Reserved-Cnt            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |    Transform Attr Length      |Transform Type |    Reserved.  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |     Transform ID              |          Reserved             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      ~                        Transform Attributes                   ~
      |                                                               |
      +---------------------------------------------------------------+

               Figure 14: IPsec SA Proposal Sub-TLV Value Field

      where:

         IPSec-SA-ID Sub-Type (8 bits): 69 (IANA assigned)

         length (8 bits): Specifies the total length in octets of the
         value field (not including the Type and Length fields).  The
         total length is variable with the length being 10 + the
         Transform attribute length.

         Reserved-Cnt (16 bits): reserved for future use.  These bits
         are ignored upon receipt and set to zero when transmitted.

         Transform Attr Length (16 bits): indicates the length of the
         Transform Attributes field.

         Transform Type (8 bits): The Transform Type values are defined
         in Section 3.3.2 of [RFC7296] and [IKEV2IANA].  Only ENCR,
         INTEG, and ESN values are permitted.




Dunbar, et al.            Expires 23 July 2025                 [Page 33]

Internet-Draft            SDWAN Edge Discovery              January 2025


         Reserved (8 bits): reserved for future use.  These bits are
         ignored upon receipt and set to zero when transmitted.

         Transform ID (16 bits): An identifer for the transform defined
         by the associated transform attributes.

         Transform Attributes: These Sub-SubTLV are defined in section
         3.3.5 of [RFC7296].

   SubTLV Error Handling:  A IPsec SA Proposal Sub-TLV is a MALFORMED
      Sub-TLV if the fields do not fit the limits specified above.  Per
      [RFC9012] a MALFORMED Sub-TLV is ignored.  The procedure for
      content checks for the IPsec SA Proposal SubTLV is specified in
      section 3.4 for client routes, and section 3.5 for underlay
      routes.

   Tunnel End Point Validation:  The IPsec SA Rekey SubTLV not help
      validate the Tunnel Egress EndPoint.

   Compatibility:  The Reserved-Cnt field was allocated to allow future
      compatiability with [Secure-EVPN] field for SA-Proposals.
      Currently only 1 transform is allowed per IPsec SA Proposal
      SubTLV.  Multiple IPsec SA-Proposal SubTLVs may be included in a
      Tunnel-encapsulation Attribute.

3.3.5.  Simplified IPsec SA Sub-TLV

   SubTLV Name:  Simplified IPsec SA - Provides simple key for 2 keys

   SubTLV Code:  69 (IANA assigned)

   SubTLV Description:  For a simple SD-WAN network with edge nodes
      supporting only a few pre-defined encryption algorithms, a simple
      IPsec sub-TLV can be used to encode the pre-defined algorithms.

   SubTLV Encoding:  The encoding is shown in the figure below:















Dunbar, et al.            Expires 23 July 2025                 [Page 34]

Internet-Draft            SDWAN Edge Discovery              January 2025


        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |IPsec-simType  |IPsecSA Length |             Reserved          |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Transform     | Mode          | AH algorithms |ESP algorithms |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |         ReKey Counter (SPI)                                   |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | key1 length   |         Key 1                                 ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | key2 length   |         Key 2                                 ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | nonce length  |         Nonce                                 ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |        Duration                                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure 15: Simplified IPsec SA Sub-TLV

   where:

      IPSec-SA-ID Sub-Type (8 bits): 70

   Length (8 bits):  variable (based on key length)

   Reserved (16 bits):  Reserved for future use.  These bits SHOULD
      be set to zero on transmission and MUST be ignored on receipt.

   Transform (8 bits):
      *  Transform = 1 means AH,

      *  Transform = 2 means ESP, or

      *  Transform = 3 means AH+ESP.

      All other transform values are outside the scope of this
      document.

   IPsec Mode (8 bits):
      *  Mode = 1 indicates that the Tunnel Mode is used.

      *  Mode = 2 indicates that the Transport mode is used.

      All mode values besides 1 and 2 are outside the scope of this
      document.

   AH algorithms (8 bits):  AH authentication algorithms supported.




Dunbar, et al.            Expires 23 July 2025                 [Page 35]

Internet-Draft            SDWAN Edge Discovery              January 2025


      The values are specified by [IANA-AH].  Each SD-WAN edge node
      can support multiple authentication algorithms; send to its
      peers to negotiate the strongest one.

   ESP algorithms (8 bits):  the ESP algorithms supported.  Its
      values are specified by [IANA-ESP].  One SD-WAN edge node can
      support multiple ESP algorithms and send them to its peers to
      negotiate the strongest one.  The default algorithm is AES-256.

         When a node supports multiple authentication algorithms, the
         initial UPDATE needs to include the "Transform Sub-TLV"

   Rekey Counter (Security Parameter Index) (4 octet):  indicates the
      count for rekeying.

   key1 length (8 bits):  indicates the IPsec public key 1 length

   Public Key 1:  IPsec public key 1

   key2 length (8 bits):  indicates the IPsec public key 2 length

   Public Key 2:  IPsec public key 2

   none-length (8 bits):  indicates the Nonce key length

   Nonce:  IPsec Nonce

   Duration (32 bits):  specifying the security association (SA) life
      span in seconds.

   SubTLV Error Handling:  A Simplified IPsec SA Sub-TLV is a MALFORMED
      Sub-TLV if the fields do not fit the limits specified above.  Per
      [RFC9012] a MALFORMED Sub-TLV is ignored.  The procedures for
      content checks for the IPsec SA Proposal SubTLV is specified in
      section 3.4 for client routes, and section 3.5 for underlay
      routes.

   Tunnel End Point Validation:  Simplified IPsec SA does not
      participate in Tunnel End Point Validation.

3.3.6.  Extended Port Attribute Sub-TLV

   SubTLV Name:  Extended Port Attribute - Provides information related
      to IPsec and NATs

   SubTLV Code:  65 (IANA assigned)

   SubTLV Description:  The Extended Port Attribute Sub-TLV advertises



Dunbar, et al.            Expires 23 July 2025                 [Page 36]

Internet-Draft            SDWAN Edge Discovery              January 2025


      the properties associated with a public Internet-facing WAN port
      that might be behind NAT.  A SD-WAN edge node can query a STUN
      Server (Session Traversal of UDP through Network address
      translation [RFC8489]) to get the NAT properties, including the
      public IP address and the Public Port number, to pass to its
      peers.

         The location of a NAT device can be:

         *  Only the initiator is behind a NAT device.  Multiple
            initiators can be behind separate NAT devices.  Initiators
            can also connect to the responder through multiple NAT
            devices.

         *  Only the responder is behind a NAT device.

         *  Both the initiator and the responder are behind a NAT
            device.

         The initiator's address and/or responder's address can be
         dynamically assigned by an ISP or when their connection crosses
         a dynamic NAT device that allocates addresses from a dynamic
         address pool.

         As one SD-WAN edge can connect to multiple peers, the pair-wise
         NAT exchange as IPsec's IKE[RFC7296] is not efficient.  In the
         BGP Controlled SD-WAN, NAT properties for a WAN port are
         encoded in the Extended Port Attribute sub-TLV.

   SubTLV Encoding:  The encoding is shown in the figure below:





















Dunbar, et al.            Expires 23 July 2025                 [Page 37]

Internet-Draft            SDWAN Edge Discovery              January 2025


        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Type=65(extPort| ExtPort Length| Reserved      |I|O|R|R|R|R|R|R|
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | NAT Type      |  Encap-Type   |Trans networkID|     RD ID     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                  Local  IP Address                            |
       |        32-bits for IPv4, 128-bits for Ipv6                    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                  Local  Port                                  |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                Public IP                                      |
       |        32-bits for IPv4, 128-bits for Ipv6                    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                Public Port                                    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |               Extended SubSub-TLV                             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 16: Extended Port Attribute Sub-TLV

   where:

   *  Extended Port Attribute Type (=65): indicating it is the
      Extended Port Attribute SubTLV

   *  ExtPort Length: the length of the subTLV in octets (variable).
      If IPv4, the length is 32 (8 header, 32 address, 8 for 1
      subSubTLV).  If IPv6, the length is 64 (8 header, 48 addresses,
      8 for 1 subSubTLV).

   *  Flags:

      -  I bit (CPE port address or Inner address scheme):

         o  If set to 0, indicate the inner (private) address is
            IPv4.

         o  If set to 1, indicates the inner address is IPv6.

      -  O bit (Outer address scheme):

         o  If set to 0, indicate the inner (private) address is
            IPv4.

         o  If set to 1, indicates the inner address is IPv6.





Dunbar, et al.            Expires 23 July 2025                 [Page 38]

Internet-Draft            SDWAN Edge Discovery              January 2025


      -  R bits: reserved for future use.  Must be set to 0, and
         ignored upon reception.

   *  NAT Type: the NAT type can be one of the following values:

      -  1: without NAT ;

      -  2: 1-to-1 static NAT;

      -  3: Full Cone;

      -  4: Restricted Cone;

      -  5: Port Restricted Cone;

      -  6: Symmetric; or

      -  7: Unknown (i.e. no response from the STUN server).

      NAT type values outside of 1-7 are invalid for this SubTLV.

   *  Encap-Type: the supported encapsulation types for the port.

      -  Encap-Type=1: GRE;

      -  Encap-Type=2: VxLAN;

      Notes:

      -  The Encap-Type inside the Extended Port Attribute Sub-TLV is
         different from the RFC9012's BGP-Tunnel-Encapsulation type.
         The port can indicate the specific encapsulations, such as:

         o  If the IPsec-SA-ID subTLV or the IPsec SA detailed
            subTLVs (Nonce/publicKey/Proposal) are included in the
            SD-WAN-Hybrid tunnel, the Encap-Type indicates the
            encapsulation type within the IPsec payload.

         o  If the IPsec SA subTLVs are not included in the SD-WAN-
            Hybrid Tunnel, the Encap-Type indicates the encapsulation
            of the payload without IPsec encryption.

      -  Encapsulation types outside of GRE and VxLAN are outside of
         the scope of this specification.

      -  The Extended Port attribute SubTLV cannot support 6to4NAT
         encoding.




Dunbar, et al.            Expires 23 July 2025                 [Page 39]

Internet-Draft            SDWAN Edge Discovery              January 2025


   *  Transport Network ID: Central Controller assigns a global
      unique ID to each transport network.  Any value in this octet
      is valid

   *  RD ID: Routing Domain ID, need to be globally unique.  Any
      value in this octet is valid.

      -  Some SD-WAN deployment might have multiple levels, zones, or
         regions that are represented as logical domains.  Policies
         can govern if tunnels can be established across domains.
         For example, a hub node can establish tunnels with different
         logical domains but the spoke nodes cannot establish tunnels
         with nodes in different domains.

   *  Local IP: The local (or private) IP address of the WAN port.

   *  Local Port: used by Remote SD-WAN edge node for establishing
      IPsec to this specific port.

   *  Public IP: The IP address after the NAT.  If NAT is not used,
      this field is set to all-zeros

   *  Public Port: The Port after the NAT.  If NAT is not used, this
      field is set to all-zeros.

   *  Extended SubSub-TLV: for carrying additional information about
      the underlay networks.

   SubTLV Error Handling:  A IPsec SA Proposal Sub-TLV is a MALFORMED
      Sub-TLV if the fields do not fit the limits specified above.  Per
      [RFC9012] a MALFORMED Sub-TLV is ignored.  The procedures for
      content checks for the IPsec SA Proposal SubTLV is specified in
      section 3.4 for client routes, and section 3.5 for underlay
      routes.

   Tunnel End Point Validation:  Simplified IPsec SA does not
      participate in Tunnel End Point Validation.

3.3.6.1.  Extended SubSub-TLV

   One Extended SubSub-TLVs is specified in this document: Underlay
   Network Transport SubSub-TLV.

3.3.6.1.1.  Underlay Network Transport SubSub-TLV

   SubSubTLV Name:  Underlay Network Transport - caries WAN port types
      and speed




Dunbar, et al.            Expires 23 July 2025                 [Page 40]

Internet-Draft            SDWAN Edge Discovery              January 2025


   SubTLV Code:  66 (IANA Assigned)

   SubTLV Description:  The Underlay Network Transport SubSub-TLV is an
      optional SubSub-TLV to carry the WAN port connection types and
      bandwidth, such as LTE, DSL, Ethernet, and other types.  It is
      only valid for the Tunnel Type Hybrid SD-WAN in the Extended Port
      Attributes Sub-TLV.

   SubTLV Encoding:  The encoding is shown in the figure below:


        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | UnderlayType  |      Length   |      Reserved                 |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Connection Type|   Port Type   |        Port Speed             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure 17: Underlay Network SubSub-TLV

   Where:

   Underlay Network Properties:  sub Type=66

   Length:  always 6 bytes

   Reserved:  2 octet of reserved bits.  It SHOULD be set to zero on
      transmission and MUST be ignored on receipt.

   Connection Type:  are listed below as:

      *  1 = Wired

      *  2 = WIFI

      *  3 = LTE

      *  4 = 5G

      *  Any value outside of 1-4 is outside the scope of this
         specification.

   Port Type:  Port type define as follows:

      *  1 = Ethernet

      *  2 = Fiber Cable




Dunbar, et al.            Expires 23 July 2025                 [Page 41]

Internet-Draft            SDWAN Edge Discovery              January 2025


      *  3 = Coax Cable

      *  4 = Cellular

      *  Any value outside of values 1-4 are outside the scope of
         this specification.

   Port Speed:  The port seed is defined as 2 octet value.  The
      values are defined as Gigabit speed.  For example, a value of 1
      would mean 1 gigabit.  The port speed of "0" is not valid.

   The connection types of equipment and port types will continue to
   grow with technology change.  Future specifications may specify
   additional connection types or port types.

   SubSub-TLV Error Handling:  Underlay Network Transport SubSub-TLV is
      a MALFORMED SubSub-TLV if the fields do not fit the limits
      specified above.  If a MALFORMED SubSubTLV is contained in the
      Extended Port Attribute SubTLV, then the Extended Port Attribute
      SubTLV is MALFORMED.  Per [RFC9012] a MALFORMED Sub-TLV is
      ignored.  The procedures for content checks for the IPsec SA
      Proposal SubTLV is specified in section 3.4 for client routes, and
      section 3.5 for underlay routes.

   Tunnel End Point Validation:  Underlay Network Transport SubSub-TLV
      does NOT contribute to Tunnel End Point Validation.

3.4.  Procedure for Client Routes with Hybrid SD-WAN Tunnel

   The Tunnel Encapsulation Attribute for the SD-WAN Hybrid Tunnel Type
   may be associated with BGP UPDATE messages with NLRI with AFI/SAFI
   IPv4 Unicast (1/1), IPv6 (2/1), L3VPN v4 Unicast (1/128), and IPv6
   L3VPN (2/128).

   The SD-WAN Secure Links topology is supported by the Unicast IPv4/
   IPv6 prefixes.  The L3VPN topologies support forming the SD-WAN
   Secure L3VPN described in [SD-WAN-BGP-USAGE] and MEF ([MEF
   70.1][MEF70.2]).

   Based on [RFC9012], there are two forms a Tunnel Encapsulation
   Attribute (TEA) can take: "Barebones" using the Encapsulation
   Extended Community (Encap-EC) and a normal Tunnel Encapsulation form.









Dunbar, et al.            Expires 23 July 2025                 [Page 42]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.4.1.  SD-WAN Tunnel in Encapsulation Extended Community (Encap-EC)

   The SD-WAN Client routes are sent with a the Encapsulation Extended
   Community (Encap-EC) BGP attribute that identifies the the Hybrid SD-
   WAN tunnel type.  Per [RFC9012], the Encapsulation Extended Community
   uses the NextHop Field in the BGP UPDATE as the Tunnel Egress
   EndPoint.  The validation for the Tunnel Egress Endpoint uses the
   validation in sections 6, 8, and 13 applied to the NextHop.

   A Color Extended Community (Color-EC) or local policy applied to the
   client route directs the traffic for the client route to across
   appropriate interface within the Hybrid SD-WAN Tunnel to the Tunnel
   Egress Endpoint.

3.4.2.  SD-WAN Tunnel in Tunnel Encapsulation Path Attribute (TEA)

   The procedures for validating a client route with a TEA does the
   following:

   1.  Check for Well-formed SD-WAN Hybrid Tunnel TLV:  A well-formed Hybrid SD-WAN Tunnel TLV MUST have a Tunnel
         Engress Endpoint SubTLV.  The validation for the Tunnel Egress
         Endpoint uses the [RFC9012] validation in section 6, 8, and 13.
         An invalid Tunnel Egress Endpoint, cause the Hybrid SD-WAN
         Tunnel TLV to be invalid, and the TLV is ignored.

         It MAY also have any of the following SubTLVs:

         *  the Color SubTLV defined in [RFC9012],

         *  IPsec-SA-ID,

         *  IPsec SA Nonce,

         *  IPsec Public Key,

         *  IPSec SA Proposal, or

         *  Simplified IPsec SA)

         A MALFORMED SubTLV is ignored in the Tunnel TLV is ignored.

         SubTLV with an unknown type is ignored.

   2.  Check for multiple instances of SubTLVs:  Multiple instances of the Tunnel Endpoint SubTLV causes the
         first one to be used, and the subsequent instances to be
         ignored.

         Multiple instances of the IPsec Public Key, IPsec SA Proposal,



Dunbar, et al.            Expires 23 July 2025                 [Page 43]

Internet-Draft            SDWAN Edge Discovery              January 2025


         and Simplified IPsec SA cause the first instance to be used,
         and subsequent instances to be ignored.

         A Hybrid SD-WAN tunnel TLV may have multiple instances of the
         IPsec-SA-ID if the IPsec SA Identifiers are unique.  If all the
         IPsec SA Identifies are not unique, the second SubTLV is
         ignored and not propagated.

   3.  Validate Tunnel Egress Endpoint:  The Tunnel Egress Endpoint MUST
      link to the remote end of one of the underlay links to be used.
      This validation adheres to the [RFC9012] Tunnel Egress Endpoint
      validation.  The tunnel link may be active or inactive.

   4.  Validate each NLRI:  Local policy is run to validate routes.

   5.  Validate Hext Hop:  The next hop must be be reachable via the
      tunnel."

3.4.3.  Multiple tunnels attached to One Client Route

   A single SD-WAN client route may be attached to multiple SD-WAN
   Hybrid tunnels.  An Update with an SD-WAN client route may express
   these tunnels as an Encap-EC or a TEA.  Each of these tunnel
   descriptions is treated as a unique Hybrid SD-WAN tunnel with a
   unique Egress Endpoint.  Local Policy on the BGP Peer determines
   which tunnel the client data traffic will use.

3.4.4.  SD-WAN VPN ID in the Client Route Update

   An SD-WAN VPN ID is the same as a client VPN ID in a BGP controlled
   SD-WAN network.  The Route Target Extended Community should be
   included in a Client Route UPDATE message to differentiate the client
   routes from routes belonging to other VPNs.  Route Target value is
   taken as the VPN ID (for 1/1 and 2/1).  For 1/128 and 2/128, the RD
   from the NLRI identifies the VPN ID.  For EVPN, picking up the VPN-ID
   from EVPN SAFI.

3.4.5.  SD-WAN VPN ID in Data Plane

   SD-WAN edge node can be reached by either an MPLS path or an IPsec
   path within the hybrid SD-WAN tunnel.  If client packets are sent via
   a secure MPLS network within the Hybrid SD-WAN tunnel, then the data
   packets will have MPLS headers with the MPLS Labels based on the
   scheme specified by [RFC8277].  It is assumed the secure MPLS network
   assures the security outer MPLS Label header.






Dunbar, et al.            Expires 23 July 2025                 [Page 44]

Internet-Draft            SDWAN Edge Discovery              January 2025


   If the packets are sent via a link with IPsec outer encryption across
   a public network, the payload is still encrypted with GRE or VXLAN
   encryption.  For GRE Encapsulation within an IPsec tunnel, the GRE
   key field can be used to carry the SD-WAN VPN ID.  For network
   virtual overlay (VxLAN, GENEVE, etc.) encapsulation within the IPsec
   tunnel, the Virtual Network Identifier (VNI) field is used to carry
   the SD-WAN VPN ID.

3.5.  Procedure for Underlay Routes with Hybrid SD-WAN Tunnel TLV

3.5.1.  Hybrid SD-WAN NLRI with a Encapsulation Extended Community

   The Hybrid SD-WAN NLRI MUST be accompanied with the TEA, and MUST NOT
   be accompanied by an Encapsulation Extended Community.

3.5.2.  Underlay Route with a TEA

   An underlay routes contains Hybrid SD-WAN NLRIs with TEA attached.
   The procedures for processing underlay routes follows the following
   steps:

   1.  Check for Well-Formed SD-WAN Hybrid Tunnel TLV:  An Hybrid SD-WAN
      Tunnel TLV is well-formed using only SubTLVs valid for association
      with the underlay Route.

         A well-formed SD-WAN Hybrid Tunnel TLV MUST contain a valid
         Tunnel Egress Endpoint subTLV to be a valid SD-WAN Hybrid TLV.
         [RFC9012] provides validation guidelines for the Tunnel Egress
         Endpoint in sections 13.

         The SD-WAN Hybrid Tunnel TLV MAY contain the following subTLVs:
         Tunnel Egress, IPsec-SA-ID, Extended Port SubTLV, IPsec Nonce,
         IPsec Public Key, IPsec SA Proposal, and Simplified IPsec SA.

         Following [RFC9012] intent, a MALFORMED SubTLV is ignored, and
         a SubTLV with an unknown type is ignored.

   2.  Multiple instances of SubTLVs within a SD-WAN Tunnel TLV
      Multiple instances of the Tunnel Endpoint, IPsec Public Key, IPsec
      SA Proposal, and Simplified IPsec SA cause only the first one to
      be used.  Subsequent subTLVs are ignored and not propagated.  The
      IPsec-SA-ID subTLVs may have multiple instances of the subTLV if
      the IPsec SA Identifiers are unique, but if the IPsec SA
      Identifiers are not unique the second subTLV is ignored and not
      propagated.  If multiple Extended Port SubTLVs exist, the TLVs
      must be validated in step 4.

   3.  Validate Tunnel Egress Endpoint  The Tunnel Egress Endpoint MUST



Dunbar, et al.            Expires 23 July 2025                 [Page 45]

Internet-Draft            SDWAN Edge Discovery              January 2025


      link to remote end point of one of the underlay links from the
      router receiving the link.

   4.  Validate Extended Port SubTLV(s)  If a single Extended Port
      SubTLV exist, then validate that the port information provides
      needed information to establish a connection to the remote port.
      A SD-WAN edge node MAY utilize local policy, local configuration,
      and the information from the SubTLV.  The exact mechanism of local
      port validation is outside the scope of this document.  If the
      Extended Port SubTLV is invalid, the SD-WAN tunnel TLV must be
      considered to be invalid.

   5.  Validate each NLRI  The typed NLRI in the SD-WAN Underlay MUST be
      Well-Formed having the format specified in section 3.2.1.  A
      MALFORMED NLRI must cause the NLRI to be discarded.  An
      implementation MAY log an error for a MALFORMED NLRI.  The local
      policy configuration in the BGP peer receiving this NLRI MUST
      determine the validity of the route based on policy.

   6.  Validate Next Hop  The IP address specified in the Next Hop field
      must be reachable by the Tunnels.

3.5.3.  Underlay Routes with Port-Local-ID of zero

   Section 3.2.1 specifies that Route Type 1 has a tuple of (Port-Local-
   ID, SD-WAN-Color, SD-WAN-Node-ID).  Port-Local-ID may be zero if the
   NLRI applies to multiple ports.  The BGP Peer receiving the NLRI must
   have pre-configured inbound filters to set the preference for the SD-
   WAN NLRI tuple.

   Since a Port-Local-ID value of zero indicates the NLRI applies to
   multiple ports, it is possible to have the following NLRI within a
   packet (or received in multiple packets):

      Port-Local-ID (0), SD-WAN-Color (10), SD-WAN-Node-ID (2.2.2.2),

      Port-Local-ID (0), SD-WAN-Color (20), SD-WAN-Node-ID (2.2.2.2),
      and

      Port-Local-ID (0), SD-WAN-Color (30), SD-WAN-Node-ID (2.2.2.2).

   These NLRI may simply indicate that there are three groups of tunnels
   for SD-WAN-Node-ID (2.2.2.2) assigned three colors.  For example,
   these tunnels could represent three types of gold, silver and bronze
   network service.






Dunbar, et al.            Expires 23 July 2025                 [Page 46]

Internet-Draft            SDWAN Edge Discovery              January 2025


3.5.4.  Multiple Tunnels attached to One Underlay Route

   An underlay route (SD-WAN NLRI) may only attach to one Hybrid SD-WAN
   Tunnel.  If there are more than one Hybrid SD-WAN Tunnel TLV within a
   single TEA, the first is processed and the subsequent Hybrid SD-WAN
   Tunnel TLVs are ignored.

3.6.  Error handling

   The Error handling for SD-WAN VPN support has two components: error
   handling for Tunnel Encapsulation signaling (Encap-EC and TEA) and
   the SD-WAN NLRI.  An SD-WAN NLRI, a Tunnel Encapsulation attribute
   MUST always accompany the SD-WAN NLRI.

   The previous sections (3.4 and 3.5) provide the normal procedures for
   handling client routes and undelay routes.

3.6.1.  Error handling for the Tunnel Encapsulation Signaling

   The error handling for the tunnel encapsulation signaling (Encap-EC
   and TEA) adheres to the error handling and validation specified by
   [RFC9012].

   The Tunnel encapsulation signaled with the client routes indicates
   the Egress endpoint via Next Hop in the Encap-EC or the TEA SubTLV
   for Tunnel Egress Endpoints.  As indicated in the procedure in
   sections 3.4.2 and 3.5.2, the SD-WAN Hybrid tunnel follows the
   validation section 6, 8, and 13 from [RFC9012].

   The SD-WAN client routes associate some of the NLRIs that [RFC9012]
   associates with the Encap-EC and the TEA using the validation
   specified in [RFC9012] in sections 6, 8, and 13.  When the SD-WAN
   Hybrid Tunnel is associated with the SD-WAN NLRI, and all RFC9012
   validation rules in section 6, 8, and 13 are extended to apply to the
   SD-WAN NLRI.

   [RFC9012] contains the necessary detail to specify validation for the
   new SubTLVs present for the SD-WAN Tunnel type.  However, to aid
   users of this document the following recap of validation of [RFC9012]
   is provided below.  The validation from section 13 of [RFC9012]
   includes:

   *  Invalid tunnel type must be treated if the TLV was not present.

   *  A malformed subTLVs must be treated as an unrecognized subTLV
      except for Tunnel Egress Endpoint.  If Tunnel Egress Endpoint is
      malformed, the entire TLV must be ignored.




Dunbar, et al.            Expires 23 July 2025                 [Page 47]

Internet-Draft            SDWAN Edge Discovery              January 2025


   *  Multiple incidents of Tunnel Egress Endpoint SubTLV cause the
      first incident of these subTLVs to be utilized.  Subsequent TLVs
      after the first one per type are ignored (per RFC9012), but
      propagated.

   *  If a subTLV is meaningless for a tunnel type, the subTLV is
      ignored, but the subTLV is not considered malformed or removed
      from the Tunnel Attribute propagated with the NLRI.

   For SD-WAN client routes with a TEA with a SD-WAN Hybrid Tunnel type
   TLV, the IPSec SubTLVs (IPsec SA-ID, IPSec nonce, IPSec Public Key,
   IPsec Proposal, and Simplified IPsec SA) are meaningful, but may be
   rarely sent.  Incorrect fields within any of these 5 TLVs.  Per
   [RFC9012], a malformed subTLV is treated as an unrecognized subTLV.

   For SD-WAN NLRI underlay routes, the Extended Port subTLV and the
   IPSec SubTLVs (IPsec SA-ID, IPSec nonce, IPSec Public Key, IPsec
   Proposal, and Simplified IPsec SA) are valid and meaningful.
   Incorrect fields within any of these 6 TLVs or subSubTLVs within the
   TLVs should cause the subTLV to be treated as malformed SubTLV.  Per
   [RFC9012], a malformed subTLV is treated as an unrecognized subTLV.

   If multiple instances of the IPsec nonce, IPsec Public Key, IPsec
   Proposal, and Simplified IPsec are received within a SD-WAN Tunnel
   TLV , only the first is processed.  The second instance is ignored
   and not propagated.  The IPsec SA-ID may have multiple copies, but
   the IPsec SA Identifiers sent in the second SubTLV must be different
   than any in the first IPsec-SA ID SubTLV.

   If multiple instances of the Extended Port subTLV are received, the
   local policy must determine which is to be used.

3.6.2.  Error Handling for NLRI

   The SD-WAN NLRI [AFI 1/SAFI = 74] utilizes a route type field to
   describe the format of the NLRI.  This specification only allows an
   NLRI with a type value of 1.  An NLRI with a type of field of another
   value is ignored and not processed.  The implementation MAY log an
   error upon the reception of a type value outside of Route Type 1.
   Error handling for the SD-WAN NLRI also adheres to the BGP Update
   error handling specified in [RFC7606].

   The local policy configuration in the BGP peer receiving this NLRI
   must determine the validity of the route based on policy.  Local
   configuration and policy must carefully constrain the SD-WAN-NLRI,
   tunnels, and IPsec security associations to create a "walled garden".





Dunbar, et al.            Expires 23 July 2025                 [Page 48]

Internet-Draft            SDWAN Edge Discovery              January 2025


   In the future, other proposals for a SD-WAN NLRI may specify a
   different route type.  Those proposals must specify the following:

      validation for new Route Type in the SD-WAN-NLRI, and

      how the new Route Type interacts with the Route Type 1.

3.6.3.  SDWAN NLRI and Tunnel Encapsulation Attribute

   The SD-WAN NLRI (AFI=1/SAFI=74) MUST be paired with Tunnel
   Encapsulation attribute with a tunnel TLV for tunnel type SD-WAN-
   Hybrid.  If the SD-WAN NLRI exist in an BGP UPDATE without a Tunnel
   Encapsulation Attribute (TEA) with a tunnel TLV for tunnel type SD-
   WAN-Hybrid, the NLRI is considered malformed and Treat-as-withdraw
   approach (per RFC7606).

   The SD-WAN NLRI SHOULD not be paired with an Encapsulation Extended
   Community.  If an SD-WAN NLRI is paried with an Encapsulation
   Extended Community rather than a Tunnel Encapsulation Attribute, the
   SD-WAN NLRI is considered malformed and the Treat-as-withdraw
   approach (per [RFC7606]) SHOULD be used.

4.  Manageability Considerations

   Unlike MPLS VPN whose PE nodes are all controlled by the network
   operators, SD-WAN edge nodes can be installed anywhere, in shopping
   malls, in 3rd party Cloud DCs, etc.

   It is very important to ensure that client routes advertisement from
   an SD-WAN edge node are legitimate.  The RR needs to ensure the SD-
   WAN Hybrid Tunnels and routes run over the appropriate Security
   associations.

4.1.  Detecting Misaligned Tunnels

   It is critical that the Hybrid SD-WAN Tunnel correctly forward
   traffic based on the local policy on the client routes, the tunnel
   egress and tunnel ingress, and the security association.  The RR
   reflector and the BGP peer must check that the client routes, tunnel
   egress, tunnel ingress, and security associations align with expected
   values for a tunnel.










Dunbar, et al.            Expires 23 July 2025                 [Page 49]

Internet-Draft            SDWAN Edge Discovery              January 2025


4.2.  IPsec Attributes Mismatch

   Each BGP peer (e.g. a C-PE) advertises a SD-WAN SAFI Underlay NLRI to
   the other BGP peers via a BGP Route Reflector to establish pairwise
   IPsec Security Associations (SA) between itself and other remote BGP
   Peers.  During the SD-WAN SAFI NLRI advertisement, the BGP Peer
   originating may pass information about security association in one of
   three forms:

   *  an identifier for a pre-configured and established IPsec Security
      Association,

   *  a simplified set of security parameters for setting up a IPsec
      Security association (Transform, IPsec Mode, AH and ESP
      Algorithms, rekey counter, 2 public keys, nonce, and duration of
      security association), or

   *  a flexible set of security parameters where Nonce, Public Key, and
      SA Proposal are uniquely specified.

   For existing IPsec Security associations, the receiving BGP peer can
   simply utilize one of these existing security associations to pass
   data.  If multiple IPsec associations are pre-configured, the local
   policy on the SD-WAN Edge Node may help select which security
   association is chosen for the SD-WAN Hybrid Tunnel.

   If the receiving and originating BGP peer engage in a set-up for the
   IPsec security associations for the link within the SD-WAN Hybrid
   tunnel, IPsec mechanisms require that there are matching IPsec
   transforms.  Without common IPsec transforms, the IPsec set-up
   process cannot operate.

4.2.1.  SD-WAN Hybrid Tunnel Mechanisms for Passing IPSec Security
        Association Info

   The TEA passes in the Tunnel TLV for the SD-WAN Hybrid Tunnel these
   three sets of information in the following subTLVs:

   IPsec-SA-ID:  passes the previous configured (pre-configured or
      generated) IPsec SA identifiers.

   Simplified IPsec SA SubTLV:  specifies a simplified set of
      information upon which to set-up the IPsec security associations
      for the tunnel.

   A sequence of the following SubTLVs:  IPsec SA Rekey Counter SubTLV,





Dunbar, et al.            Expires 23 July 2025                 [Page 50]

Internet-Draft            SDWAN Edge Discovery              January 2025


      IPsec Public Key SubTLV, and a IPSec Proposal SubTLV.
      Configuration on the local node uses this information and any
      information in the underlay to create security associations.

   Each BGP Peer needs to send the IPSec SA attributes received on the
   SD-WAN NLRI in the TEA between the local and remote WAN ports.  If
   there is a match on the SA Attributes between the two ports, the
   IPSec Tunnel is established.  If there is a mismtach on the SA
   Attributes, no IPsec Tunnel is established.

   The C-PE devices do not try to negotiate the base IPSec-SA parameters
   between the local and the remote ports in the case of simple IPSec SA
   exchange or the Transform sets between local and remote ports.  If
   there is a mismatch in the IPsec SA, then no IPsec Tunnel is created.
   If there is a mismatch on the Transform sets in the case of full-set
   of IPSec SA Sub-TLVs, no tunnel is created.

4.2.2.  Example creation of IPsec Security Association over SD-WAN
        Hybrid tunnel

   This section provides one example of how IPsec Security associations
   are created over the SD-WAN Hybrid tunnel.  Figure 1 in Section 3
   shows an establish an IPsec Tunnel being created between C-PE1 and
   C-PE2 WAN Ports A2 and B2 (A2: 192.168.1.10 - B2:192.168.2.1).

   To create this tunnel C-PE1 needs to advertise the following
   attributes for establishing the IPsec SA:

   *  NextHop: 192.168.1.10

   *  SD-WAN Node ID: 1.1.1.1

   *  SD-WAN-Site-ID: 15.0.0.2

   *  Tunnel Encap Attr (Type=SD-WAN) -

      -  Extended Port Attribute Sub-TLV containing

         o  Transport SubSubTLV - with information on ISP3.

      -  IPSec information for detailed information about the ISP3

      -  IPsec SA Rekey Counter Sub-TLV,

      -  IPsec SA Public Key Sub-TLV,

      -  Proposal Sub-TLV (type = ENCR, transform ID = 1)




Dunbar, et al.            Expires 23 July 2025                 [Page 51]

Internet-Draft            SDWAN Edge Discovery              January 2025


         o  type: ENCR

         o  Transform ID: 1

         o  Tranform attributes = trans 1 [from RFC7296]

   C-PE2 needs to advertise the following attributes for establishing
   IPsec SA:

   Next Hop:  192.168.2.1

   SD-WAN Node ID:  2.2.2.2

   SD-WAN-Site-ID:  1500

   Tunnel Encap Attr (Type=SD-WAN)
      *  Extended Port Attribute SubTLV

         -  Transport SubSubTLV - with information on ISP3.

      *  IPsec SA Rekey Counter Sub-TLV,

      *  IPsec SA Public Key Sub-TLV,

      *  IPSec Proposal Sub-TLV with

         -  transform type: ENCR

         -  Transform ID = 1

         -  Transform attributes = trans 2

   As there is no matching transform between the WAN ports A2 and B2 in
   C-PE1 and C-PE2, respectively, no IPsec Tunnel will be established.

5.  Security Considerations

   The document describes the encoding for SD-WAN edge nodes to
   advertise its properties to their peers to its RR, which propagates
   to the intended peers via a secure link across an untrusted network.

   The secure propagation is achieved by secure channels, such as TLS,
   SSL, or IPsec, between the SD-WAN edge nodes and the local controller
   RR.  It is assumed that the SD-WAN edges communication will result in
   aligned IPsec Attributes.  Please see section 4.2 for a discussion of
   what happens if the IPsec Attributes are mismatched.





Dunbar, et al.            Expires 23 July 2025                 [Page 52]

Internet-Draft            SDWAN Edge Discovery              January 2025


   SD-WAN edge nodes MUST have a secure transport connection to the RR.
   This secure BGP connection can be established as BGP using TCP over
   IPsec, SSL or TLS.

   In a walled garden SD-WAN deployment where all SD-WAN edges and the
   central controller are under one administrative control and the
   network operates within a closed environment, the threat model is
   primarily on internal threats, misconfigurations, and localized
   physical risks.  Unauthorized physical access to SD-WAN edge devices
   in remote locations is a concern.  Such access might allow attackers
   to compromise the edge devices and potentially manipulate the
   advertised Client prefixes with VPN IDs (or Route Targets) that do
   not belong to them.  This can lead to unauthorized data interception
   and traffic redirection.

   Ensuring secure communication between SD-WAN edge nodes and the
   central controller within a walled garden deployment is crucial.  It
   is essential to utilize secure communication channels such as TLS,
   SSL or TCP over IPsec for all communications between edge nodes and
   the controller.

   Therefore, it is necessary to ensure physical security controls are
   in place at remote locations, including locks, surveillance, and
   access controls.  Additionally, the RR needs to verify the BGP
   advertisements from each SD-WAN edge to ensure in the SD-WAN Secure
   L3VPN that their advertised VPN IDs (and Route Targets) are truly
   theirs.  In addition, local BGP policy should careful set-up access
   lists for the routes received and propagated.  These verifications
   help prevent unauthorized advertisement of prefixes and ensures the
   integrity of the routing information within the SD-WAN environment.

   This document does not specify a SD-WAN deployment outside of the
   above walled garden described above.  Such a deployment is outside
   the scope of this document.

6.  IANA Considerations

6.1.  Hybrid (SD-WAN) Overlay SAFI

   IANA has assigned SAFI = 74 as the Hybrid (SD-WAN) SAFI.

6.2.  Tunnel Encapsulation Attribute Type

   IANA is requested to assign a type from the BGP Tunnel Encapsulation
   Attribute Tunnel Types as follows [RFC8126]:






Dunbar, et al.            Expires 23 July 2025                 [Page 53]

Internet-Draft            SDWAN Edge Discovery              January 2025


    Value   Description    Reference
   -----   ------------   ---------
    25       SD-WAN-Hybrid   (this document)

6.3.  Tunnel Encapsulation Attribute Sub-TLV Types

   IANA is requested to assign the following sub-Types in the BGP Tunnel
   Encapsulation Attribute Sub-TLVs registry:


      Value   Type Description            Reference     Section
      -----   -----------------------     ------------- -------
       64  IPSEC-SA-ID Sub-TLV            This document  3.3.1
       65  Extended Port Property Sub-TLV This document  3.3.6
       66  Underlay Transport Sub-TLV     This document  3.3.6.1
       67  IPsec SA Rekey Counter Sub-TLV This document  3.3.2
       68  IPsec Public Key Sub-TLV       This document  3.3.3
       69  IPsec SA Proposal Sub-TLV      This document  3.3.4
       70  Simplified IPsec SA sub-TLV    This document  3.3.5

6.4.  Tunnel Encapsulation Attribute Type

   IANA is requested to assign a type from the BGP Tunnel Encapsulation
   Attribute Tunnel Types as follows [RFC8126]:


    Value   Description    Reference
   -----   ------------   ---------
    25       SD-WAN-Hybrid   (this document)

7.  References

7.1.  Normative References

   [RFC1997]  Chandra, R., Traina, P., and T. Li, "BGP Communities
              Attribute", RFC 1997, DOI 10.17487/RFC1997, August 1996,
              <https://www.rfc-editor.org/info/rfc1997>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
              DOI 10.17487/RFC4271, January 2006,
              <https://www.rfc-editor.org/info/rfc4271>.




Dunbar, et al.            Expires 23 July 2025                 [Page 54]

Internet-Draft            SDWAN Edge Discovery              January 2025


   [RFC4360]  Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
              Communities Attribute", RFC 4360, DOI 10.17487/RFC4360,
              February 2006, <https://www.rfc-editor.org/info/rfc4360>.

   [RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
              Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
              2006, <https://www.rfc-editor.org/info/rfc4364>.

   [RFC4456]  Bates, T., Chen, E., and R. Chandra, "BGP Route
              Reflection: An Alternative to Full Mesh Internal BGP
              (IBGP)", RFC 4456, DOI 10.17487/RFC4456, April 2006,
              <https://www.rfc-editor.org/info/rfc4456>.

   [RFC4659]  De Clercq, J., Ooms, D., Carugi, M., and F. Le Faucheur,
              "BGP-MPLS IP Virtual Private Network (VPN) Extension for
              IPv6 VPN", RFC 4659, DOI 10.17487/RFC4659, September 2006,
              <https://www.rfc-editor.org/info/rfc4659>.

   [RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
              "Multiprotocol Extensions for BGP-4", RFC 4760,
              DOI 10.17487/RFC4760, January 2007,
              <https://www.rfc-editor.org/info/rfc4760>.

   [RFC4761]  Kompella, K., Ed. and Y. Rekhter, Ed., "Virtual Private
              LAN Service (VPLS) Using BGP for Auto-Discovery and
              Signaling", RFC 4761, DOI 10.17487/RFC4761, January 2007,
              <https://www.rfc-editor.org/info/rfc4761>.

   [RFC4762]  Lasserre, M., Ed. and V. Kompella, Ed., "Virtual Private
              LAN Service (VPLS) Using Label Distribution Protocol (LDP)
              Signaling", RFC 4762, DOI 10.17487/RFC4762, January 2007,
              <https://www.rfc-editor.org/info/rfc4762>.

   [RFC5701]  Rekhter, Y., "IPv6 Address Specific BGP Extended Community
              Attribute", RFC 5701, DOI 10.17487/RFC5701, November 2009,
              <https://www.rfc-editor.org/info/rfc5701>.

   [RFC6793]  Vohra, Q. and E. Chen, "BGP Support for Four-Octet
              Autonomous System (AS) Number Space", RFC 6793,
              DOI 10.17487/RFC6793, December 2012,
              <https://www.rfc-editor.org/info/rfc6793>.

   [RFC7296]  Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T.
              Kivinen, "Internet Key Exchange Protocol Version 2
              (IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October
              2014, <https://www.rfc-editor.org/info/rfc7296>.





Dunbar, et al.            Expires 23 July 2025                 [Page 55]

Internet-Draft            SDWAN Edge Discovery              January 2025


   [RFC7606]  Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
              Patel, "Revised Error Handling for BGP UPDATE Messages",
              RFC 7606, DOI 10.17487/RFC7606, August 2015,
              <https://www.rfc-editor.org/info/rfc7606>.

   [RFC8092]  Heitz, J., Ed., Snijders, J., Ed., Patel, K., Bagdonas,
              I., and N. Hilliard, "BGP Large Communities Attribute",
              RFC 8092, DOI 10.17487/RFC8092, February 2017,
              <https://www.rfc-editor.org/info/rfc8092>.

   [RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
              Writing an IANA Considerations Section in RFCs", BCP 26,
              RFC 8126, DOI 10.17487/RFC8126, June 2017,
              <https://www.rfc-editor.org/info/rfc8126>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8277]  Rosen, E., "Using BGP to Bind MPLS Labels to Address
              Prefixes", RFC 8277, DOI 10.17487/RFC8277, October 2017,
              <https://www.rfc-editor.org/info/rfc8277>.

   [RFC8489]  Petit-Huguenin, M., Salgueiro, G., Rosenberg, J., Wing,
              D., Mahy, R., and P. Matthews, "Session Traversal
              Utilities for NAT (STUN)", RFC 8489, DOI 10.17487/RFC8489,
              February 2020, <https://www.rfc-editor.org/info/rfc8489>.

   [RFC9012]  Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
              "The BGP Tunnel Encapsulation Attribute", RFC 9012,
              DOI 10.17487/RFC9012, April 2021,
              <https://www.rfc-editor.org/info/rfc9012>.

7.2.  Informative References

   [IANA-AH]  IANA, "IANA-AH", <https://www.iana.org/assignments/isakmp-
              registry/isakmp-registry.xhtml#isakmp-registry-9>.

   [IANA-ESP] IANA, "IANA-ESP", <https://www.iana.org/assignments/
              isakmp-registry/isakmp-registry.xhtml#isakmp-registry-9>.

   [MEF70.1]  MEF, "SD-WAN Service Attributes and Service Framework",
              November 2021, <https://www.mef.net/resources/mef-70-1-sd-
              wan-service-attributes-and-service-framework/>.

   [MEF70.2]  MEF, "SD-WAN Service Attributes and Service Framework",
              October 2023, <https://www.mef.net/resources/mef-70-2-sd-
              wan-service-attributes-and-service-framework/>.



Dunbar, et al.            Expires 23 July 2025                 [Page 56]

Internet-Draft            SDWAN Edge Discovery              January 2025


   [Net2Cloud]
              L. Dunbar, A Malis, C. Jacquenet, M. Toy and K. Majumdar,
              "Dynamic Networks to Hybrid Cloud DCs: Problem Statement
              and Mitigation Practice", September 2023,
              <https://datatracker.ietf.org/doc/draft-ietf-rtgwg-
              net2cloud-problem-statement/>.

   [RFC5114]  Lepinski, M. and S. Kent, "Additional Diffie-Hellman
              Groups for Use with IETF Standards", RFC 5114,
              DOI 10.17487/RFC5114, January 2008,
              <https://www.rfc-editor.org/info/rfc5114>.

   [RFC5903]  Fu, D. and J. Solinas, "Elliptic Curve Groups modulo a
              Prime (ECP Groups) for IKE and IKEv2", RFC 5903,
              DOI 10.17487/RFC5903, June 2010,
              <https://www.rfc-editor.org/info/rfc5903>.

   [RFC9016]  Varga, B., Farkas, J., Cummings, R., Jiang, Y., and D.
              Fedyk, "Flow and Service Information Model for
              Deterministic Networking (DetNet)", RFC 9016,
              DOI 10.17487/RFC9016, March 2021,
              <https://www.rfc-editor.org/info/rfc9016>.

   [SD-WAN-BGP-USAGE]
              L. Dunbar, A Sajassi, J Drake, and B. Najem, "BGP Usage
              for SD-WAN Overlay Networks", September 2023,
              <https://datatracker.ietf.org/doc/draft-ietf-bess-bgp-
              sdwan-usage/>.

   [Secure-EVPN]
              A Sajassi, A. Banerjee, S. Thoria, D. Carrell, B. Weis, J.
              Drake, "Secure EVPN", November 2024,
              <https://datatracker.ietf.org/doc/draft-ietf-bess-secure-
              evpn/>.

Appendix A.  Acknowledgments

   Acknowledgements to Wang Haibo, Shunwan Zhuang, Hao Weiguo, and
   ShengCheng for implementation contribution.  Many thanks to Yoav Nir,
   Graham Bartlett, Jim Guichard, John Scudder, and Donald Eastlake for
   their review and suggestions.

Contributors

   Below is a list of other contributing authors:

   *  Gyan Mishra,




Dunbar, et al.            Expires 23 July 2025                 [Page 57]

Internet-Draft            SDWAN Edge Discovery              January 2025


   *  Shunwan Zhuang,

   *  Sheng Cheng, and

   *  Donald Eastlake.

Authors' Addresses

   Linda Dunbar
   Futurewei
   Dallas, TX,
   United States of America
   Email: ldunbar@futurewei.com


   Susan Hares
   Huawei
   United States of America
   Email: shares@ndzh.com


   Kausik Majumdar
   Oracle
   California,
   United States of America
   Email: kmajumdar@microsoft.com


   Robert Raszuk
   Arrcus
   United States of America
   Email: robert@raszuk.net


   Venkit Kasiviswanathan
   Arista
   United States of America
   Email: venkit@arista.com













Dunbar, et al.            Expires 23 July 2025                 [Page 58]