DetNet D. Guo Internet Draft X. You Intended status: Informational R. Liu Expires: 11 June 2024 S. Zhu New H3C Technologies Co., Ltd 11 December 2023 Compensation Reference for Jitter Reduction Mechanism draft-guo-detnet-compensation-reference-00.txt Abstract This document describes several methods for obtaining reference delay applicable to different scenarios. These methods are in conjunction with mechanisms aimed at reducing end-to-end delay jitter in a scalable deterministic network. The goal is to meet specific business requirements and provide greater flexibility in the multipath planning for service protection. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on June 11, 2024. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents Guo, et al. Expires June 11, 2024 [Page 1] Internet-Draft Compensation Reference for JRM December 2023 carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents Abstract..........................................................1 Status of this Memo...............................................1 Table of Contents.................................................2 1. Introduction...................................................2 2. Terminology....................................................3 3. Network Model..................................................4 3.1. SN Model..................................................6 3.2. AN Model..................................................7 4. Compensation Reference Delay Collection Methods...............10 4.1. Compensation Reference Collection for SN.................12 4.1.1. Implementation Principle............................12 4.1.2. Data Plane..........................................13 4.1.3. Control Plane.......................................13 4.2. One-Way Compensation Reference Collection for AN.........15 4.2.1. Implementation Principle............................15 4.2.2. Data Plane..........................................18 4.2.3. Control Plane.......................................18 4.3. Round-Trip Compensation Reference Collection for AN......20 4.3.1. Implementation Principle............................21 4.3.2. Data Plane..........................................24 4.3.3. Control Plane.......................................25 5. Security Considerations.......................................26 6. IANA Considerations...........................................26 7. Acknowledgments...............................................26 8. Contributors..................................................26 9. References....................................................27 9.1. Informative References...................................27 Authors' Addresses...............................................29 1. Introduction In scaling deterministic networks, as defined in [I-D.ietf-detnet- scaling-requirements], end-to-end services may traverse multiple network domains and employ various queuing mechanisms within each domain. [I-D.guo-detnet-jitter-reduction-mechanism]describes a compensation mechanism designed to reduce end-to-end jitter and meet the requirements of applications with tightly bounded jitter. This Guo, et al. Expires June 11, 2024 [Page 2] Internet-Draft Compensation Reference for JRM December 2023 document expands on the Jitter Reduction Mechanism for DetNet [I- D.guo-detnet-jitter-reduction-mechanism]. For many applications, it is crucial that different copies of the same data transmitted through multiple paths maintain consistent delays within specified upper and lower bounds. This ensures that in the event of a path failure, the application remains unaware of the fault. Therefore, a unified value is required as the compensation delay reference for the transmission delays in multi-path transport providing service protection. This value is referred to as the Path Delay Reference, denoted as PthRefD in [I-D.guo-detnet-jitter- reduction-mechanism]. This value will be greater than the actual transmission delays on each path to accommodate additional delays introduced by anticipated future path changes (e.g., when a new path is added to the service protection path group after a path failure, and the new path is longer than the original paths in the service protection path group). By using the Path Delay Reference to compensate for transmission delays, applications remain unaware of such fault occurrences and recoveries within a bounded delay range. In certain scenarios, complete transmission delays for data during transmission may not be obtainable. According to [I-D.guo-detnet- jitter-reduction-mechanism], it is sufficient to obtain the upper limit of the compensation delay required for the corresponding path. In this document, the "upper limit of the compensation delay required for the corresponding path" is referred to as Cap_i, simplified as the compensation reference delay value in this document. Given the existence of both time-synchronized and non-time- synchronized scenarios in practical requirements, and considering the diversity of actual network deployments and the complexity of the issues themselves, it is almost impossible to have a unified method to address all requirements. Our approach is to abstract the network into two typical models and provide suitable methods for each model. In the following sections, we will first categorize and describe the network models and then provide methods for collecting the compensation reference delay values corresponding to different types. Different methods will be implemented for different network types in actual deployment to simplify the realization. 2. Terminology The following terminology is introduced in this document on the basis of [I-D.guo-detnet-jitter-reduction-mechanism]: HN Guo, et al. Expires June 11, 2024 [Page 3] Internet-Draft Compensation Reference for JRM December 2023 Abbreviation of Head Node. For the definition of Head Node, please refer to [I-D.guo-detnet-jitter-reduction-mechanism]. CN Abbreviation of Compensation Node. For the definition of Compensation Node, please refer to [I-D.guo-detnet-jitter- reduction-mechanism]. Synchronous Networking (SN) It is a networking model in which time synchronization is implemented between HN and CN. Asynchronous Networking (AN): It is a networking model in which there is no time synchronization between HN and CN. Synchronous Domain (SD) If time synchronization is achieved between the I-GW and E-GW of the deterministic network domain through which the deterministic flow path passes, the deterministic network domain where the I-GW and E-GW are located is considered a synchronous domain. 3. Network Model Before deploying deterministic service flows, the controller plane performs path planning for the network, identifies the networking model to which the completed planned paths belong, and labels the paths accordingly. In the path from the Head Node (HN) to the Compensation Node (CN), if time synchronization is achieved between HN and CN, we consider the networking model for the deployment of deterministic flows to be SN, and we collect reference values using the SN method. In the path from HN to CN, if time synchronization is not achieved between HN and CN, we consider the networking model for the deployment of deterministic flows to be AN, and we collect reference values using the AN method. For AN, the path from HN to CN may contain Synchronous Domains (SD) consisting of multiple network nodes, each with independent Ingress Gateway (I-GW) and Egress Gateway (E-GW) devices. There may also be individual network nodes within the path that are not time- synchronized with other nodes. These nodes can be considered as special SDs, where a single node forms its own domain, with its I-GW Guo, et al. Expires June 11, 2024 [Page 4] Internet-Draft Compensation Reference for JRM December 2023 and E-GW being the node itself, and time synchronization is required within the node. Upon entering an SD from the I-GW, Packet Replication Functions (PRF) may be implemented, and PRF and Packet Elimination Functions (PEF) may be implemented before or within the E-GW. The packets output from the E-GW of the SD may not be the original packets but rather multiple copies of the original packets. The copies generated by the internal PRF function within the SD may follow different transmission paths, and the transmission delays on different paths are different. The differences in delay introduced by the selection of packets from different paths by the PEF are considered as jitter within the SD. The controller plane actually performs path planning to plan out candidate paths that meet application requirements. Some of the candidate paths are selected to form a service protection group, while the remaining candidate paths can serve as backup paths in the event of a failure. The method described in this document is based on the paths that have already been planned, to collect compensation reference delay values. Guo, et al. Expires June 11, 2024 [Page 5] Internet-Draft Compensation Reference for JRM December 2023 3.1. SN Model HN(PE1) and CN(PE2) synchronized ^ ^ ^ ^ ^ ^ ^ P1--------------P2 ^ ^ /| |\ ^ ^ / | | \ ^ PE1 + | | + PE2 (HN) + | | + (CN) | \ | | / | | \| |/ | | P3--------------P4 | | | | | | | | | Talker Listener (a) an example of SN +---------------------+ Tanlker------+ I-GW(HN)---E-GW(CN) +-------Listener +---------------------+ (b) the model of SN Figure 1 Example and Model Abstraction of SN Figure 1(a) shows an example of SN. The Talker is connected to PE1, which acts as the Head Node (HN), while the Listener is connected to PE2, which acts as the Compensation Node (CN). Time synchronization exists between PE1 and PE2. The application networking for accessing SN adopts the abstraction model shown in Figure 1(b), which is represented by the first I-GW (i.e., HN) and the last E-GW (i.e., CN). The residence delay between HN and CN is considered as variable delay within the SN, without focusing on the specific paths within the SN, thus simplifying implementation. Guo, et al. Expires June 11, 2024 [Page 6] Internet-Draft Compensation Reference for JRM December 2023 3.2. AN Model no time synchronization between HN and CN ^ ^ ^ ^ ^ ^ ^ P1--------------P2 ^ ^ /| |\ ^ ^ / | | \ ^ PE1 + | | + PE2 (HN) + | | + (CN) | \ | | / | | \| |/ | | P3--------------P4 | | | | | | | | | Talker Listener (a) an example of AN +------+ +------+ | +---P1----P2------------------+ | | | | | | +---P1----P2----P4------------+ | | | | | | +---P1----P3----P4------------+ | | HN | | CN | |(PE1) +---P1----P3----P4----P2------+ (PE2)| | | | | | +---P3----P1----P2----P4------+ | | | | | | +---P3----P1----P2------------+ | | | | | | +---P3----P4----P2------------+ | | | | | | +---P3----P4------------------+ | +------+ +------+ (b) the model of AN Figure 2 Example and Model Abstraction of AN Guo, et al. Expires June 11, 2024 [Page 7] Internet-Draft Compensation Reference for JRM December 2023 Figure 2(a) shows a simple example of AN. PE1 is connected to the Talker and acts as the Head Node (HN), while PE2 is connected to the Listener and acts as the Compensation Node (CN). There is no requirement for time synchronization between the various P, PE1, and PE2 nodes (Note: time synchronization can be present but is not utilized). Assuming each P node is a physical node, two-downstream- interface PRF is planned for each P node. Figure 2(b) shows the abstract model of AN. In this diagram, there are 8 different paths formed between PE1 and PE2 through different nodes. After path planning, a portion or all of the paths are used to form service protection members between HN and CN. Guo, et al. Expires June 11, 2024 [Page 8] Internet-Draft Compensation Reference for JRM December 2023 no time synchronization between HN and CN ^ ^ ^ ^ ^ 1 2 ^ ^ SD1--------------SD2 ^ ^ 2/|0 0|\1 ^ ^ 1 / | | \ ^ PE1 + | | + PE2 (HN) + | | + (CN) | 0 \ | | / | | 2\|1 1|/0 | | SD3--------------SD4 | | 0 2 | | | | | | | Talker Listener (a) an example of complex AN +------+ +------+ | +--2-SD1-1--2-SD2-1--------------------+ | | | | | | +--2-SD1-1--2-SD2-0--1-SD4-0-----------+ | | | | | | +--2-SD1-0--1-SD3-0--2-SD4-0-----------+ | | HN | | CN | |(PE1) +--2-SD1-0--1-SD3-0--2-SD4-1--0-SD2-1--+ (PE2)| | | | | | +--2-SD3-1--0-SD1-1--2-SD2-0-----------+ | | | | | | +--2-SD3-1--0-SD1-1--2-SD-1------------+ | | | | | | +--2-SD3-0--2-SD4-1--0-SD2-1-----------+ | | | | | | +--2-SD3-0--2-SD4-0--------------------+ | +------+ +------+ (b) a model of complex AN Figure 3 Complex AN Example and Model Abstraction Guo, et al. Expires June 11, 2024 [Page 9] Internet-Draft Compensation Reference for JRM December 2023 Figure 3(a) shows an example of complex AN. PE1 is connected to the Talker and acts as the Head Node (HN), while PE2 is connected to the Listener and acts as the Compensation Node (CN). SD represents a deterministic network domain after time synchronization by edge gateways. There is no requirement for time synchronization between SD, PE1, and PE2 (Note: time synchronization can be present but is not utilized). Assuming two-downstream-interface PRF copies are planned for each SD, Figure 3(b) shows the abstract model of this complex AN. Taking "2-SD1-1" in the figure as an example, 2 represents the input interface number of the I-GW of SD1, the first 1 represents the corresponding SD number in Figure 3(a), and the second 1 represents the output interface number of E-GW of SD1. The numbering of SDs and interfaces is for illustrative purposes, primarily for the sake of simplicity and clarity in describing the network topology as multiple paths. In Figure 3(b), the differences in transmission delay within the SD are considered as jitter within the domain, and 8 different paths are formed between PE1 and PE2 through different nodes. After path planning, a portion or all of the paths are used to form service protection members between HN and CN. 4. Compensation Reference Delay Collection Methods As mentioned earlier, time synchronization between the I-GW and E-GW of SD (treated the same as SN) allows for the direct acquisition of residence delay within it, eliminating the need to consider where copies are generated inside the SD/SN, which path each copy is transmitted on, and the final selection of the copy. Instead, the differences in transmission delay of copies from different paths are uniformly considered as jitter within the SD/SN, greatly simplifying implementation without compromising the reliability of deterministic elements, covering the application scenarios proposed in [I-D.draft- ietf-detnet-mpls-over-ip-preof]. In AN, there is no time synchronization between HN and CN, making it impossible to directly obtain end-to-end network residence time. In a complex AN containing SD, the residence delay within the SD through which the HN->CN path passes can be obtained through the cooperation of the SD's I-GW and E-GW, independent of other nodes within the SD. The input interface of the SD's I-GW and the output interface of the E-GW participate in path identification, as do the input and output interfaces of individual nodes. Guo, et al. Expires June 11, 2024 [Page 10] Internet-Draft Compensation Reference for JRM December 2023 +------------+ +---------------E1---+ | | +---+ | | +---R3---+ | +---+ |src|------R1 +---+ | E3----O----+dst| +---+ | | E2-------+ +---+ +----------R2 | +-----------------+ R: replication function (PRF) E: elimination function (PEF) O: ordering function (POF) Figure 4 PREOF scenario in a DetNet network Figure 4 is copied from [I-D.ietf-detnet-mpls-over-ip-preof]. During data forwarding, the E-node needs to implement PEF to remove the redundant copies generated by multiple PRFs. However, when collecting compensation reference delay values in OAM packets, the copies generated by PRFs need to be retained, and the CN discards them after compensation reference delay values are applied. Next, we will introduce the compensation reference value collection methods for synchronous and asynchronous networks, respectively. Guo, et al. Expires June 11, 2024 [Page 11] Internet-Draft Compensation Reference for JRM December 2023 4.1. Compensation Reference Collection for SN 4.1.1. Implementation Principle | | |<------------------PthRefD------------------------>| | | |t0 |t1 |t2 |t3 |tr +------------------------+-----+-----------+------->+ | | | | | | | | | | | Copy3 /------------>+ | | | | / | | | | Copy2| /-----/---------------------+---------->+<-Tr_2->+ | / | | | Copy1+/---------------------------->+ | | | | Figure 5 SN Compensation Reference Delay For the SN, the compensation reference delay is denoted as the path reference delay (PthRefD). When the timestamp entering the HN is set as the starting time t0, and the arrival times of the copies generated by each PRF at the CN are denoted as t_i (where 0 < i < n + 1, and n is the number of received copies). Let t_k (where 0 < k < n + 1) represent the latest arrival time of all the copies at the CN. Based on t_k - t_0, an additional value is added as the compensation reference value, determined by the user or controller based on engineering requirements. This additional value includes the extra transmission delay, the margin for variable delay, and the processing delay at the CN before scheduling. This value is the PthRefD. Note: To simulate obtaining the HN receive timestamp, after constructing the OAM message, the HN sends it to the receive interface, and the receive interface then loops it back to the HN. As shown in Figure 5, the timestamp for receiving the OAM loopback message at the HN is t0. On the HN, PRF is implemented, generating Copy1 and Copy2. Copy2 is further processed by a node inside the SD, generating Copy3. The timestamps for the arrival of the three copies at the CN are denoted as t3. Adding Tr_3 to t3 provides the value tr, where Tr_3 is a value chosen by the user or controller based on Guo, et al. Expires June 11, 2024 [Page 12] Internet-Draft Compensation Reference for JRM December 2023 engineering requirements. The value tr represents the reference time point for the path reference delay (PthRefD) relative to t0 on the CN. It can also be understood as adding the user or controller- selected value Tr_3 to (t3-t0) to obtain the path reference delay (PthRefD). Therefore, we have: PthRefD = tr - t0, or PthRefD = (t3 - t0) + Tr_3. The PthRefD for the SN represents the compensation reference. 4.1.2. Data Plane For the one-way collection of compensation reference for the SN, the data plane behaviors of each node are described as follows: 1. The HN data plane receives the OAM message for collecting compensation reference delay and transforms it into an OAM packet for collecting compensation reference delay. 2. The HN sends the OAM packet for collecting compensation reference delay. (e.g.: To simulate the reception of app-flow through loopback on the HN: assuming the Talker is connected to interface if_0 of the HN, an OAM packet with loopback indication is sent to if_0. Upon receiving this packet after looping back, the receiving timestamp is recorded. After performing the PRF on the HN, the OAM copies are sent to the CN via different paths planned by the controller plane.) 3. At each node implementing PEF on the HN->CN path, when identifying OAM for collecting SN compensation reference delay, the node does not eliminate duplicate copies and directly forwards the message. 4. At the CN node, upon receiving the OAM packet for collecting SN compensation reference delay, the measurement data is reported to the control plane. At the CN, it is necessary to ensure that a set of copies of an OAM packet covers all the PREOF paths within the domain. 4.1.3. Control Plane For the one-way collection of compensation reference for the SN, the control plane behaviors of each node are described as follows: Guo, et al. Expires June 11, 2024 [Page 13] Internet-Draft Compensation Reference for JRM December 2023 1. Plan network paths. 2. Configure operations for nodes HN and CN. 3. The control plane of CN needs to specify conditions of integrity check (ensuring the delays for all planned paths are collected). 4. The control plane needs to allocate OAM packet sequence numbers and send messages to HN to construct OAM packets with these sequence numbers. These OAM packets are used to collect the compensation reference delay. 5. Coordinate with the data plane to perform measurements and calculate the reference delay. The control plane receives measurement data reported by CN, calculates the longest transmission delay between HN and CN, and adds a value chosen by the user or controller based on engineering requirements (including additional values for line transmission, margin for variable delay, and processing delay at CN before scheduling) to obtain the compensation reference delay for HN->CN. Subsequently, the transmission delay for packets of DetNet flow is compensated based on the actual residence delay and this compensation reference delay. 6. Configure the compensation reference delay value to HN or CN. If configured on the HN node, the HN node can determine the point in time after compensation, simplifying CN implementation. If configured on the CN node, the CN node determines the point in time after compensation. 7. Runtime maintenance: periodically collect and calculate the full path reference delay for HN->CN, and refresh the reference delay for HN or CN. Isolate paths that have faults, and release isolation after recovery. For the collection of residence delay, the control plane needs to provide relevant information to the data plane of each network node in two ways: Option 1: The control plane globally assigns node identifiers and distributes these identifiers to each network node. The control plane sends delay collection-related operations to HN. Option 2: The control plane configures operations on a per-flow basis to HN and CN. Guo, et al. Expires June 11, 2024 [Page 14] Internet-Draft Compensation Reference for JRM December 2023 4.2. One-Way Compensation Reference Collection for AN The one-way collection method is suitable for cases where the round- trip paths are different. Refer to [RFC9016] for the description of BiDir. According to the compensation delay formula in [I-D.guo- detnet-jitter-reduction-mechanism]: CompD_i = Cap_i - (T'1_i + T'2_i + T'3_i + T'4_i) Cap_i is obtained before the deployment of DetNet-flow sessions and is used as the compensation reference delay value. This value is the difference between PthRefD and FixD_i. Since time synchronization has not been performed, PthRefD and FixD_i cannot be directly obtained. Therefore, the compensation reference delay value Cap_i needs to be transformed into a comparable relative value. 4.2.1. Implementation Principle | | |<------------------PthRefD------------------------>| | | |t_0 |t_1 |t_3 |tr +--------------------------------+---------+------->+ | | | | | | | | | | | | | | | | Path2+--------------------------------+-------->+<-Tr_2->+ | | | | Path1+------------------------------->+<------Tr_1------>+ | | | Figure 6 Multi-path reference delay and compensation delay According to section 3.2, the paths between HN and CN can be abstracted as multiple paths. To simplify, consider Figure 6 with 2 paths and describe the method for calculating the compensation reference delay value. Let t0 be the time when the measurement data packet is received at HN, and let t_1 and t_2 be the times when two copies of the same measurement data packet arrive at CN via Path1 and Path2, respectively. Obviously, t_2 is the arrival time of the last copy. tr is the reference time point based on t_2 delayed by Tr_2, Guo, et al. Expires June 11, 2024 [Page 15] Internet-Draft Compensation Reference for JRM December 2023 where the value of Tr_2 is chosen by the user or SDN controller based on engineering requirements (including additional values for line transmission, margin for variable delay, and processing delay at CN before scheduling). The time difference between t_0 and tr corresponds to PthRefD as described in [I-D.guo-detnet-jitter- reduction-mechanism]. Since t_0 represents the time at HN, and t_1, t_2, and tr represent the time at CN, and there is no time synchronization between HN and CN, PthRefD cannot be directly obtained by subtraction. However, t_1, t_2, and tr are all times at CN, and the relative delay differences for each path can be obtained. PthRefD = (FixD_1 + ActRefD_1) + Tr_1 = FixD_1 + (ActRefD_1 + Tr_1) PthRefD - FixD_1 = ActRefD_1 + Tr_1 Where Cap_1 = PthRefD - FixD_1, Therefore, Cap_1 = ActRefD_1 + Tr_1 That is, Cap_1 is the sum of (ActRefD_1 + Tr_1), FixD_1 is the fixed delay of Path1, which is constant and not considered in compensation. ActRefD_1 is the sum of the residence delays of the measurement packets in each domain, and Tr_1 is the delay from t1 to tr, i.e., tr-t1. Based on the method for obtaining actual transmission delay within the domain as described in [I-D.guo-detnet-jitter-reduction- mechanism], ActRefD_1 can be obtained, and thus Cap_1 can be obtained; similarly, Cap_2 can also be obtained. Path1 --------------------------------> ---- ---- / \ Path2 / \ | HN | --------------------------------> | CN | \ / \ / ---- Path3 ---- --------------------------------> Figure 7 Illustration of OAM packet transmission Guo, et al. Expires June 11, 2024 [Page 16] Internet-Draft Compensation Reference for JRM December 2023 As shown in Figure 7, according to the service protection implementation, HN will send multiple copies of the same OAM packet on multiple paths, and CN will collect information and calculate the compensation reference delay value for each member path. The process to obtain the Cap_i value is as follows: 1. Record the corresponding path and reception time at CN; simultaneously, according to the method described in [I-D.guo-detnet- jitter-reduction-mechanism], obtain the residence delays in each domain and accumulate them together to obtain the actual residence delay of the OAM packets for each path. For ease of description, let's assume that Path i is an arbitrary path for service protection, and the accumulated value of the actual residence delay within each domain after the OAM packet passes through this path is denoted as ActRefD_i. 2. Collect the latest arrival time t_2 (as shown in Figure 7). 3. Based on engineering requirements, select the required delay adjustment Tr_2 (as shown in Figure 7), and delay the latest arrival time to obtain the reference time point tr, i.e., tr = t_2 + Tr_2. 4. Calculate the difference between the arrival time of the measurement packet for each path relative to tr, denoted as Tr_i (such as Tr_1, Tr_2 in Figure 7). Tr_i = tr - t_i; 5. Calculate the compensation reference delay value for each path i as Cap_i = ActRefD_i + Tr_i. When calculating the compensation delay for each path, if a data packet passes through Path i and the residence delay in each domain is ActD_i (ActD_i is a variable delay and may not be the same after each transmission), then the required compensation delay is CompD_i = Cap_i - ActD_i. Cap_i can be distributed to the HN node, encapsulated into the DetNet-flow data packet, and used by CN to perform delay compensation using Cap_i and ActD_i obtained from the data packet; or Cap_i can be distributed to the flow-related compensation information table at CN for delay compensation. Guo, et al. Expires June 11, 2024 [Page 17] Internet-Draft Compensation Reference for JRM December 2023 4.2.2. Data Plane Description of the data plane behavior for the one-way collection of compensation reference delay at each node: 1. HN's data plane receives the OAM message for collecting compensation reference delay values and converts it into an OAM packet for collecting compensation reference delay values. 2. HN sends the OAM packet for collecting compensation reference delay. Example: Simulating the reception of deterministic data via loopback at HN: Assuming the Talker is connected to interface if_0 of HN, an OAM packet marked for loopback is sent to if_0. When this packet is looped back and received, the reception timestamp is recorded. After performing the PRF, different copies of the OAM packet are sent to CN via different paths planned by the controller plane. 3. Each node (if it is an edge gateway I-GW/E-GW of the SD, it is handled by the SD's I-GW and E-GW) collects the residence delay of the OAM packets passing through the path and, with the assistance of the control plane, generates the necessary identification information for identifying the path. Similar to the requirements in [I-D.ietf- detnet-pof], when CN nodes receive an OAM packet or a service data packet, they need to obtain path-related information. 4. At each node implementing PEF on the HN->CN path, when identifying OAM for collecting compensation reference delay, the node does not eliminate duplicate copies and directly forwards the message. 5. When CN nodes receive OAM packets, they record the reception time of each OAM packet and send it to the control plane. 4.2.3. Control Plane The control plane behavior of each node for one-way collection of compensation reference delay for AN is described as follows: 1. Plan network paths and identify the network as SN or AN. 2. Configure the necessary information for generating path identifiers at each node. Similar to the requirements in [I-D.ietf- detnet-pof], when CN nodes receive an OAM packet or a service data packet, they need to obtain path-related information. Guo, et al. Expires June 11, 2024 [Page 18] Internet-Draft Compensation Reference for JRM December 2023 3. Configure the path identifier information on CN to identify different paths on which OAM packets or app-flow packets transmitted. 4. Set up a timeout mechanism in the control plane to identify if the received OAM packet arrival time is acceptable. The historical window mechanism on TSN in [IEEE802.1CB] can be used as a reference. 5. Configure operations at intermediate nodes to collect residence delays. 6. Perform integrity checks in the control plane. 7. Allocate OAM packet sequence numbers and send messages to HN for sending OAM packets to collect compensation reference delays. 8. Coordinate with the data plane to perform measurements and calculate reference delay values. The control plane receives measurement data reported by CN, calculates compensation reference delay values, and records the arrival timestamps of multiple copies of the same OAM packet received on each path. It also records the total number of copies received on each path (used for integrity checks) and the sum of residence delays within each SD and non-SD node on that path. Based on the latest timestamp recorded on all paths, the control plane adds a value chosen by the user or controller based on engineering requirements (including additional values for line transmission, variable delay upper limit margin, and processing delay at CN before scheduling) to establish a common compensation delay reference point tr for all paths from HN to CN. tr is the timestamp at CN. The compensation reference delay values for each path are calculated as described in section 4.2.1. 9. Configure the compensation reference delay values to HN or CN. 10. Runtime maintenance: periodically collect and calculate the full path reference delay and update the reference delays to HN or CN. For delay collection in each domain, the control plane provides the relevant information required by the data plane and determines the provision method. The control plane can provide this information in the following ways: Method 1: Globally allocate node identifiers in the control plane and distribute this identifier to each node. The control plane distributes the information required for delay collection to HN. Guo, et al. Expires June 11, 2024 [Page 19] Internet-Draft Compensation Reference for JRM December 2023 Method 2: The control plane configures per-flow operations at intermediate nodes, and if SD exists, configures it at the SD's I-GW and E-GW. 4.3. Round-Trip Compensation Reference Collection for AN The round-trip collection method is suitable for situations where the round-trip paths are the same. See [RFC9016] for a description of BiDir. Due to the lack of global clock synchronization, we cannot directly obtain the end-to-end path delay. It would be very complicated to directly obtain the delay of each path segment and then accumulate the delay of each segment. Therefore, we draw on the idea of 4.2 and use relative-delay method. By using the relative-delay method to accurately obtain the approximate invariant part of the delay, combined with the variable residence delay on the path and other delays, PthRefD is obtained by comprehensive evaluation. At the same time, the difference between the approximate invariant part of the delay for the specified path and PthrefD is adjusted into the compensation reference delay Cap_i. According to the formula of compensation delay in [I-D.guo-detnet-jitter-reduction-mechanism]: CompD_i = (PthRefD - FixD_i - (T1_i + T2_i + T3_i + T4_i) Let Cap_i = PthRefD - FixD_i, We need to design a method to obtain PthRefD and FixD_i in order to obtain Cap_i before deploying DetNet flows. During the runtime, the approximate invariant part of the delay FixD_i in the service protection member path_i is periodically obtained, and the newly obtained value is used to update the previous FixD_i. When the DetNet-flow packets are actually transmitted, the new FixD_i and PthRefD are used to calculate the new Cap_i. Guo, et al. Expires June 11, 2024 [Page 20] Internet-Draft Compensation Reference for JRM December 2023 4.3.1. Implementation Principle HN CN +----------+ Req_1 +----------+ | th_R_1|<---------------------------|tc_R_1 | | | | | | th_A_1| Ack_1 | | | |--------------------------->|tc_A_1 | | | Ack_2 | | | th_A_2|--------------------------->|tc_A_2 | | | ...... |...... | | | | | | | Ack_n | | | th_A_n|--------------------------->|tc_A_n | +----------+ +----------+ (a) Round-Trip Collection Message |<-------t_Req_1------>|<-------------PthRefD----------->| | | | |tc_R_1 |th_R_1 |tc_A_1 |tc_A_2 | |----------------------+------------+--------+-----------|->time | | | | | | | | | | | Path_2|------------+------->|<---Tr_2-->| | | | | | Path_1|----------->|<--------Tr_1------>| | | | | | | | | (b) Delay Model for Round-Trip Collection Method Figure 9 Round-Trip Collection Message and Delay model According to section 3.2, the paths between HN and CN can be abstracted as multiple paths. Figure 9(a) is a diagram for measuring and collecting message delay information. CN sends a request type OAM message, Req_1, to HN through Path_1 (Path_1 being any path within the service protection group). Upon receiving the message, HN generates n response type OAM messages, Ack_1 to Ack_n, and sends them through Path_1 to Path_n back to CN. CN records the transmission timestamp, tc_R_1, when sending the request message. During the transmission of Req_1, each node (including the SD's I-GW and E-GW, or non-SD nodes) collects and Guo, et al. Expires June 11, 2024 [Page 21] Internet-Draft Compensation Reference for JRM December 2023 calculates the residence delay on the path of CN->HN at SD or non-SD nodes, which is designated as ActD_Req (excluding the line delay between SDs, or between non-SD nodes, or between SD and non-SD nodes). During the transmission of Act_i (i = 1 to n), each node records the residence delay on the path of HN->CN at SD or non-SD nodes, denoted as ActD_Act_i (i = 1 to n) (excluding the line delay between SD, non-SD nodes, or between SD and non-SD nodes). HN records the reception timestamp th_R_1 for Req_1 and the transmission timestamps th_A_1 to th_A_n for the n response-type OAM messages Ack_1 to Ack_n. HN calculates n residence delays and uses them as the initial values for ActD_Act_1 to ActD_Act_n: ActD_Act_i = th_A_i - th_R_1 (i = 1 to n) Along with carrying ActD_Act_i (i = 1 to n), Act_i (i = 1 to n) also carries ActD_Req. CN records the reception timestamp tc_A_i (i = 1 to n) for receiving Act_i. CN can obtain ActD_Req and ActD_Ack_i from Act_i. Figure 9(b) shows a comprehensive delay model for multiple stages, using an example with two paths. In this figure, th_R_1 and th_A_1~2 are timestamps at HN, while tc_R_1 and tc_A_1~2 are timestamps at CN. The tr represents the compensation delay reference point for all paths from HN to CN, calculated as the latest timestamp of receiving the response-type OAM in CN (tc_A_2 in the figure), with additional values determined by the user or controller based on engineering requirements. These additional values include extra link delay, variable residence delay upper limit, and any processing delay at CN before scheduling. The specific processing steps are described as follows: 1. delay information collection. As shown in Figure 9(a), Path_1 to Path_n are n paths with the same HN and CN, used for service protection. a) CN sends Req_j message to HN via any path j (0CN path, when identifying OAM for collecting compensation reference delay, the node does not eliminate duplicate copies and directly forwards the message. 7. At the CN node, the reception time of each response message is recorded and sent to the control plane. Guo, et al. Expires June 11, 2024 [Page 24] Internet-Draft Compensation Reference for JRM December 2023 4.3.3. Control Plane The control plane configures gateway identifiers, path information, and related operations to the nodes through signaling negotiation or controller configuration. For the collection of compensation reference delay for AN, the control plane needs to coordinate with the data plane to accomplish the following process: 1. Plan network paths and identify the network as SN or AN. 2. Configure the necessary information for generating path identifiers to each node. Similar to the requirements in [I-D.ietf- detnet-pof], HN nodes and CN nodes need to obtain path-associated information when receiving OAM messages or service data packets. 3. Configure path identifier information on CN (for response-type OAM) and HN (for request-type OAM) for identifying the OAM messages transmitted along different paths and forwarding service data packets. 4. Set a timeout mechanism. Reference can be made to the historical window mechanism on TSN in [IEEE 802.1 CB]. 5. Define and configure operations for collecting residence delays at nodes (including SD's I-GW and E-GW) along the path. 6. Conduct integrity checks in the control plane. 7. Allocate OAM message sequence numbers and send messages to CN to send compensation reference delay collection OAM messages. 8. Upon receiving an OAM message, the CN control plane cooperates with the data plane to complete the measurement and calculate the reference delay. For each copy of the same OAM message received by the CN control plane, the maximum residence delay and fixed transmission delay within each node are calculated, based on which the compensation reference delay for each path is determined as described in section 4.3.1. 9. Configure the compensation reference delay value to HN or CN. 10. Runtime maintenance: periodically collect and calculate the full path reference delay and update the reference delay to CN or HN. Guo, et al. Expires June 11, 2024 [Page 25] Internet-Draft Compensation Reference for JRM December 2023 For the delay collection in each domain, the control plane coordinates to provide the relevant information required by the data plane and determines the providing method. The control plane can provide in the following ways: Method one: The control plane globally allocates node identifiers and distributes this identifier to each node (including SD's I-GW and E- GW). The control plane sends down the required information for delay collection to HN. Method two: The control plane configures operations on a per-flow basis at the intervening nodes (including SD's I-GW and E-GW). 5. Security Considerations TBD 6. IANA Considerations TBD 7. Acknowledgments TBD 8. Contributors The editor wishes to thank and acknowledge the following contributors for contributing text to this document. Guo, et al. Expires June 11, 2024 [Page 26] Internet-Draft Compensation Reference for JRM December 2023 Shenchao Xu New H3C Technologies Co., Ltd 310052 Email: xushenchao@h3c.com Ning Pan New H3C Technologies Co., Ltd 100094 Email: panning@h3c.com Xusheng Chen New H3C Technologies Co., Ltd 100094 Email: cxs@h3c.com Wei Wang New H3C Technologies Co., Ltd 100094 Email: david_wang@h3c.com Xinmin Liu New H3C Technologies Co., Ltd 100094 Email: liuxinmin@h3c.com 9. References 9.1. Informative References [I-D.guo-detnet-jitter-reduction-mechanism] Guo, D., Xu, S., "Jitter Reduction Mechanism for DetNet",Work in Progress, Internet-Draft, draft-guo-detnet- jitter-reduction-mechanism-01, 23 October 2023, < https://datatracker.ietf.org/doc/draft-guo-detnet-jitter- reduction-mechanism/>. [I-D.ietf-detnet-scaling-requirements] Liu, P., Li, Y., Eckert, T., Xiong, Q., and J. Ryoo, "Requirements for Scaling Deterministic Networks",Work in Progress, Internet-Draft, draft-ietf-detnet-scaling- requirements-05, November 2023, . [I-D.ietf-detnet-mpls-over-ip-preof] Guo, et al. Expires June 11, 2024 [Page 27] Internet-Draft Compensation Reference for JRM December 2023 Varga, B., Farkas, J., and A. G. Malis, "Deterministic Networking (DetNet): DetNet PREOF via MPLS over UDP/IP",Work in Progress, Internet-Draft, draft-ietf- detnet-mpls-over-ip-preof-07, 25 July 2023, . [I-D.ietf-detnet-pof] Varga, B., Farkas, J., Kehrer, S., Heer, T., "Deterministic Networking (DetNet): Packet Ordering Function", Work in Progress, Internet-Draft, draft-ietf-detnet-pof-06, 8 November 2023, . [RFC9016] Varga, B., Farkas, J., Cummings, R., Jiang, Y., Fedyk, D., "Flow and Service Information Model for Deterministic Networking (DetNet)", RFC 8964, DOI 10.17487/RFC9016, March 2021, . [IEEE802.1CB] IEEE,"IEEE Standard for Local and metropolitan area networks-Frame Replication and Elimination for Reliability", IEEE Std 802.1CB-2017 (2017), 1--102. . Guo, et al. Expires June 11, 2024 [Page 28] Internet-Draft Compensation Reference for JRM December 2023 Authors' Addresses Daorong Guo New H3C Technologies Co., Ltd Beijing 100094 China Email: guodaorong@h3c.com XUEJUN YOU New H3C Technologies Co., Ltd Beijing 100094 China Email: yoe@h3c.com Rubing Liu New H3C Technologies Co., Ltd Hangzhou 310052 China Email: liurubing@h3c.com Shiyin Zhu New H3C Technologies Co., Ltd Beijing 100094 China Email: zhushiyin@h3c.com Guo, et al. Expires June 11, 2024 [Page 29]