]>

United Kingdom renan.krishna@gmail.com

Ericsson

349 Terry Fox Drive Ottawa Ontario K2K 2V6 Canada Akbar.Rahman@ericsson.com

OPS mops This document explores the issues involved in the use of edge computing resources to operationalize a media use case that involves an Extended Reality (XR) application. In particular, this document discusses an XR application that can run on devices having different form factors (such as different physical sizes and shapes) and needs edge computing resources to mitigate the effect of problems such as the need to support interactive communication requiring low latency, limited battery power, and heat dissipation from those devices. This document also discusses the expected behavior of XR applications, which can be used to manage traffic, and the service requirements for XR applications to be able to run on the network. Network operators who are interested in providing edge computing resources to operationalize the requirements of such applications are the intended audience for this document.

Introduction Extended Reality (XR) is a term that includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) . AR combines the real and virtual, is interactive, and is aligned to the physical world of the user . On the other hand, VR places the user inside a virtual environment generated by a computer . MR merges the real and virtual along a continuum that connects a completely real environment at one end to a completely virtual environment at the other end. In this continuum, all combinations of the real and virtual are captured . XR applications have several requirements for the network and the mobile devices running these applications. Some XR applications (such as AR applications) require real-time processing of video streams to recognize specific objects. This processing is then used to overlay information on the video being displayed to the user. In addition, other XR applications (such as AR and VR applications) also require generation of new video frames to be played to the user. Both the real-time processing of video streams and the generation of overlay information are computationally intensive tasks that generate heat and drain battery power on the mobile device running the XR application. Consequently, in order to run applications with XR characteristics on mobile devices, computationally intensive tasks need to be offloaded to resources provided by edge computing. Edge computing is an emerging paradigm where, for the purpose of this document, computing resources and storage are made available in close network proximity at the edge of the Internet to mobile devices and sensors . A computing resource or storage is in close network proximity to a mobile device or sensor if there is a short and high-capacity network path to it such that the latency and bandwidth requirements of applications running on those mobile devices or sensors can be met. These edge computing devices use cloud technologies that enable them to support offloaded XR applications. In particular, cloud implementation techniques such as the following can be deployed:

Disaggregation:

Using Software-Defined Networking (SDN) to break vertically integrated systems into independent components. These components can have open interfaces that are standard, well documented, and non-proprietary.

Virtualization:

Being able to run multiple independent copies of those components, such as SDN Controller applications and Virtual Network Functions, on a common hardware platform.

Commoditization:

Being able to elastically scale those virtual components across commodity hardware as the workload dictates.

Such techniques enable XR applications that require low latency and high bandwidth to be delivered by proximate edge devices. This is because the disaggregated components can run on proximate edge devices rather than on a remote cloud several hops away and deliver low-latency, high-bandwidth service to offloaded applications . This document discusses the issues involved when edge computing resources are offered by network operators to operationalize the requirements of XR applications running on devices with various form factors. For the purpose of this document, a network operator is any organization or individual that manages or operates the computing resources or storage in close network proximity to a mobile device or sensor. Examples of form factors include the following: 1) head-mounted displays (HMDs), such as optical see-through HMDs and video see-through HMDs, 2) hand-held displays, and 3) smartphones with video cameras and location-sensing capabilities using systems such as a global navigation satellite system (GNSS). These devices have limited battery capacity and dissipate heat when running. Also, as the user of these devices moves around as they run the XR application, the wireless latency and bandwidth available to the devices fluctuates, and the communication link itself might fail. As a result, algorithms such as those based on Adaptive Bitrate (ABR) techniques that base their policy on heuristics or models of deployment perform sub-optimally in such dynamic environments . In addition, network operators can expect that the parameters that characterize the expected behavior of XR applications are heavy-tailed. Heaviness of tails is defined as the difference from the normal distribution in the proportion of the values that fall a long way from the mean . Such workloads require appropriate resource management policies to be used on the edge. The service requirements of XR applications are also challenging when compared to current video applications. In particular, several Quality-of-Experience (QoE) factors such as motion sickness are unique to XR applications and must be considered when operationalizing a network. This document examines these issues with the use case presented in the following section.

Use Case This use case involves an XR application running on a mobile device. Consider a group of tourists who are taking a tour around the historical site of the Tower of London. As they move around the site and within the historical buildings, they can watch and listen to historical scenes in 3D that are generated by the XR application and then overlaid by their XR headsets onto their real-world view. The headset continuously updates their view as they move around. The XR application first processes the scene that the walking tourist is watching in real time and identifies objects that will be targeted for overlay of high-resolution videos. It then generates high-resolution 3D images of historical scenes related to the perspective of the tourist in real time. These generated video images are then overlaid on the view of the real world as seen by the tourist. This processing of scenes and generation of high-resolution images are discussed in greater detail below.

Processing of Scenes The task of processing a scene can be broken down into a pipeline of three consecutive subtasks: tracking, acquisition of a model of the real world, and registration .

Tracking:

The XR application that runs on the mobile device needs to track the six-dimensional pose (translational in the three perpendicular axes and rotational about those three axes) of the user's head, eyes, and objects that are in view . This requires tracking natural features (for example, points or edges of objects) that are then used in the next stage of the pipeline.

Acquisition of a model of the real world:

The tracked natural features are used to develop a model of the real world. One of the ways this is done is to develop a model based on an annotated point cloud (a set of points in space that are annotated with descriptors) that is then stored in a database. To ensure that this database can be scaled up, techniques such as combining client-side simultaneous tracking and mapping with server-side localization are used to construct a model of the real world . Another model that can be built is based on a polygon mesh and texture mapping technique. The polygon mesh encodes a 3D object's shape, which is expressed as a collection of small flat surfaces that are polygons. In texture mapping, color patterns are mapped onto an object's surface. A third modeling technique uses a 2D lightfield that describes the intensity or color of the light rays arriving at a single point from arbitrary directions. Such a 2D lightfield is stored as a two-dimensional table. Assuming distant light sources, the single point is approximately valid for small scenes. For larger scenes, many 3D positions are additionally stored, making the table 5D. A set of all such points (either a 2D or 5D lightfield) can then be used to construct a model of the real world .

Registration:

The coordinate systems, brightness, and color of virtual and real objects need to be aligned with each other; this process is called "registration" . Once the natural features are tracked as discussed above, virtual objects are geometrically aligned with those features by geometric registration. This is followed by resolving occlusion that can occur between virtual and real objects . The XR application also applies photometric registration by aligning brightness and color between the virtual and real objects. Additionally, algorithms that calculate global illumination of both the virtual and real objects are executed. Various algorithms are also required to deal with artifacts generated by lens distortion , blur , noise , etc.

Generation of Images The XR application must generate a high-quality video that has the properties described above and overlay the video on the XR device's display. This step is called "situated visualization". A situated visualization is a visualization in which the virtual objects that need to be seen by the XR user are overlaid correctly on the real world. This entails dealing with registration errors that may arise, ensuring that there is no visual interference , and finally maintaining temporal coherence by adapting to the movement of user's eyes and head.

Technical Challenges and Solutions As discussed in , the components of XR applications perform tasks that are computationally intensive, such as real-time generation and processing of high-quality video content. This section discusses the challenges such applications can face as a consequence and offers some solutions. As a result of performing computationally intensive tasks on XR devices such as XR glasses, excessive heat is generated by the chipsets that are involved in the computation . Additionally, the battery on such devices discharges quickly when running such applications . A solution to problem of heat dissipation and battery drainage is to offload the processing and video generation tasks to the remote cloud. However, running such tasks on the cloud is not feasible as the end-to-end delays must be within the order of a few milliseconds. Additionally, such applications require high bandwidth and low jitter to provide a high QoE to the user. In order to achieve such hard timing constraints, computationally intensive tasks can be offloaded to edge devices. Another requirement for our use case and similar applications, such as 360-degree streaming (streaming of video that represents a view in every direction in 3D space), is that the display on the XR device should synchronize the visual input with the way the user is moving their head. This synchronization is necessary to avoid motion sickness that results from a time lag between when the user moves their head and when the appropriate video scene is rendered. This time lag is often called "motion-to-photon delay". Studies have shown that this delay can be at most 20 ms and preferably between 7-15 ms in order to avoid motion sickness . Out of these 20 ms, display techniques including the refresh rate of write displays and pixel switching take 12-13 ms . This leaves 7-8 ms for the processing of motion sensor inputs, graphic rendering, and round-trip time (RTT) between the XR device and the edge. The use of predictive techniques to mask latencies has been considered as a mitigating strategy to reduce motion sickness . In addition, edge devices that are proximate to the user might be used to offload these computationally intensive tasks. Towards this end, a 3GPP study suggests an Ultra-Reliable Low Latency of 0.1 to 1 ms for communication between an edge server and User Equipment (UE) . Note that the edge device providing the computation and storage is itself limited in such resources compared to the cloud. For example, a sudden surge in demand from a large group of tourists can overwhelm the device. This will result in a degraded user experience as their XR device experiences delays in receiving the video frames. In order to deal with this problem, the client XR applications will need to use ABR algorithms that choose bitrate policies tailored in a fine-grained manner to the resource demands and play back the videos with appropriate QoE metrics as the user moves around with the group of tourists. However, the heavy-tailed nature of several operational parameters (e.g., buffer occupancy, throughput, client-server latency, and variable transmission times) makes prediction-based adaptation by ABR algorithms sub-optimal . This is because with such distributions, the law of large numbers (how long it takes for the sample mean to stabilize) works too slowly and the mean of sample does not equal the mean of distribution ; as a result, standard deviation and variance are unsuitable as metrics for such operational parameters . Other subtle issues with these distributions include the "expectation paradox" (the longer the wait for an event, the longer a further need to wait) and the mismatch between the size and count of events . These issues make designing an algorithm for adaptation error-prone and challenging. In addition, edge devices and communication links may fail, and logical communication relationships between various software components change frequently as the user moves around with their XR device .

XR Network Traffic

Traffic Workload As discussed in Sections and , the parameters that capture the characteristics of XR application behavior are heavy-tailed. Examples of such parameters include the distribution of arrival times between XR application invocations, the amount of data transferred, and the inter-arrival times of packets within a session. As a result, any traffic model based on such parameters is also heavy-tailed. Using these models to predict performance under alternative resource allocations by the network operator is challenging. For example, both uplink and downlink traffic to a user device has parameters such as volume of XR data, burst time, and idle time that are heavy-tailed. below shows various streaming video applications and their associated throughput requirements . Since our use case envisages a 6 degrees of freedom (6DoF) video or point cloud, the table indicates that it will require 200 to 1000 Mbps of bandwidth. Also, the table shows that XR applications, such as the one in our use case, transmit a larger amount of data per unit time as compared to regular video applications. As a result, issues arising from heavy-tailed parameters, such as long-range dependent traffic and self-similar traffic , would be experienced at timescales of milliseconds and microseconds rather than hours or seconds. Additionally, burstiness at the timescale of tens of milliseconds due to the multi-fractal spectrum of traffic will be experienced . Long-range dependent traffic can have long bursts, and various traffic parameters from widely separated times can show correlation . Self-similar traffic contains bursts at a wide range of timescales . Multi-fractal spectrum bursts for traffic summarize the statistical distribution of local scaling exponents found in a traffic trace . The operational consequence of XR traffic having characteristics such as long-range dependency and self-similarity is that the edge servers to which multiple XR devices are connected wirelessly could face long bursts of traffic . In addition, multi-fractal spectrum burstiness at the scale of milliseconds could induce jitter contributing to motion sickness . This is because bursty traffic combined with variable queueing delays leads to large delay jitter . The operators of edge servers will need to run a "managed edge cloud service" to deal with the above problems. Functionalities that such a managed edge cloud service could operationally provide include dynamic placement of XR servers, mobility support, and energy management . Providing support for edge servers in techniques such as those described in , , and could guarantee performance of XR applications. For example, these techniques could be used for the link between the XR device and the edge as well as within the managed edge cloud service. Another option for network operators could be to deploy equipment that supports differentiated services or per-connection Quality-of-Service (QoS) guarantees using RSVP . Thus, the provisioning of edge servers (in terms of the number of servers, the topology, the placement of servers, the assignment of link capacity, CPUs, and Graphics Processing Units (GPUs)) should be performed with the above factors in mind. Throughput Requirements for Streaming Video Applications

Application	Throughput Required
Real-world objects annotated with text and images for workflow assistance (e.g., repair)	1 Mbps
Video conferencing	2 Mbps
3D model and data visualization	2 to 20 Mbps
Two-way 3D telepresence	5 to 25 Mbps
Current-Gen 360-degree video (4K)	10 to 50 Mbps
Next-Gen 360-degree video (8K, 90+ frames per second, high dynamic range, stereoscopic)	50 to 200 Mbps
6DoF video or point cloud	200 to 1000 Mbps

Traffic Performance Metrics The performance requirements for XR traffic have characteristics that need to be considered when operationalizing a network. These characteristics are discussed in this section. The bandwidth requirements of XR applications are substantially higher than those of video-based applications. The latency requirements of XR applications have been studied recently . The following characteristics were identified:

• The uploading of data from an XR device to a remote server for processing dominates the end-to-end latency.
• A lack of visual features in the grid environment can cause increased latencies as the XR device uploads additional visual data for processing to the remote server.
• XR applications tend to have large bursts that are separated by significant time gaps.

Additionally, XR applications interact with each other on a timescale of an RTT propagation, and this must be considered when operationalizing a network. shows a taxonomy of applications with their associated required response times and bandwidths (this data is from Table V in ). Response times can be defined as the time interval between the end of a request submission and the end of the corresponding response from a system. If the XR device offloads a task to an edge server, the response time of the server is the RTT from when a data packet is sent from the XR device until a response is received. Note that the required response time provides an upper bound for the sum of the time taken by computational tasks (such as processing of scenes and generation of images) and the RTT. This response time depends only on the QoS required by an application. The response time is therefore independent of the underlying technology of the network and the time taken by the computational tasks. Our use case requires a response time of 20 ms at most and preferably between 7-15 ms, as discussed earlier. This requirement for response time is similar to the first two entries in . Additionally, the required bandwidth for our use case is 200 to 1000 Mbps (see ). Since our use case envisages multiple users running the XR application on their devices and connecting to the edge server that is closest to them, these latency and bandwidth connections will grow linearly with the number of users. The operators should match the network provisioning to the maximum number of tourists that can be supported by a link to an edge server. Traffic Performance Metrics of Selected XR Applications

Application	Required Response Time	Expected Data Capacity	Possible Implementations/ Examples
Mobile XR-based remote assistance with uncompressed 4K (1920x1080 pixels) 120 fps HDR 10-bit real-time video stream	Less than 10 milliseconds	Greater than 7.5 Gbps	Assisting maintenance technicians, Industry 4.0 remote maintenance, remote assistance in robotics industry
Indoor and localized outdoor navigation	Less than 20 milliseconds	50 to 200 Mbps	Guidance in theme parks, shopping malls, archaeological sites, and museums
Cloud-based mobile XR applications	Less than 50 milliseconds	50 to 100 Mbps	Google Live View, XR-enhanced Google Translate

Conclusion In order to operationalize a use case such as the one presented in this document, a network operator could dimension their network to provide a short and high-capacity network path from the edge computing resources or storage to the mobile devices running the XR application. This is required to ensure a response time of 20 ms at most and preferably between 7-15 ms. Additionally, a bandwidth of 200 to 1000 Mbps is required by such applications. To deal with the characteristics of XR traffic as discussed in this document, network operators could deploy a managed edge cloud service that operationally provides dynamic placement of XR servers, mobility support, and energy management. Although the use case is technically feasible, economic viability is an important factor that must be considered.

IANA Considerations This document has no IANA actions.

Security Considerations The security issues for the presented use case are similar to those described in , , , and . This document does not introduce any new security issues.

Informative References APSys '14: 5th Asia-Pacific Workshop on Systems, pp. 1-7 Computer, vol. 50, no. 1, pp. 30-39 2019 IEEE International Conference on Edge Computing (EDGE), pp. 33-40 SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197-210 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI '20), pp. 495-511 John Wiley and Sons Revised Edition, STEM Academic Press Ubiquitous Computing Fundamentals, 1st Edition, Chapman and Hall/CRC Press, pp. 37-94 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 422-429 Computer Vision - ECCV 2014, pp. 16-31 IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 6, pp. 825-838 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 229-238 Computer Graphics Forum, vol. 15, no. 3, pp. 11-22 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 135-143 IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 4, pp. 573-580 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 133-141 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 39-44 Virtual Environments 2000, pp. 3-12 Eurographics 2012 - Short Papers, pp. 89-92 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 205-208 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 191-201 Sensors, vol. 20, no. 5, p. 1446 IEEE Communication Surveys and Tutorials, vol. 19 no. 4, pp. 2573-2620 APGV '04: Proceedings of the 1st Symposium on Applied perception in graphics and visualization, pp. 39-47 3GPP WWW '21: Proceedings of the Web Conference 2021, pp. 295-304 Road to VR Human Factors, vol. 54, no. 2, pp. 235-249 3GPP Addison-Wesley Professional Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 413-432 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 1-9 Synthesis Lectures on Network Systems Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385 ABI Research IEEE/ACM Transactions on Networking, vol. 3, no. 3, pp. 226-244 IEEE/ACM Transactions on Networking, vol. 5, no. 1, pp. 71-86 Applied and Computational Harmonic Analysis, vol. 10, no. 3, pp. 185-202 O'Reilly Media, Inc. IEEE Communications Surveys and Tutorials, vol. 23, no. 2, pp. 1160-1192 John Wiley and Sons Addison-Wesley NIST SANS Institute NIST

Acknowledgements Many thanks to , , , , , , , , , , , , , and for providing helpful feedback, suggestions, and comments.