Unicode Character Repertoire Subsets

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 4 September 2025.¶

Copyright Notice

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

When a protocol or data format has text fields, that text is normally composed of Unicode [UNICODE] characters, to support use by speakers of many languages. Unicode characters are represented by numeric code points, and the "set of all Unicode code points" is generally not a good choice for use in text fields. Unicode recognizes different types of code points, not all of which are appropriate in protocols, or even associated with characters. Therefore, even if the desire is to support "all Unicode characters" a subset of the Unicode code point repertoire should be specified. Subsets such as those discussed in this document are appropriate choices.¶

In this document, "subset" means a subset of the Unicode character repertoire. This document specifies subsets that exclude some or all of the code points that are "problematic" as defined in Section 2.2. Authors should have a way to concisely and exactly reference a stable specification that identifies which subset a protocol or data format accepts.¶

This document discusses issues that apply in choosing subsets, names two subsets that have been popular in practice, and suggests one new subset. The intended use is to serve as a convenient target for cross-reference from other specifications whose authors wish to exclude problematic code points from the data format or protocol being specified.¶

2. Characters and Code Points

Definition D9 in section 3.4 of [UNICODE] defines "Unicode codespace" as "a range of integers from 0 to 10FFFF₁₆". Definition D10 defines "code point" as "Any value in the Unicode codespace".¶

The Unicode Standard's definition of "Unicode character" is conceptual. However, each Unicode character is assigned a code point, used to represent the characters in computer memory and storage systems and, in specifications, to specify allowed subsets.¶

There are 1,114,112 (17 ⨉ 2**16) code points; as of Unicode 16.0 (2024), about 155,000 have been assigned to characters. Since unassigned code points regularly become assigned when new characters are added to Unicode, it is usually not a good practice to specify that unassigned code points should be avoided.¶

3. Dealing With Problematic Code Points

[RFC9413], "Maintaining Robust Protocols", provides a thorough discussion of strategies for dealing with issues in input data.¶

Different types of problematic code points cause different issues. Noncharacters and legacy controls are unlikely to cause software failures, but they cannot usefully be displayed to humans, and can be used in attacks based on attempting to display text that includes them.¶

The behavior of software which encounters surrogates is unpredictable and differs among programming-language implementations, even between different API calls in the same language.¶

Section 3.9 of [UNICODE] makes it clear that a UTF-8 byte sequence which would map to a surrogate is ill-formed. If a specification requires that input data be encoded with UTF-8, and if all input were well-formed, implementors would never have to concern themselves with surrogates.¶

Unfortunately, industry experience teaches that problematic code points, including surrogates, can and do occur in program input where the source of input data is not controlled by the implementor. In particular, the specification of JSON allows any code point to appear in object member names and string values [RFC8259].¶

For example, the following is a conforming JSON text:¶

{"example": "\u0000\u0089\uDEAD\uD9BF\uDFFF"}

The value of the "example" field contains the C0 control NUL, the C1 control "CHARACTER TABULATION WITH JUSTIFICATION", an unpaired surrogate, and the noncharacter U+7FFFF encoded per JSON rules as two escaped UTF-16 surrogate code points. It is unlikely to be useful as the value of a text field. That value cannot be serialized into well-formed UTF-8, but the behavior of libraries asked to parse the sample is unpredictable; some will silently parse this and generate an ill-formed UTF-8 string.¶

Two reasonable options for dealing with problematic input are either rejecting text containing problematic code points, or replacing the problematic code points with placeholders.¶

Silently deleting an ill-formed part of a string is a known security risk. Responding to that risk, [UNICODE] section 3.2 recommends dealing with ill-formed byte sequences by signaling an error, or replacing problematic code points, ideally with "�" (U+FFFD, REPLACEMENT CHARACTER).¶

4. Subsets

This section describes subsets that can be used in specifying acceptable content for text fields in protocols and data types. Specifications can refer to these subsets by the names "Unicode Scalars", "XML Characters", and "Unicode Assignables".¶

unicode-scalar =
   %x0-D7FF / %xE000-10FFFF  ; exclude surrogates

xml-character =
   %x9 / %xA / %xD /   ; useful controls
   %x20-D7FF /         ; exclude surrogates
   %xE000-FFFD/        ; exclude FFFE and FFFF nonchars
   %x100000-10FFFF

unicode-assignable =
   %x9 / %xA / %xD /               ; useful controls
   %x20-7E /                       ; exclude C1 controls and DEL
   %xA0-D7FF /                     ; exclude surrogates
   %xE000-FDCF                     ; exclude FDD0 nonchars
   %xFDF0-FFFD /                   ; exclude FFFE and FFFF nonchars
   %x10000-1FFFD / %x20000-2FFFD / ; (repeat per plane)
   %x30000-3FFFD / %x40000-4FFFD /
   %x50000-5FFFD / %x60000-6FFFD /
   %x70000-7FFFD / %x80000-8FFFD /
   %x90000-9FFFD / %xA0000-AFFFD /
   %xB0000-BFFFD / %xC0000-CFFFD /
   %xD0000-DFFFD / %xE0000-EFFFD /
   %xF0000-FFFFD / %x100000-10FFFD

5. Using Subsets

Many IETF specifications rely on well-known data formats such as JSON, I-JSON, CBOR, YAML, and XML. These formats specify default subsets. For example, JSON allows object member names and string values to include any Unicode code point, including all the problematic types.¶

A protocol based on JSON can be made more robust and implementor-friendly by restricting the contents of object member names and string values to one of the subsets described in Section 4. Equivalent restrictions are possible for other packaging formats such as I-JSON, XML, YAML, and CBOR.¶

Note that escaping techniques such as those in the JSON example in Section 3 cannot be used to circumvent this sort of restriction, which applies to data content, not textual representation in packaging formats. If a specification restricted a JSON field value to the Unicode Assignables, the example would remain a conforming JSON Text but the data it represents would not constitute Unicode Assignable code points.¶

6. IANA Considerations

This document has no actions for IANA.¶

7. Security Considerations

Section 3 of this document discusses security issues.¶

Unicode Security Considerations [TR36] is a wide-ranging survey of the issues implementors should consider while writing software to process Unicode text. Unicode Source Code Handling [TR55] discusses use of Unicode in programming languages, with a focus on security issues. Many of the attacks they discuss are aimed at deceiving human readers, but vulnerabilities involving issues such as surrogates and noncharacters are also covered, and in fact can contribute to human-deceiving exploits.¶

The Security Considerations in Section 12 of [RFC8264] generally applies to this document as well.¶

Note that the Unicode-character subsets specified in this document successively exclude an increasing number of problematic code points, and thus should be less and less susceptible to many of these exploits. The Section 4.3 subset, "Unicode Assignables", excludes all of these code points.¶

Acknowledgements

Thanks are due to Guillaume Fortin-Debigaré, who filed an Errata Report against RFC 8259, The JavaScript Object Notation, noting frequent references to "Unicode characters", when in fact the RFC formally specifies the use of Unicode Code Points.¶

Thanks also to Asmus Freytag for careful review and many constructive suggestions aimed at making the language more consistent with the structure of the Unicode Standard.¶

Thanks also to James Manger for the correctness of the ABNF and JSON samples.¶

Thanks also to Addison Phillips and the W3C Internationalization Working Group for helpful suggestions on language and references.¶

This document got a great deal of thoughtful discussion during the late stages of review which helped tighten up wording and make difficult points clearer.¶

Authors' Addresses