url.spec.whatwg.orgURL Standard

URL Living Standard — Last Updated 1 April 2020 Participate: GitHub whatwg/url ( new issue , open issues ) IRC: #whatwg on Freenode Commits: GitHub whatwg/url/commits Snapshot as of this commit @urlstandard Tests: web-platform-tests url/ ( ongoing work ) Translations (non-normative) : 日本語 Abstract The defines URLs, domains, IP addresses, the application/x-www-form-urlencoded format, and their API. Table of Contents Goals 1 Infrastructure 1.1 Writing 1.2 Parsers 1.3 Percent-encoded bytes 2 Security considerations 3 Hosts (domains and IP addresses) 3.1 Host representation 3.2 Host miscellaneous 3.3 IDNA 3.4 Host writing 3.5 Host parsing 3.6 Host serializing 3.7 Host equivalence 4 URLs 4.1 URL representation 4.2 URL miscellaneous 4.3 URL writing 4.4 URL parsing 4.5 URL serializing 4.6 URL equivalence 4.7 Origin 4.8 URL rendering 4.8.1 Simplify non-human-readable or irrelevant components 4.8.2 Elision 4.8.3 Internationalization and special characters 5 application/x-www-form-urlencoded 5.1 application/x-www-form-urlencoded parsing 5.2 application/x-www-form-urlencoded serializing 5.3 Hooks 6 API 6.1 URL class 6.2 URLSearchParams class 6.3 URL APIs elsewhere Acknowledgments Index Terms defined by this specification Terms defined by reference References Normative References Informative References IDL Index Goals The URL standard takes the following approach towards making URLs fully interoperable: Align RFC 3986 and RFC 3987 with contemporary implementations and obsolete them in the process. (E.g., spaces, other "illegal" code points, query encoding, equality, canonicalization, are all concepts not entirely shared, or defined.) URL parsing needs to become as solid as HTML parsing. [RFC3986] [RFC3987] Standardize on the term URL. URI and IRI are just confusing. In practice a single algorithm is used for both so keeping them distinct is not helping anyone. URL also easily wins the search result popularity contest . Supplanting Origin of a URI [sic] . [RFC6454] Define URL’s existing JavaScript API in full detail and add enhancements to make it easier to work with. Add a new URL object as well for URL manipulation without usage of HTML elements. (Useful for JavaScript worker environments.) Ensure the combination of parser, serializer, and API guarantee idempotence. For example, a non-failure result of a parse-then-serialize operation will not change with any further parse-then-serialize operations applied to it. Similarly, manipulating a non-failure result through the API will not change from applying any number of serialize-then-parse operations to it. As the editors learn more about the subject matter the goals might increase in scope somewhat. 1. Infrastructure This specification depends on the Infra Standard. [INFRA] Some terms used in this specification are defined in the following standards and specifications: DOM Standard [DOM] Encoding Standard [ENCODING] File API [FILEAPI] HTML Standard [HTML] Media Source Extensions [MEDIA-SOURCE] Unicode IDNA Compatibility Processing [UTS46] Web IDL [WEBIDL] To serialize an integer , represent it as the shortest possible decimal number. 1.1. Writing A validation error indicates a mismatch between input and valid input. User agents, especially conformance checkers, are encouraged to report them somewhere. A validation error does not mean that the parser terminates. Termination of a parser is always stated explicitly, e.g., through a return statement. It is useful to signal validation errors as error-handling can be non-intuitive, legacy user agents might not implement correct error-handling, and the intent of what is written might be unclear to other developers. 1.2. Parsers The EOF code point is a conceptual code point that signifies the end of a string or code point stream. Within a parser algorithm that uses a pointer variable, c references the code point the pointer variable points to. Within a string-based parser algorithm that uses a pointer variable, remaining references the substring after pointer in the string being processed. If " mailto:username@example " is a string being processed and pointer points to @, c is U+0040 (@) and remaining is " example ". 1.3. Percent-encoded bytes A percent-encoded byte is U+0025 (%), followed by two ASCII hex digits . Sequences of percent-encoded bytes , after conversion to bytes, should not cause UTF-8 decode without BOM or fail to return failure. To percent encode a byte into a percent-encoded byte , return a string consisting of U+0025 (%), followed by two ASCII upper hex digits representing byte . To percent decode a byte sequence input , run these steps: Using anything but UTF-8 decode without BOM when the input contains bytes that are not ASCII bytes might be insecure and is not recommended. Let output be an empty byte sequence . For each byte byte in input : If byte is not 0x25 (%), then append byte to output . Otherwise, if byte is 0x25 (%) and the next two bytes after byte in input are not in the ranges 0x30 (0) to 0x39 (9), 0x41 (A) to 0x46 (F), and 0x61 (a) to 0x66 (f), all inclusive, append byte to output . Otherwise: Let bytePoint be the two bytes after byte in input , decoded , and then interpreted as hexadecimal number. Append a byte whose value is bytePoint to output . Skip the next two bytes in input . Return output . To string percent decode a string input , run these steps: Let bytes be the UTF-8 encoding of input . Return the percent decoding of bytes . The C0 control percent-encode set are the C0 controls and all code points greater than U+007E (~). The fragment percent-encode set is the C0 control percent-encode set and U+0020 SPACE, U+0022 ("), U+003C (<), U+003E (>), and U+0060 (`). The path percent-encode set is the fragment percent-encode set and U+0023 (#), U+003F (?), U+007B ({), and U+007D (}). The userinfo percent-encode set is the path percent-encode set and U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), U+005B ([), U+005C (\), U+005D (]), U+005E (^), and U+007C (|). To UTF-8 percent encode a codePoint , using a percentEncodeSet , run these steps: If codePoint is not in percentEncodeSet , then return codePoint . Let bytes be the result of running UTF-8 encode on codePoint . Percent encode each byte in bytes , and then return the results concatenated, in the same order. 2. Security considerations The security of a URL is a function of its environment. Care is to be taken when rendering, interpreting, and passing URLs around. When rendering and allocating new URLs "spoofing" needs to be considered. An attack whereby one host or URL can be confused for another. For instance, consider how 1/l/I, m/rn/rri, 0/O, and а/a can all appear eerily similar. Or worse, consider how U+202A LEFT-TO-RIGHT EMBEDDING and similar code points are invisible. [UTR36] When passing a URL from party A to B , both need to carefully consider what is happening. A might end up leaking data it does not want to leak. B might receive input it did not expect and take an action that harms the user. In particular, B should never trust A , as at some point URLs from A can come from untrusted sources. 3. Hosts (domains and IP addresses) At a high level, a host , valid host string , host parser , and host serializer relate as follows: The host parser takes an arbitrary string and returns either failure or a host . A host can be seen as the in-memory representation. A valid host string defines what input would not trigger a validation error or failure when given to the host parser . I.e., input that would be considered conforming or valid. The host serializer takes a host and returns a string. (If that string is then parsed , the result will equal the host that was serialized .) 3.1. Host representation A host is a domain , an IPv4 address , an IPv6 address , an opaque host , or an empty host . Typically a host serves as a network address, but it is sometimes used as opaque identifier in URLs where a network address is not necessary. The RFCs referenced in the pa...