TIP #388: Extending Unicode literals past the BMP
_____________________________________________________________________________
TIP:	388
Title:	Extending Unicode literals past the BMP
Version:	$Revision: 1.7 $
Authors:	Jan Nijtmans <jan.nijtmans@users.sf.net>
	Jan Nijtmans <jan.nijtmans@gmail.com>
State:	Final
Type:	Project
Tcl-Version:	8.6
Vote:	Done
Created:	Wed Aug 10 00:00:00 UTC 2011
Keywords:	Tcl
Abstract
This TIP proposes to extend Tcl's syntax in order to be able to cope with quoted forms of Unicode characters outside the Basic Multilingual Plane.
Summary
Tcl provides backslash substitutions of the form  for unicode characters, but this form is not sufficient to model unicode literals past the BMP. The outcome of the discussion on Tcl-Core was to add the form  (one up to 8 hexadecimal digits), but still it is ambiguous how characters > 0x10ffff are to be handled. This TIP is meant to sort that out, it is not meant to specify how characters outside the BMP are handled. The reference implementation just replaces any character in the range  - ffff with , but as soon as Tcl has support for characters outside the BMP this range is reserved for exactly that.
Currently, the form  is parsed by Tcl as a literal U, so - however small - this change results in a non-trivial potential incompatibility which therefore requires a TIP.
Considering backslash sequences, there are two other forms which are currently not consistent:  accepts an unlimited number of hex digits, unlike other modern languages, and the form , where the first octal digit is in the range 4..7 is currently not handled consistently in Tcl. Now is an opportunity to reconsider this.
In tcl.h there is a remark regarding the possible values of TCL_UTF_MAX:
3 Currently the only supported value, defining Tcl_UniChar as unsigned short
6 Not supported, but reserved for a hypothetical 32-bit Unicode
1 Not supported, possibly for a ASCII-only variant of Tcl.
This document proposes to add another value:
4 Not supported. The same as 3, but allowing the use of Unicode surrogate pairs to represent the range  - ffff
Rationale
Consider the string 01, how is that supposed to be interpreted? Tcl specifies octal sequences as 8 bits, and silently strips the 9th bit, the same as gcc does. In Tcl's regular expression engine, the 9th bit is not stripped, there it is equivalent to 1. Java - for example - parses it as 0 - a valid 8-bit octal value - followed by 1, so it's a string of length 2.
Then the string . Tcl specifies this as 8 bits as well, and silently strips all higher bits, so it is equivalent to ". This is the same as gcc does, but Java - for example - considers it as  followed by 34, so it's a string of lenght 3.
Consider the string , which would result in an invalid Unicode character. In the Tcl parser we don't have the possibility to flag invalid backslash sequences, in Tcl's regexp engine we have. Unicode characters higher than ffff cannot appear in an UTF-8 stream.
In tcl.h, we find Tcl_UniChar to be defined as unsigned int when TCL_UTF_MAX > 3 and as unsigned short otherwise. It would be useful to reserve the value 4 and still define Tcl_UniChar as unsigned short. That would allow the path to a full support for out-of BMP Unicode characters shorter, because Unicode Surrogate pairs can be used for that.
Specification
This document proposes:
Change the parsers in Tcl to handle octal sequences higher than 77 differently, splitting it in a two-digit valid octal secuence and handling the additional character separately. So 01 is handled as the valid sequence 0 followed by 1. This is a potential incompatibility.
Change the parsers in Tcl to handle the  sequence to parse just 2 digits, and not silently strip all higher hex digits any more. This is a potential incompatibility.
Add the  handling, similar to the  handling, only accepting up to 8 characters. The parser will stop parsing earlier when a code point  or higher is reached, as shifting it 4 bits more will lead to a code point outside the Unicode range. The regexp engine already handles , but currently it always generates a character in the BMP and strips all higher bits. This is a potential incompatibility.
Compatibility
Tcl scripts using the form  where the first digit is in the range 4-7, will now interpred the string as  followed by o. There is currently no test case in the Tcl test suite affected by that.
Tcl scripts using the form , which used to strip off the higher digits, will now end the sequence when two digits are handled.
Tcl scripts using  as a literal U will no longer work when it is followed with at least one hexadecimal digit. There is currently no test case in the Tcl test suite affected by that.
Alternatives
Unicode Noncharacters or Surrogages are not supposed to be in an UTF-8 stream, the Unicode standard recommends to replace those with . This TIP does not propose that, because they may be used internally e.g. as part of a character range in a regular expression. A better place to do that is in the Utf8-To-Utf8 conversion, but that's outside the scope of this TIP.
Reference Implementation
A reference implementation is available at http://core.tcl.tk/tcl in branch tip-388-impl
Copyright
This document has been placed in the public domain.
_____________________________________________________________________________
Colophon
TIP AutoGenerator - written by Donal K. Fellows