TIP #412: Dynamic Locale Changing for msgcat with On-Demand File Load
_____________________________________________________________________________
TIP:	412
Title:	Dynamic Locale Changing for msgcat with On-Demand File Load
Version:	$Revision: 1.9 $
Authors:	Harald Oehlmann <harald.oehlmann@elmicron.de>
	Harald Oehlmann <oehhar@users.sourceforge.net>
State:	Final
Type:	Project
Tcl-Version:	8.6
Vote:	Done
Created:	Tue Mar 27 00:00:00 UTC 2012
Keywords:	Tcl, localization, msgcat
Abstract
This TIP adds dynamic locale switching capabilities to the msgcat package.
Rationale
Dynamic Locale Switching
Within a multi-language application like a web-server, one may change the locale quite frequently, for example if users with different locales are requesting pages. Unfortunately, this does not fit well with the model adopted by the msgcat package, which assumes that all code follows this sequence:
1.	 Set locale list: mclocale locale
2.	 Load language files with other package load: mcload msg-folder
3.	 Translate strings: mc key args...
Note that if the locale should be changed after other packages are loaded, one must restart at step 2. This requires reloading all packages which is mostly not practical.
The aim of this TIP is to extend the package by dynamic locale change capabilities.
msgcat will reload any missing message catalog files of all currently loaded packages on a locale change. In addition, any package may register to get informed to a locale change. Other packages may do changes to reflect the locale change like rebuilding the GUI.
This TIP compares to [399] that the package is able to load message catalog files on demand, e.g. specially on a locale change.
package locale
If the clock command gets called with the argument "-locale", the locale is changed using msgcat::mclocale. After processing, the initial value is restored. The package keeps track, which locales where already used and calls msgcat::mcload for any new locale. The locale is restored after processing.
This is an implementation of dynamic locales but conflicts with the new features described above. Other packages may be informed to change the locale and may trigger expensive operations like a rebuild of the GUI.
In consequence, each package may define a package locale which is independent of the default locale.
Overview of the proposed solution
Proposed changes in brief:
Dynamically load message catalog files
if the locale is changed by mclocale locale, the message file load process is executed for every present package.
Package locale
A package may install a package local locale which is independent to the global locale.
Locale change callback
A callback may be registered to get informed about the change of locale. A use case is to refresh a GUI if the locale changed.
Non message file operation
A program may use message files to issue mcset commands or may issue them by other means, if the message catalogs are, for example, stored in a data base.
Each package may register a callback to get informed that a certain locale should be loaded and may issue the corresponding mcset commands.
Package mcunknown
A package may have a certain way to provide translations for message keys not included in the message catalog. Thus, it may register an own package message unknown callback to provide a translation.
Specification
Package Equals Client Namespace
A client package is a package which uses msgcat. A unique namespace is required for each client package. Within msgcat, namespace and package is always connected.
Up to now, the msgcat package used this namespace as an identifier to store the catalog data of a certain package. This is now extended to additional properties which are stored for a package.
Package locale
A package locale may be used by a package instead the default locale set by msgcat::mclocale. A package may choose to use a package locale or the default locale.
Default and Package State
Some state values (like the locale) are available as default (global) values. In addition, each package may choose to use a package locale state.
The used naming is:
 default
state	valid for all packages which do not set a package state.
 package
state	only valid for one package if it has set a package state.
The following state values are present as default state and may be set individually per package:
 The locale like "de_ch". The preferences property is a list of locales in their preference order and is automatically computed from locale. Example locale = "de_ch" -> preferences = "de_ch de ". The loadedlocales state value is the list of currently loaded locales.
Default State
The following standard methods exist to get or set the default state:
msgcat::mclocale
The default locale. It may be read using msgcat::mclocale.
It may be set using msgcat::mclocale locale. This command is extended, that the message catalogs of all missing locales for all packages not having set a package state are loaded.
msgcat::mcpreferences
Get the default preferences (derived from the default locale).
msgcat::mcloadedlocales
The following new command may be used to deal with the default state:
msgcat::mcloadedlocales subcommand ?locale?
The parameter locale is mandatory for the subcommand present.
The following subcommands are available:
Subcommand "get"
Get the list of current loaded locales
Subcommand "present"
Returns true, if the given locale is loaded
Subcommand "clear"
The list of currently loaded locales is set to mcpreferences and all message catalog keys of packages without a package locale set and with locales not in mcpreferences are unset.
Package Configuration
The package configuration of the calling package may be changed using the following new command:
msgcat::mcpackagelocale subcommand ?locale?
The parameter locale is mandatory for the subcommands set and present.
Available subcommands are:
Subcommand "set"
Set or change the package locale.
The global state values are copied, if there were no package locale set before.
The package locale is changed to the optional given new package locale.
Subcommand "get"
Return the package locale or the default locale, if no package locale set.
Subcommand "preferences"
Return the package preferences or the default preferences, if no package locale set.
Subcommand "loaded"
The list of locales loaded for this package is returned.
Subcommand "isset"
Returns true, if a package locale is set.
Subcommand "unset"
Unset the package locale and use the default state for the package. Load all message catalog files of the package for locales, which were not present in the package loadedlocales list and are present in the default list.
Subcommand "present"
Returns true, if the given locale is loaded
Subcommand "clear"
Set the current loaded locales list of the package to preferences and unset all message catalog keys of the package with locales not included in the package preferences.
Package Configuration Options
Each package may have a set of configuration options set to invoke certain actions. They may be retrieved or changed with the following new command:
msgcat::mcpackageconfig subcommand option ?value?
Available subcommands are:
get	Get the current value of the option or an error if not set.
isset	Returns true if option is set.
set	Set the given value to the option. May have additional consequences and return values as described in the option section.
unset	Unset the option.
Available options are:
Package Option "mcfolder"
This is the message folder of the package. This option is set by mcload and by the subcommand set. Both are identical and both return the number of loaded message catalog files.
Setting or changing this value will load all locales contained in the preferences valid for the package. This implies also to invoke any set loadcmd (see below).
Unsetting this value will disable message file load for the package.
If the locale valid for this package changes, this value is used to eventually load message catalog files.
Message catalog files are always sourced in the namespace of the package registering the value.
Package Option "loadcmd"
This callback is invoked before a set of message catalog files are loaded for the package which has this property set.
This callback may be used to do any preparation work for message file load or to get the message data from another source like a data base. In this case, no message files are used (mcfolder is unset).
See chapter callback invocation below. The parameter list appended to this callback is the list of locales to load.
If this callback is changed, it is called with the preferences valid for the package.
Package Option "changecmd"
This callback is invoked when a default local change was performed. Its purpose is to allow a package to update any dependency on the default locale like showing the GUI in another language.
Tk may be extended to register to this callback and to invoke a virtual event.
See the callback invocation section below. The parameter list appended to this callback is mcpreferences. All registered packages are invoked in no particular order.
Package Option "unknowncmd"
Use a package locale mcunknown procedure instead of the standard version supplied by the msgcat package (msgcat::mcunknown).
The called procedure must return the formatted message which will finally be returned by msgcat::mc.
A generic unknown handler is used if set to the empty string. This consists in returning the key if no arguments are given. With given arguments, format is used to process the arguments.
See chapter callback invocation below. The appended arguments are identical to mcunknown.
Callback Invocation
Callbacks are invoked under the following conditions:
 the callback command is set, the command is not the empty string, the registration namespace exists.
Any error within the callback stops the operation which invoked the callback. This might be surprising, as the error might be in another package.
Test if Message Key is Set
Message catalog keys may be expensive to calculate and thus may be set on demand.
The following new procedure returns false, if mc would call mcunknown for a key:
msgcat::mcexists src
There are two options, to limit the key search to just the current namespace (don't search in parent namespaces) and just the current locale (don't search the preferences but the first item):
msgcat::mcexists ?-exactnamespace? ?-exactlocale? src
forget package
A package may clear all its keys and state using the new command:
msgcat::mcforgetpackage
Locale and Preferences Format
Locales set by mcset may eventually not correspond to the current preferences, as the preferences are treated as follows:
 put to lower case, remove any multiple "_" and any "_" at the beginning or at the end of the
 locale.
It is proposed, that:
 the locale and the first preferences element is always identical to the
 lowercase passed locale,
 any multiple "_" are seen as one separator.
Example: preferences of locale "sy__cyrl_win"
current preferences	"sy_cyrl_win sy_cyrl sy" proposed preferences: "sy__cyrl_win sy__cyrl sy".
Alternatively, all locales may normalized using the upper algorithm, which felt heavy in computation with little gain.
Example Usage
Example from TIP #399
Imagine an application which supports the current user language and French, German and English. An external package tp is used. The package uses msgcat and installs itself during the package require tp call:
package require msgcat
msgcat::mcload [file join [file dirname [info script]] msgs]
An implementation of the application with the current msgcat 1.5.0 would require the following initialization sequence:
package require msgcat
package require np
and the following code to change the locale to French:
package forget np
msgcat::mclocale fr
package require np
Using the extension of this TIP, one may load as usual:
package require msgcat
package require np
and to change to french locale:
msgcat::mclocale fr
The first time, a locale is required, all corresponding message files of all packages which use msgcat get loaded. This might be a heavy operation.
If a locale is reactivated (and the message catalog data was not cleared), it is a quick operation.
Without this TIP, it is computational expensive (if possible, as many packages are not reloadable or a reload may disturb current processing, e.g., by forcing the closing of sockets, etc.).
Change with No Need to Come Back
If it is certain that a locale is changed and the then obsolete data is of no use, one may clear unused message catalog items:
msgcat::mclocale fr
msgcat::mcloadedlocale clear
Use a Callback to be Notified About a Locale Change
Packages which display a GUI may update their widgets when the locale changes. To register to a callback, use:
namespace eval gui 
 msgcat::mcpackageconfig changecmd updateGUI

 proc updateGui args 
 puts "New locale is '[lindex $args 0]'."

% msgcat::mclocale fr
fr
% New locale is 'fr'.
To Use Another Locale Source than Message Catalog Files
If locales (or additional locales) are contained in another source like a data base, a package may use the load callback and not mcload:
namespace eval db 
 msgcat::mcpackageconfig loadcmd loadMessages
 msgcat::mcconfig loadedpackages\par [concat [msgcat::mcconfig loadedpackages] namespace current]

 proc loadMessages args 
 foreach locale $args 
 if [LocaleInDB $locale] 
 msgcat::mcmset $locale [GetLocaleList $locale]

 

Use a package locale
The reference implementation also contains a changed clock command which uses a package locale. Here are some sketches from the implementation.
First, a package locale is initialized and the generic unknown function is activated:
msgcat::mcpackagelocale set
msgcat::mcpackageconfig unknowncmd ""
If the user requires the week day in a certain locale, it is changed:
clock format clock seconds -format %A -locale fr
and the code:
msgcat::mcpackagelocale set $locale
return [lindex [msgcat::mc DAYS_OF_WEEK_FULL] $day]
### Returns "mercredi"
Some message-catalog items are heavy in computation and thus are dynamically cached using:
proc ::tcl::clock::LocalizeFormat  locale format  
 set key FORMAT_$format
 if  [::msgcat::mcexists -exactlocale -exactnamespace $key]  
 return [mc $key]
 
 #...expensive computation of format clipped...
 mcset $locale $key $format
 return $format

Reference Implementation
See Tcl fossil tag msgcat_dyn_locale [1].
Compatibility
Imagined incompatibilities:
 If packages call mcload multiple times with different folders, the
 data was currently appended. This is still the case, but only the last folder is used for any reload. The property mcfolder may be transformed to a list to cover this case.
 The return value of mcload (file count) may be much higher as there
 may be loaded much more files. I suppose, this value is only used by the test suite to verify functionality and is not for big general use.
 Message files may not be aware, that they may be loaded at any moment and
 not only after their own mcload. I suppose, this is the biggest issue but I think, there is no alternative.
 Message files do not get reloaded any more, if a second mcload is
 issued with the same path argument.
 Package which temporary change the default locale trigger any callback
 and may lead to user visible side effects.
Issues
Known issues:
 Packages might not be aware of a locale change and may buffer
 translations outside of msgcat. Packages should not buffer msgcat messages if they are used in a dynamic locale application (like tklib tooltip does for example).
 The clock command currently has a small dynamic patch for msgcat
 implemented. This must be removed in favor to new msgcat features due to the temporarily change of the default locale.
Extensions
 Expose the function to calculate the preference list from a given locale. Load a message catalog file for a given locale without changing the
 default/package locale.
 Methods isloaded to check if a locale is currently loaded. Access message catalog with specified namespace, locale and search
 behavior.
Alternatives
The alternative is the former [399], but that is problematic because the list of locales must be known before any package load. The additional complexity of this TIP is a justifiable trade-off against the greatly improved flexibility in the loading and locale selection order.
Copyright
This document has been placed in the public domain.
_____________________________________________________________________________
Colophon
TIP AutoGenerator - written by Donal K. Fellows