Rapid Application Development toolkit for building Administrative Web Applications

Internationalisation and the Radicore Development Infrastructure (Part 1)

By Tony Marston

15th July 2005
Amended 1st February 2008

As of 10th April 2006 the software discussed in this article can be downloaded from www.radicore.org

Introduction
Possible Methods
Design Decisions
My Implementation
- Directory Structure
- File Names
- File Contents
- Determine User Language
- Locate Language Subdirectory
- Load Screen Structure file
- Get Language Text
- Get Language Array
- Handling Dates
- Handling Numbers
-- Convert to external (user's) format
-- Convert to internal format
- Character Encoding
Conclusion
References
Amendment History

Introduction

The term "internationalisation" is sometimes referred to as "globalisation" or "localisation", but what does it actually mean? The following description is taken from java.sun.com:

Internationalisation is the process of designing an application so that it can be adapted to various languages and regions without engineering changes. Sometimes the term internationalisation is abbreviated as i18n, because there are 18 letters between the first "i" and the last "n."

An internationalised program has the following characteristics:

Internationalisation in a software application covers the ability to communicate with a user in his/her own language. It can be said to exist at the following levels:

Level 1 is supported in the Radicore framework by having the text for such things as screen titles, field labels, button labels and error messages contained in text files which are separate from the program code. Each set of files contains text in a single language and is held in a subdirectory whose name identifies that language. Each supported language therefore has a copy of these files in its own subdirectory. The framework will detect the user's preferred language, and will access the text files in the appropriate subdirectory. The details are explained in the following sections of this document.

Level 2 is supported in the Radicore framework by maintaining translated text in separate tables within the application database. The framework will detect the user's preferred language, and will retrieve either the native text or the translated text as appropriate. Please refer to Internationalisation and the Radicore Development Infrastructure (Part 2) for full details.

Possible Methods

There are several ways in which text in language 'A' can be replaced with text language 'B'. Before a solution can be designed it is necessary to examine the range of possibilities and weigh up the pros and cons of each one.

  1. Are you going to run strings of text through a general-purpose translator, or replace one identifiable string with another?
  2. Are you going to perform the translation/substitution as early as possible (i.e. as soon as you know what text needs to be output), or as late as possible (i.e. just before it is presented to the user)?
  3. Are you going to put text into the output area and then translate it, or translate it first and then put it into the output area?
  4. Are you going to identify each piece of text as a complete string, or give each one a smaller identity code?
  5. Are you going to store the language variations in a database or in text files?
  6. Are you going to put all the language variations into a single file, or have a separate file for each language?
  7. If you use XML and XSL to produce all HTML output (as I do in my development infrastructure) could you perform all the translation during the XSL transformation?

Design Decisions

While reviewing the possible options I made the following decisions:

My Implementation

Directory Structure

My main development infrastructure consists of a series of discrete subsystems each of which has its own directory in the file system. (Note that my sample application consists of just a single subsystem). Each of these directories will contain the following new subdirectories:

Each of these new directories will be further broken down into subdirectories where the subdirectory name matches a language code, such as:

All subdirectories except 'en' English (my native language) are optional. The files in the 'en' subdirectories identify every piece of text which can be translated, so these should be used as the patterns for any non-English translations.

Each language subdirectory will contain a copy of the file(s) containing the text for that particular language code. If there is no text for a particular language code then there should be no subdirectory for that language code - in other words, there should not be any language subdirectories which are empty.

I also decided to split the existing 'screens' directory (for the screen structure files) into language subdirectories with the intention to use these files to contain the translated field label text, but when I realised that most of the labels were duplicated I decided that it would be easier to define each field label just once in the file which contains all the other translatable text. However, I decided to keep the facility for separate versions of the screen structure files for different languages just in case it should be necessary to adjust more than the label text to suit the needs of a particular language. If no adjustments need to be made to any of the screen structure files then no additional language subdirectories will be necessary as everything will still work successfully with the original files in the 'en' subdirectory - the structures will be identical, but the label text will still be translated.

File Names

Within the 'help' directory there will be a separate file for each PHP script with the suffix '.help.txt'. For example, the script 'person_list.php' will have 'person_list.php.help.txt'. In my full development infrastructure I will use the task_id instead of the script_id.

Within the 'screens' subdirectory there will be a separate file for each screen structure with the suffix '.screen.inc'.

Within the 'text' subdirectory there will be the following files:

  1. A file called 'language_text.inc' that will contain all the translations for that application in that language.
  2. A file called 'language_array.inc' that will contain all the arrays of values that are normally used in picklists (dropdown lists or radio groups) where the key (as referenced internally) remains consistent, but where the value (as seen by the user) may be expressed in different languages.

For each installation there will also be a 'sys.language_text.inc' and 'sys.language_array.inc' to contain all the translatable text required by the system libraries. In the sample application these will reside in the 'sample/text/' directory, but in my full infrastructure these will reside in the 'menu/text/' directory. These 'system' files contains the text that may be used by any of the system libraries (controller scripts, validation class, generic table class and DML class) regardless of the application in which any particular component belongs.

This means that:

File Contents

language_array.inc

This is for storing arrays of values that will be used for such things as dropdown lists and radio groups. Each entry should look something like the following:

$array['direction'] = array('L' => 'Left',
                            'R' => 'Right',
                            'U' => 'Up'
                            'D' => 'Down');

Entries from this file can be extracted using the getLanguageArray() function.

language_text.inc

This should have several sections, one for each category of text.

The first section is for all your error messages, such as:

// application error messages
$array['e0001'] = "This is error message number 1";
$array['e0002'] = "This is error message number 2";
$array['e0003'] = "This is error message number 3";
....
$array['e0099'] = "This is error message number 99";

The second, third and fourth sections are for text that can be extracted from the MENU database. Once you have finished entering or amending your details in the MENU database you can go to the List Subsystem screen and press the Export button. One of the files it creates will be <subsys>.menu_export.txt, and the contents can be copied directly into the language_text.inc file which deals with your native language.

// menu details for subsystem MENU
$array['Audit']                         = 'Audit';
$array['Dialog Type']                   = 'Dialog Type';
$array['Dictionary']                    = 'Dictionary';
$array['Menu Controls']                 = 'Menu Controls';
$array['Menu System']                   = 'Menu System';
$array['Role']                          = 'Role';
$array['Subsystem']                     = 'Subsystem';
$array['Task (All)']                    = 'Task (All)';
$array['Task (Menu)']                   = 'Task (Menu)';
$array['Task (Proc)']                   = 'Task (Proc)';
$array['ToDo']                          = 'ToDo';
$array['User']                          = 'User';
$array['Workflow']                      = 'Workflow';

// task details for subsystem MENU
$array['logon']                         = 'Logon screen';
$array['menu']                          = 'Home Page';
$array['mnu_control(upd)']              = 'Maintain Menu Controls';
$array['mnu_help_text(multi)']          = 'Maintain Help Text';
$array['mnu_menu(add)']                 = 'Add Menu Items';
$array['mnu_menu(del)']                 = 'Delete Menu Items';
$array['mnu_menu(link)']                = 'Maintain Menu Items (2)';
$array['mnu_menu(list)']                = 'List Menu Items';
$array['mnu_menu(multi)']               = 'Maintain Menu Items (1)';
$array['mnu_menu(search)']              = 'Search Menu Items';
$array['mnu_menu(upd)']                 = 'Update Menu Item';

// navigation button details for subsystem MENU
$array['Change Password']               = 'Change Password';
$array['Export']                        = 'Export';
$array['Field Access']                  = 'Field Access';
$array['Help Text']                     = 'Help Text';
$array['List Fields']                   = 'List Fields';
$array['List Task']                     = 'List Task';
$array['List User']                     = 'List User';

The fifth section is for field labels. These are defined in your various screen structure files, and in your native language each key will probably be the same as the value. Even though it may seem to be wasted effort, the real benefit comes when creating a copy of the text in a different language as all the text your require can be found in a single place.

// field labels for subsystem MENU
$array['Access']                        = 'Access';
$array['Button Id']                     = 'Button Id';
$array['Button Text']                   = 'Button Text';
$array['Default Language']              = 'Default Language';
$array['Dialog Type']                   = 'Dialog Type';
$array['Directory']                     = 'Directory';
$array['Field Id']                      = 'Field Id';

Entries from this file can be extracted using the getLanguageText() function.

Determine User Language

The first step is determine the user's language as provided by the client browser (user agent) in the $_SERVER["HTTP_ACCEPT_LANGUAGE"] variable. For this I am using the User Agent Language Detection script provided by http://techpatterns.com/. The code I use is as follows:

if (!isset($_SESSION['user_language_array'])) {
    // get language codes from HTTP header
    require 'language_detection.inc';
    $_SESSION['user_language_array'] = get_languages();
} // if

This returns an array of entries, one for each of the languages that the user may have set in his/her browser. Each language entry is another array of 4 entries as follows:

  1. Full language abbreviation, such as: 'en-gb' or 'en-us'
  2. Primary language, such as: 'en'
  3. Full language string, such as: 'English (United Kingdom)' or 'English (United States)'
  4. Primary language string, such as: 'English'

When looking for the subdirectory containing the language text file the array of language codes provided by the User Agent (client browser) will be examined first to last. If no subdirectory exists with the same name as the language subtype (such as 'en-us' or 'fr-ca') the software will look for a subdirectory with the same name as the primary language (such as 'en' or 'fr'). If nothing can be found for the first entry in the User Agent array the software will move on to the next entry.

If no subdirectory is found for any entry in the User Agent array the software will drop back to the installation's default language. In my sample application this is hard coded as 'en', but in my full development environment this is configurable using the Update Control Data screen.

Within my full development environment it is also possible for each user to define his/her preferred language code in the Update User screen. This overrides both the User Agent array and the installation default language.

Locate Language Subdirectory

Before loading the contents of a particular file it is first necessary to locate a valid subdirectory. This is done with the following function:

function getLanguageSubDir ($path)
// get subdirectory which corresponds with user's language code.
{
    // build an array of subdirectory names for specified $path
    $found = array();
    if (is_dir($path)) {
        $dir = dir($path);
        while (false !== ($entry = $dir->read())) {
            if ($entry == '.' or $entry == '..') {
                // ignore
            } else {
                if (is_dir("$path/$entry")) {
                    $found[] = $entry;
                } // if
            } // if
        } // if
        $dir->close();
    } // if
    
    if (!empty($found)) {
        if (isset($_SESSION['user_language_array']))) {
            // scan $user_language_array looking for a matching entry
            foreach ($_SESSION['user_language_array'] as $language) {
                // look for language subtype
                if (in_array($language[0], $found)) {
                    return "$path/$language[0]";
                } // if
                // look for primary language
                if (in_array($language[1], $found)) {
                    return "$path/$language[1]";
                } // if
            } // foreach
        } // if
    } // if
    
    // no language specified, so default to English
    return $path .'/en';
    
} // getLanguageSubDir

Note here that as an absolute minimum there MUST be a subdirectory for the default language, and this subdirectory MUST contain a full set of the expected files.

Load Screen Structure file

This uses the getLanguageSubDir() function to locate the relevant language subdirectory for the ./screens path before loading in the specified screen structure file.

function setScreenStructure ($xml_doc, $root, $screen, $xsl_file)
// extract screen structure from named file and insert details into XML document.
{
    // get subdirectory which matches user's language code
    $subdir = getLanguageSubDir ('./screens');
    
    $screen = "$subdir/$screen";  // look in subdirectory for this screen name
    if (!file_exists($screen)) {
        // 'File $screen cannot be found'
        trigger_error(getLanguageText('sys0056', $screen), E_USER_ERROR);
    } // if
    
    require $screen;                // import contents of disk file
    .....

Note that it does not matter if the only subdirectory that exists for screen structure files is in the default language as all the field labels will be translated into the chosen language at a later stage.

Get Language Text

Individual pieces of translated text will be extracted from the relevant 'language_text.inc' or 'sys.language_text.inc' files using the following function:

function getLanguageText ($id, $arg1=null, $arg2=null, $arg3=null, $arg4=null, $arg5=null)
// get text from the language file and include up to 5 arguments.
{
    static $array1;
    static $array2;
    
    if (!is_array($array1)) {
        $array1 = array();
        // include standard system text from current directory
        $subdir = getLanguageSubDir ('./text');
        $fname = "$subdir/sys.language_text.inc";
        if (!file_exists($fname)) {
            // filename does not exist
            trigger_error(getLanguageText('sys0057', $fname), E_USER_ERROR);
        } // if
        $array1 = require_once $fname;
        unset ($array);
    } // if
        
    if (!is_array($array2)) {
        $array2 = array();
        // include application text from current directory
        $subdir = getLanguageSubDir ('./text');
        $fname = "$subdir/language_text.inc";
        if (!file_exists($fname)) {
            // filename does not exist
            trigger_error(getLanguageText('sys0057', $fname), E_USER_ERROR);
        } // if
        $array2 = require_once $fname;
        unset ($array);
        // use this language code for the HTML output
        $pos = strrpos($subdir, '/');
        $GLOBALS['language'] = substr($subdir, $pos +1);
    } // if
    
    // perform lookup for specified $id ($array2 first, then $array1)
    if (isset($array2[$id])) {
        $string = $array2[$id];
    } elseif (isset($array1[$id])) {
        $string = $array1[$id];
    } else {
        // nothing found, so return original input
        return $id;
    } // if
    
    $string = convertEncoding($string, 'UTF-8');

    if (!is_null($arg1)) {
        // insert argument(s) into string
    	  $string = sprintf($string, $arg1, $arg2, $arg3, $arg4, $arg5);
    } // if
    
    return $string;
    
} // getLanguageText

Please note the following:

This new function has been inserted into the following places:

  1. Inside addParams2XMLdoc() to load script titles: 
    $xsl_params['title'] = getLanguageText($task_id);
  2. Inside setActBar() to load action buttons: 
    $label = getLanguageText($label);
  3. Inside setMenuBar() to load menu buttons: 
    $button['button_text'] = getLanguageText($button['button_text']);
  4. Inside setNavBar() to load navigation buttons: 
    $button['button_text'] = getLanguageText($button['button_text']);
  5. Inside setScreenStructure() to load field labels: 
    $fieldlabel = getLanguageText($fieldlabel);
  6. In various places for all error messages, such as: 
    if (strlen($fieldvalue) > $size) {
    // '$fieldname cannot be > $size characters
    $this->errors[$fieldname] = getLanguageText('sys0021', $fieldname, $size);
} // if

Get Language Array

Individual arrays of translated text will be extracted from the relevant 'language_array.inc' or 'sys.language_array.inc' files using the following function:

function getLanguageArray ($id)
// get named array from the language file.
{
    static $array1;
    static $array2;
    
    if (!is_array($array1)) {
        $array1 = array();
        // include standard system text from current subdirectory
        $subdir = getLanguageSubDir ('./text');
        $fname = "$subdir/sys.language_array.inc";
        if (!file_exists($fname)) {
            // filename does not exist
            trigger_error(getLanguageText('sys0057', $fname), E_USER_ERROR);
        } // if
        $array1 = require_once $fname;
        unset ($array);
    } // if
        
    if (!is_array($array2)) {
        $array2 = array();
        // include application text from current directory
        $subdir = getLanguageSubDir ('./text');
        $fname = "$subdir/language_array.inc";
        if (!file_exists($fname)) {
            // filename does not exist
            trigger_error(getLanguageText('sys0057', $fname), E_USER_ERROR);
        } // if
        $array2 = require_once $fname;
        unset ($array);
    } // if
    
    // perform lookup for specified $id ($array2 first, then $array1)
    if (isset($array2[$id])) {
        $result = $array2[$id];
    } elseif (isset($array1[$id])) {
        $result = $array1[$id];
    } else {
        // nothing found, so return original input as an array
        $result = array($id => $id);
    } // if
    
    foreach ($result as $key => $value) {
        $result[$key] = convertEncoding($value, 'UTF-8');
    } // foreach
    
    return $result;
    
} // getLanguageArray

Please note the following:

This new function should be used to obtain any array where the values will be displayed to the user. For example, instead of:

$languages = array('en' => 'English',
                   'es' => 'Spanish',
                   'fr' => 'French');

you should use the following:

$languages =  getLanguageArray('languages');

Handling Dates

I already use a standardised class to handle all my date validation (refer to A class for validating and formatting dates) so all I needed to do was change this:

    $this->monthalpha = array(1 => 'Jan','Feb','Mar','Apr','May','Jun',
                                   'Jul','Aug','Sep','Oct','Nov','Dec');

or

    $this->monthalpha = array(1 => 'Janv', 'Févr', 'Mars', 'Avr', 'Mai', 'Juin',
                                   'Juil', 'Août', 'Sept', 'Oct', 'Nov', 'Déc');

to this:

    $this->monthalpha = getLanguageArray('month_names_short');

As my default output format for dates is 'dd Mmm yyyy' this means that the 'Mmm' portion will use whatever language text is available.

Please note the following:

Handling Numbers

Although the English notation for numbers is to use '.' (period) for the decimal separator and ',' (comma) for the thousands separator there are some countries which use a different notation. Some have the two separators completely reversed, and some use ' ' (space) as the thousands separator. Regardless of any national conventions all numbers are processed within the program code, and stored within the database, in a common format. That is, the decimal point is a '.' (period) and there are no thousands separators.

This means that all decimal values must be formatted before they can be output to the user, and any user input must be unformatted before it can be handled by the program.

Convert to external (user's) format

A very important step in this process is therefore to identify all the decimal format conventions expected by the user. Fortunately all the relevant information can be provided by the localeconv() function. Unfortunately this requires the user's actual locale to be identified first with the setlocale() function. I say 'unfortunately' because the input to this function is the user's current locale or location whereas the only information available at present is the user's preferred language as supplied in the HTTP variables. I have got round this minor annoyance by modifying the 'languages' array used to determine the user's language to include a locale in the full language string, as in the following examples:

This means that I can now set the user's locale using code similar to the following:

    // get full language string from first entry in user_language_array
    $country = $_SESSION['user_language_array'][0][2];
    // extract locale which is enclosed in '[' and ']'
    if (!preg_match('?\[[^\[]+\]?', $country, $regs)) {
        // 'Locale is not defined in string' 
        trigger_error(getLanguageText('sys0078', $country), E_USER_ERROR);
    } // if
    $locale = trim($regs[0], '[]');
    // find out if this is a valid locale
    if (!$locale = setLocale(LC_ALL, $locale)) {
        // 'Cannot set locale'
        trigger_error(getLanguageText('sys0079', $locale), E_USER_ERROR);
    } // if

Having set the locale it is then a relatively simple exercise to convert any decimal number from internal to external format using code similar to the following:

    $decimal_places = $this->fieldspec[$fieldname]['scale'];
    $locale = localeconv();
    $decimal_point  = $locale['decimal_point'];
    $thousands_sep  = $locale['thousands_sep'];
    if ($thousands_sep == chr(160)) {
        // change non-breaking space into ordinary space
        $thousands_sep = chr(32);
    } // if
    $fieldvalue = number_format($fieldvalue,
                                $decimal_places,
                                $decimal_point,
                                $thousands_sep);

Convert to internal format

When the user presses the SUBMIT button any numbers that have been input will need to be converted back into internal format before they can be processed. This is done with code similar to the following:

function number_unformat ($input)
// convert input string into a number using settings from localeconv()
{
    $locale = localeconv();
    $decimal_point  = $locale['decimal_point'];
    $thousands_sep  = $locale['thousands_sep'];
    if ($thousands_sep == chr(160)) {
        // change non-breaking space into ordinary space
        $thousands_sep = chr(32);
    } // if
    
    $count = count_chars($input, 1);
    if ($count[ord($decimal_point)] > 1) {
        // too many decimal places
        return $input;
    } // if
    
    // split number into 2 distinct parts
    list($integer, $fraction) = explode($decimal_point, $input);
    
    // remove thousands separator
    $integer = str_replace($thousands_sep, NULL, $integer);
    
    // join the two parts back together again
    $number = $integer .'.' .$fraction;
    
    return $number;
    
} // number_unformat

Character Encoding

The process of internationalisation can be as simple as replacing a string of text in one language with a string of text in another language, or it can be much more complicated involving the use of different character sets, as explained in Notes on Internationalisation.

In order to deal with the various accented characters that exist in various languages it is necessary to use the correct character encoding, otherwise the display may be incorrect. In some cases it may result in an invalid character being written to the XML file, which will cause the XSL transformation to fail.

The best character set for internationalisation is UTF-8, which is why the next version of PHP, version 6, will have much better UTF-8 support. In the mean time, if you wish to support as wide a range of foreign languages as possible it is recommended hat you take the following steps:

Conclusion

Now that the software is in place it should be simple (in theory) to cater for new languages simply by creating a new subdirectory for the language, then dropping in a set of files which contain the translated text. This system may not be able to cater for every language or locale that exists, but it will deal with the most common ones.

My sample application has been updated with all this code, so feel free to download it and try it out. Contributions of translated files will be most welcome.

If it is necessary to provide application data in multiple languages from a single installation, then please refer to Internationalisation and the Radicore Development Infrastructure (Part 2).

References

© Tony Marston
15th July 2005

http://www.tonymarston.net
http://www.radicore.org

Amendment history:

01 Feb 2008 Included a reference to Internationalisation and the Radicore Development Infrastructure (Part 2) which describes a method of providing translations of application data.
27 Jan 2007 Added a new section on Character Encoding.
10 Mar 2006 Added a new section on File Contents.

counter