Introduction UIZE provides a system for automating various processes relating to the localization of a codebase. Localization as a Service UIZE exposes localization automation processes through a service, whose interface is defined in the =Uize.Services.Loc= abstract class. The =Uize.Services.Loc= service module defines the following `localization service methods`... - =metrics= - generates a `metrics report` for the `primary language` resource strings of a project - =export= - gathers strings from `codebase resource files` and exports them to `consolidated language resources files`, for each of the project's `supported languages` - =import= - distributes strings from the `consolidated language resources files` to `codebase resource files`, for each of the project's `supported languages` - =exportJobs= - exports `translation job files` from the `consolidated language resources files`, for each of the project's `translatable languages` - =importJobs= - imports translated strings from `translation job files` and merges them into the `consolidated language resources files`, for each of the project's `translatable languages` - =usage= - generates a `usage report` that contains details about usage of resource strings throughout a project, including information about unreferenced resource strings - =extract= - extracts strings from the project's codebase and generates `codebase resource files` for the project's `primary language` (may not be supported for a project) Localization Service Adapter As a convenience, UIZE provides an adapter base class for the =Uize.Services.Loc= service that can be applied, with some protected method overrides and configuration, to a variety of different types of projects. While it is possible to implement any adapter for the localization service, in reality there are sufficient commonalities between many different types of projects, such that it is beneficial to share implementation for the service adapter across as many different types of projects as possible. This is where the =Uize.Services.LocAdapter= module comes in. This module can be subclassed to create adapters for the localization service for different types of projects with minimal additional project type specific code needed. Accessed Through a Build Script The various `localization service methods` can be accessed using the =Uize.Build.Loc= build script. Running the Build Script The localization build script can be run in NodeJS using the following command... SYNTAX .................................................................................. node [pathToUize]build.js Uize.Build.Loc method=[methodName] project=[projectName] .................................................................................. Parameters All localization service methods support the following set of common parameters... project The =project= parameter is used to specify the project for which the specified `localization service method` should be executed. The value specified for the =project= parameter should match one of the keys in the =moduleConfigs ['Uize.Build.Loc'].projects= object of the =uize-config.json= file. Executing a Method For All Projects To execute a `localization service method` for all projects listed in the config, one can either omit the =project= parameter or one can specify the special "*" wildcard value. EXAMPLES ...................................................................... node [pathToUize]build.js Uize.Build.Loc method=[methodName] node [pathToUize]build.js Uize.Build.Loc method=[methodName] project=* ...................................................................... method The =method= parameter is used to specify the `localization service method` that should be executed for the specified project. The value of the =method= parameter should be the name of any of the `localization service methods` supported by the localization service, such as... - =metrics= - =export= - =import= - =exportJobs= - =importJobs= - =usage= - =extract= console The =console= parameter is used to specify the amount of information that should be logged to the console while the `localization service method` is being executed. The =console= parameter supports the following possible values... - =silent= - no information will be logged to the console - =summary= - only a summary will be logged to the console once execution of the `localization service method` is complete - =verbose= - information will be logged to the console for every step that is performed while the `localization service method` is being executed ### Project ~~ Projects . ### Languages Primary Language . Supported Languages . Translatable Languages . ### Key Features . Supports Multiple Projects . Adaptable to Different Project Types . Performs Diffing to Reduce Translation Load . Performs Automatic Pseudo-localization . Localization Service Methods ~~ Localization Service Method The =Uize.Services.Loc= service module defines a number of different localization service methods. metrics Generates a metrics report for a project using the =metrics= method of the localization service. SYNTAX ............................................................................. node [pathToUize]build.js Uize.Build.Loc method=metrics project=[projectName] ............................................................................. Generates a Metrics Report File The metrics method analyzes the resource strings for the project's `primary language` and generates a JSON format `metrics report file` with the path... .............................................. [workingFolder]/metrics/[primaryLanguage].json .............................................. Using this scheme, if the value of the =workingFolder= config option is ='loc'=, and the value of the =primaryLanguage= config option is ='en-US'=, then the path for the metrics file would be... ...................... loc/metrics/en-US.json ...................... Generates Strings Info Files In addition to generating the strings metrics JSON file, the =metrics= method also generates JSON and CSV format `strings info files` for the primary language. These files are output to the following paths... ................................................... [workingFolder]/strings-info/[primaryLanguage].csv [workingFolder]/strings-info/[primaryLanguage].json ................................................... Using this scheme, if the value of the =workingFolder= config option is ='loc'=, and the value of the =primaryLanguage= config option is ='en-US'=, then the paths for the strings info files would be... ........................... loc/strings-info/en-US.csv loc/strings-info/en-US.json ........................... export Exports the resource strings from the `codebase resource files` of a project to the `master resource files`. SYNTAX ............................................................................ node [pathToUize]build.js Uize.Build.Loc method=export project=[projectName] ............................................................................ The `master resource files` are output to the following paths... ............................................ [workingFolder]/[primaryLanguage].json [workingFolder]/[pseudoLocale].json [workingFolder]/[translatableLanguage1].json [workingFolder]/[translatableLanguage2].json ... ... ... ... ... ... ... ... ... ... ... [workingFolder]/[translatableLanguageN].json ............................................ EXAMPLE .............. loc/en-US.json loc/en-ZZ.json loc/en-GB.json loc/fr-FR.json loc/de-DE.json .............. In the above example, the =workingFolder= config option is set to ='loc'=, the =primaryLanguage= config option is set to ='en-US'=, the =pseudoLocale= config option is set to ='en-ZZ'=, and the configured translatable languages are ='en-GB'= (English for Great Britain), ='fr-FR'= (French), and ='de-DE'= (German). Parameters The =export= method supports the following parameters... initNonTranslatable By default, the value for each non-translatable string is initialized to the corresponding value of the string from the primary language, but only if the non-translatable string is currently blank. If this behavior is not suitable, one of the other available initialization modes can be specified using the =initNonTranslatable= parameter. The following initialization modes are supported... - =primary-if-blank= (default) - initializes the values of only blank non-translatable strings to the corresponding values of the strings from the primary language - =primary= - initializes the values of all non-translatable strings to the corresponding values of the strings from the primary language, regardless of their current values - =blank= - initializes the values of all non-translatable strings to blank (empty string), regardless of their current values - =never= - leaves the values of all non-translatable strings as is, keeping their current values languages An optional, comma-separated list of one or more of the project's `translatable languages` for which the =export= operation should be performed. For more detailed information, consult the section `Performing Methods for a Filtered Set of Languages`. import Imports the resource strings from the `master resource files` of a project back into the `codebase resource files`. SYNTAX ............................................................................ node [pathToUize]build.js Uize.Build.Loc method=import project=[projectName] ............................................................................ Executing the =import= method will have the effect of overwriting all the `codebase resource files` for the `pseudo-locale` and the translatable languages so that their contents reflects the contents of the `master resource files`. The =import= method will also create new codebase resource files, as necessary, if they did not previously exist. Parameters The =import= method supports the following parameters... languages An optional, comma-separated list of one or more of the project's `translatable languages` for which the =import= operation should be performed. For more detailed information, consult the section `Performing Methods for a Filtered Set of Languages`. exportJobs Exports resource strings for translation from the `master resource files` of a project to `translation job files`. SYNTAX ................................................................................ node [pathToUize]build.js Uize.Build.Loc method=exportJobs project=[projectName] ................................................................................ Parameters The =exportJobs= method supports the following parameters... filter The =exportJobs= method supports the ability to filter the strings that are exported to the translation job files, using the optional =filter= paramter. SYNTAX .................................................................................................... node [pathToUize]build.js Uize.Build.Loc method=exportJobs project=[projectName] filter=[filterName] .................................................................................................... filterName The following filters are supported for the =exportJobs= method... - =missing= (defualt) - only the translatable strings for which translations are missing for a translatable language will be exported in the translation job file for the language - =translated= - only the translatable strings for which translations exist for a translatable language will be exported in the translation job file for the language - =all= - all translatable strings for a translatable language will be exported in the translation job file for the language, whether they have existing translations or not languages An optional, comma-separated list of one or more of the project's `translatable languages` for which the =exportJobs= operation should be performed. For more detailed information, consult the section `Performing Methods for a Filtered Set of Languages`. Only Translatable Languages The =exportJobs= method generates `translation job files` for only the `translatable languages` configured for the project. In particular, this means that translation job files are never generated for the `primary language` or `pseudo-locale` configured for the project. Only Translatable Strings The `translation job files` generated by the =exportJobs= method contain only translatable strings. All non-translatable strings will be excluded from the translation job files. For more detailed information, consult the section `Non-translatable Strings`. File Format The =exportJobs= method will write `translation job files` in the `translation job file format` that is configured for the project. NOTES - before executing the =exportJobs= method, make sure that the `master resource files` are up-to-date by executing the =export= method, if necessary importJobs Imports translated resource strings from `translation job files` into the `master resource files` of a project. SYNTAX ................................................................................ node [pathToUize]build.js Uize.Build.Loc method=importJobs project=[projectName] ................................................................................ Parameters The =importJobs= method supports the following parameters... languages An optional, comma-separated list of one or more of the project's `translatable languages` for which the =importJobs= operation should be performed. For more detailed information, consult the section `Performing Methods for a Filtered Set of Languages`. auditTranslations Performs an audit of the resource strings in the `master resource files` for all the `translatable languages` of the project, to detect potential issues of `inconsistent translations` and `identical translations`. EXAMPLE ....................................................................................... node [pathToUize]build.js Uize.Build.Loc method=auditTranslations project=[projectName] ....................................................................................... The =auditTranslations= method analyzes the resource strings for each of the `translatable languages` of the project, comparing them to the corresponding values from the `primary language` to detect potential issues of `inconsistent translations` and `identical translations`, and then writes a =.json= report file for each translatable language with a path of the form... ........................................................................... [workingFolder]/[projectName]/translation-audit/[translatableLanguage].json ........................................................................... Parameters The =auditTranslations= method supports the following parameters... languages An optional, comma-separated list of one or more of the project's `translatable languages` for which the =auditTranslations= operation should be performed. For more detailed information, consult the section `Performing Methods for a Filtered Set of Languages`. For a more detailed discussion and for information on the contents of the reports, consult the section `Translation Audit Reports`. NOTES - before executing the =auditTranslations= method, make sure that the `master resource files` are up-to-date by executing the =export= method, if necessary diffLanguages Performs a diff between the resource strings in the `master resource files` of the two specified languages. EXAMPLE ..................................................................................................... node [pathToUize]build.js Uize.Build.Loc method=diffLanguages project=... languageA=... languageB=... ..................................................................................................... The =diffLanguages= method loads the resource strings from the `master resource files` for each of the two specified languages, performs a diff comparison of the resource strings for the two languages, and then writes =.json= and =.csv= report files containing details regarding the differences, with paths of the form... ............................................................................ [workingFolder]/[projectName]/language-diffs/[languageA]-vs-[languageB].json [workingFolder]/[projectName]/language-diffs/[languageA]-vs-[languageB].csv ............................................................................ Parameters The =diffLanguages= method supports the following parameters... languageA A locale code string, specifying the first of the two languages whose resource strings should be diffed. The default value for this property, if not specified, is the locale code of the `primary language` configured for the project. languageB A locale code string, specifying the second of the two languages whose resource strings should be diffed. The default value for this property, if not specified, is the locale code of the `primary language` configured for the project. For a more detailed discussion and for information on the contents of the reports, consult the section `Language Diff Reports`. NOTES - before executing the =diffLanguages= method, make sure that the `master resource files` are up-to-date by executing the =export= method, if necessary usage Generates a `usage report file` for the project, containing information about resource string references in the project's code, along with an assessment of unreferenced strings. SYNTAX ........................................................................... node [pathToUize]build.js Uize.Build.Loc method=usage project=[projectName] ........................................................................... The =usage= method generates a JSON format usage report file with the path... ......................................... [workingFolder]/metrics/usage-report.json ......................................... In order for the =usage= method to work reliably, the localization adapter subclass for the project must implement the =getReferencingCodeFiles= and =getReferencesFromCodeFile= instance methods. NOTES - see also the =extract= localization service method extract Extracts resource strings from references inside the code files of a project and generates one or more `codebase resource files`. SYNTAX ............................................................................. node [pathToUize]build.js Uize.Build.Loc method=extract project=[projectName] ............................................................................. This method can be useful for projects where developers typically create resource strings by first referencing them in translation calls in the code, rather than by first creating entries in resource files. The =extract= method is not implemented by the localization service adapter base class - the base class' version of this method just throws an error. Therefore, the localization adapter subclass for the project must implement this method. The =getReferencingCodeFiles= and =getReferencesFromCodeFile= instance methods that need to be implemented by an adapter subclass in order to support the =usage= method can be used in one's implementation of the =extract= method. NOTES - see also the =usage= localization service method Configuring Projects The behavior of the =Uize.Build.Loc= build script is configurable to support one or more projects of potentially different types. `Config options` for the =Uize.Build.Loc= build script should be placed inside the =uize-config.json= config file, under the path =moduleConfigs ['Uize.Build.Loc']=, as illustrated by the example below... EXAMPLE .............................................................. { // other config options // other config options // other config options // other config options moduleConfigs:{ 'Uize.Build.Loc':{ workingFolder:'loc', projects:{ Uize:{ serviceAdapter:'Uize.Services.LocAdapter.Uize', rootFolderPath:'site-source/js', languages:[ 'en-US', 'de-DE', 'fr-FR', 'ja-JP', 'nl-NL', 'ru-RU', 'zh-CN' ], primaryLanguage:'en-US', pseudoLocale:'en-ZZ' } } } } } .............................................................. The above example shows the configuration of the =Uize.Build.Loc= build script for the UIZE project. Config Options The =Uize.Build.Loc= build script supports a number of configuration options. Working Folder ~~ workingFolder The =workingFolder= config option specifies the path, relative to the current working directory, to the working folder for the =Uize.Build.Loc= build script. The working folder is used to store the output of the various `localization service methods`. Per Project Configuration Options The localization service supports a number of per project configuration options. serviceAdapter ~~ Service Adapter The service adapter can be configured for a project by specifying a module name for the =serviceAdapter= config option. EXAMPLE ................................................................ serviceAdapter:'MyNamespace.LocServiceAdapters.MobileAppAndroid' ................................................................ Typically, each project has its own localization service adapter subclass, but it is possible for multiple projects to share the same service adapter module if they are all of the same type but differ in other aspects of their configuration. rootFolderPath ~~ Root Folder Path The =rootFolderPath= config option lets you specify the path to the root folder under which all the project's `codebase resource files` can be found. EXAMPLE ........................................... rootFolderPath:'~/git-repos/my-project/res' ........................................... If a relative path is specified for the =rootFolderPath= option, the path should be relative to the current directory at the time that the localization service scripts are run. languages ~~ Languages The =languages= config option lets you specify the `translatable languages` of the project, for which translation should be managed by the localization service. It is recommended that the value specified for the =languages= option be an array of [[http://en.wikipedia.org/wiki/IETF_language_tag][BCP 47]] locale codes, containing both the language and region codes. EXAMPLE ........................ languages:[ 'de-DE', // German 'fr-FR', // French 'ja-JP', // Japanese 'nl-NL', // Dutch 'ru-RU', // Russian 'zh-CN' // Chinese ] ........................ The languages list may redundantly contain the primary language that is configured for the project in the separate =primaryLanguage= config option, but it is not required and it is recommended to omit the primary language from this list. In any event, the primary language is not considered to be a `translatable language`. brandLanguages ~~ Brand Languages Lets you configure the `translatable languages` supported per brand. It is not uncommon, in situations where a project supports multiple brands, for those brands to be associated with markets in different regions, with different regional languages for those regions. In projects that have `brand-specific resource strings`, the =brandLanguages= config option lets you configure different sets of translatable languages on a per brand basis. EXAMPLE .......................................... brandLanguages:{ fooBrand:[ 'de-DE', // German 'fr-FR', // French 'en-GB' // English (Great Britain) ], barBrand:[ 'es-MX', // Spanish (Mexico) 'pt-BR' // Portuguese (Brazil) ] } .......................................... In the above example, the brand "fooBrand" supports three languages, while brand "barBrand" supports two languages. There is no overlap in the languages supported by these two brands. Resource strings that are specific to "fooBrand" are not translated to Spanish or Portuguese, while resource strings that are specific to "barBrand" are not translated to German, French, or British English. Resource strings that are not brand-specific and that are, therefore, common to these two brands will be translated for the superset of all the brand languages. primaryLanguage ~~ Primary Language Lets you specify the primary language for the project. The primary language is the language that will be used as the source of truth, and the resource strings from the primary language resource files will drive the translation process. Resource strings for the configured `translatable languages` of the project will be produced by translating the corresponding values of the strings from the primary language resource files. EXAMPLE ...................................... primaryLanguage:'en-US' // US English ...................................... It is fairly common for the primary language to be a regional variant of English (such as US English), but this is not a strict requirement. It is recommended that the value specified for the =primaryLanguage= option be a `BCP 47` locale code, containing both the language and region codes. pseudoLocale ~~ Pseudo-locale Lets you specify the locale code to be used for the pseudo-locale. It is recommended that the value specified for the =pseudoLocale= option be a `BCP 47` locale code, containing both the language and region codes. It is further recommended that specifically the "en-ZZ" locale code be used for the pseudo-locale, since this locale code exists in a range of BCP 47 locale codes that are not assigned to specific languages and are free to be used for proprietary purposes such as pseudo-localization. EXAMPLE .................... pseudoLocale:'en-ZZ' .................... pseudoLocalization ~~ Pseudo-localization Lets you specify configuration options for the `pseudo-localization` process that is performed when executing the =export= localization service method. EXAMPLE .................... pseudoLocalization:{ expansion:1.15, wrapper:'[]' } .................... Generally, it is not necessary to explicitly configure the pseudo-localization process, since the default configuration should suit most projects. If any pseudo-localization option is to be configured differently for specific projects, it is likely to be the expansion factor. The most suitable expansion factor to use will be influenced by the set of `translatable languages` that the project needs to support, since different languages introduce different amounts of expansion during translation. resourceFileWhitespace ~~ Resource File Whitespace Lets you configure the whitespace options that should be used by the =import= localization service method when it writes `codebase resource files`. EXAMPLE ........................ resourceFileWhitespace:{ indentChars:' ', linebreakChars:'\n' } ........................ By default, resource files are serialized using a single tab character for indentation and a single new line character linebreaks. If these defaults are not suitable for the project's codebase, then the whitespace options can be specified explicitly using the =resourceFileWhitespace= config option. Whitespace Options The value specified for the =resourceFileWhitespace= option should be an object containing any of the following supported properties... indentChars ~~ Indent Characters A string, specifying the character(s) that should be used to denote one level of indentation. The default value for this property, if not specified, is a single tab character. linebreakChars ~~ Linebreak Characters A string, specifying the character(s) that should be used to denote a linebreak. The default value for this property, if not specified, is a single new line character. translationJobFileFormat ~~ Translation Job File Format The format of `translation job files` written by the =exportJobs= method can be configured using the =translationJobFileFormat= config option. The =translationJobFileFormat= config option supports the following possible values... - =csv= - translation job files will be exported in CSV (Comma-separated Values) format - =xliff= - translation job files will be exported in XLIFF (XML Localization Interchange File Format) ### Common Project Configuration Options ~~ common . Localization Service Workflows Performing Translation In order to have the resource strings for a project be translated, one must first `export the resource strings for translation` and then `import the translated resource strings`. Export the Resource Strings for Translation Exporting the resource strings for translation for a project involves the following steps... +. Execute the =export= `localization service method` to update the =.json= `master resource files` to reflect the current state of the `codebase resource files`. +. Execute the =exportJobs= `localization service method` to generate updated `translation job files`. Once the `translation job files` have been generated to reflect the current state of the project's codebase and the strings that actually need translation, these files can be sent to translators for translation. Import the Translated Resource Strings Once the translation job files have been processed by the translators and the strings have been translated, the translations can be imported back into the project's codebase with the following steps... +. Replace the `translation job files` inside the *jobs* folder with the updated files delivered by the trabslators. +. Execute the =importJobs= `localization service method` to update the =.json= `master resource files` with the translations contained inside the `translation job files`. +. Execute the =import= `localization service method` to update the `codebase resource files` of the project with the strings from the updated =.json= `master resource files`. Once the `codebase resource files` have been updated, a diff sanity check in one's source control tool should verify that the translations have made their way into the resource files in the project's codebase. Changes to Resource Strings Deleting Resource Strings At times, it will be necessary to delete one or more resource strings because a component has been modified. Consider that the application may support multiple different languages and there may be translations in the resource files for different languages for the resource strings you wish to delete. With the Loc service, it is not necessary to remove the resource strings from the translated resource files - they will be repaired if you follow these steps... +. Delete the resource strings in the primary language resource files. Be sure to remove all brand-specific overrides, if any exist, from the primary language resource files. +. Execute the =export= method of the Loc service to update the =.json= `master resource files`. +. Execute the =import= method of the Loc service to re-generate the resource files for the supported languages of the project. Performing the =export= followed by the =import= will have the effect of repairing the resource files in the codebase so that the removed resource strings will no longer exist in the resource files for the supported languages. ### Modifying Resource Strings . ### Adding Resource Strings . Renaming Resource Strings At times, it will be necessary to rename one or more resource strings because a component has been modified. Consider that the application may support multiple different languages and there may be translations in the resource files for different languages for the resource strings you wish to rename. The Loc service does not provide any special handling for such situations and you will need to rename the resource strings in the resource files for the primary language and all other supported languages. ### Re-translating Resource Strings . ### Changes to Resource Files Deleting Resource Files . Adding Resource Files . ### Changes to Language Support Adding Support for a New Language . Removing Support for a Language . ### Changes in Projects Adding a New Project . Branching a Project . Localization Loop .................................................................................................. -------> 4. translate jobs -------> -------> 9. translate jobs -------> -----------> ∧ | ∧ | ∧ | | | | | | 3. export 5. import | | 8. export 10. import | | 13. export | jobs jobs | | jobs jobs | | jobs | ∨ | ∨ | |-------------------------------------------------------------------------------------| | | | MASTER RESOURCE FILES | | | |-------------------------------------------------------------------------------------| ∧ | ∧ | ∧ | | | | | | | 1. export | | 2. import 6. export | | 7. import 11. export | | 12. import | | | | | | | ∨ | ∨ | ∨ |---------------------------------------------------------------------------------------------| | | | | | | | | | | A | ------> 4. develop -------> | B | ------> 9. develop -------> | C | | | | | | | | | | |---------------------------------------------------------------------------------------------| .................................................................................................. +. `export` +. `import` +. `export jobs` +. `translate jobs` and `develop` +. `import jobs` +. `export` +. `import` +. `repeat from step 3` ### Iterative Translation revision A - initiate translations export exportJobs - XLIFF files for translatable languages will have entries for only strings that are empty (the delta) revision B - one string was added - one string was removed - one string was changed - .json mater file should reflect state of codebase at revision A - importJobs - import - resource files for primary language are not modified - after import, codebase resource files for translatable languages will contain... - no string for string that was added since revision A - the old translations for the string that was removed since revision A - the old translations for the string that was changed since revision A - repair, using export + import - after repair... - there will be an empty string value for the string that was added since revision A - translations for string that was removed since revision A will be gone - the defunct translations for the string that was changed since revision A will be blanked out - initiate translations export exportJobs - XLIFF files for translatable languages will have entries for... - strings that were added since revision A - strings that were changed since revision A Advanced Topics Performing Methods for a Filtered Set of Languages Various of the localization service methods support the optional =languages= parameter, which lets you execute the methods for a filtered set of the project's `translatable languages`. EXAMPLE ........................................................................................ node [pathToUize]build.js Uize.Build.Loc method=export project=Foo languages=fr-FR,fr-CA ........................................................................................ In the above example, the =export= method's operation will be performed for just the "fr-FR" (French French) and "fr-CA" (Canadian French) languages, even though the "Foo" project may support a larger set of translatable languages. The following methods support the optional =languages= parameter... - =export= - =exportJobs= - =import= - =importJobs= - =auditTranslations= When the optional =languages= parameter is either not specified (or if an empty value is specified) when executing any of the above methods, then the method's operation will be performed for all the `trnslatable languages` configured for the project. Non-translatable Strings Non-translatable strings are strings that should never be pseudo-localized and should never be sent to translators for translation. Examples of non-translatable resource strings would be... - URLs - media asset IDs - dimension values - color values - support e-mail addresses or phone numbers - item codes Reasons for Treating Some Strings as Non-translatable There are good reasons to identify certain strings as being non-translatable and then handle such strings differently in the automation process. A Waste of Money For one thing, it is a waste of money to send non-translatable strings to translators. Translators will likely not know what to do with such strings, and determining how to handle them may involve some costly back-and-forth communication between the project managers on both sides of the process. And if they actually translate such strings by accident, then they will charge for that translation work. Breakage from Accidental Translation Accidental translation of strings that should not be translated can result in breakage of the application. For example, a resource string may be used for storing a color value that should be different per language. Technologies such as CSS support named color values, such as "red" or "green". If the intention is to use the color red for one language and the color green for another language, it is not helpful if the actual text "red" is translated to some other language by the translators. Translating "red" to "rouge" in French would break the application in that the translated color value would not be recognized by CSS. Instead, such a string should be treated as a non-translatable string, and it would then be the responsibility of a developer to manually change the value in the resource files of the translatable languages, under the direction of a product manager who makes the decision what colors best suit different language audiences. Breakage from Pseudo-localization Accidental pseudo-localization of some non-translatable strings (such as IDs or URLs) may break the pseudo-localized version of the application and hinder localization QA testing of the pseudo-localized version. For example, if a resource string was being used to store a code like ="AdvancedMode"=, where this code is being used by the application logic and is intended to be different per language, then pseudo-localizing the value from the primary language to ="[ÅðṽåñçéðṀöðé____]"= would break the application as it would not understand this value. Non-translatable Strings Vary by Project Different types of projects will likely have different varieties of non-translatable strings that it is convenient for the project to store in `codebase resource files`. Therefore, it is up to each project to determine which strings are translatable and which strings are not. This is accomplished by implementing an override to the new =isTranslatableString= method in the `localization service adapter`. isTranslatableString Implementing support for non-translatable strings in a project involves providing an implementation for the =isTranslatableString= instance method in the localization adapter subclass for the project. The implementation in the adapter base class just returns =true=, so all resource strings are considered translatable by default. An implementation for the =isTranslatableString= method should expect to receive a single argument, being a string info object that describes a resource string. The string info object will contain a =key= property that specifies the resource string's key name, along with a =value= property that specifies the value of the resource string for the `primary language`. The method implementation should use the information in the string info object to determine a boolean return value, indicating whether or not the string is translatable. The implementation can use either or both of the =key= and =value= properties to reach its determination- whichever is best suited to the nature of the resource strings in the project. EXAMPLE .................................................................................. function (stringInfo) { return ( !/_(ID|EMAIL)$/.test (stringInfo.key) && // it's an ID or an e-mail address !/^https?:\/\//.test (stringInfo.value) // it's a URL ); } .................................................................................. In the above example implementation for the =isTranslatableString= method, the method is first performing a test on the key of the resource string, checking to see if it ends with "_ID" or "_EMAIL". If this test fails, the method is then testing if the value of the resource string starts with "http://" or "https://". Establish a Convention for Non-translatable Strings While it is possible to write complex matching logic to test whether or not resource strings should be translatable, it is recommended that projects establish a simple and robust convention. For example, a project could follow the convention that all non-translatable resource strings are indicated with a specific suffix or prefix in their key names. For instance, all non-translatable resource strings could have a "$" character appended to their key names. EXAMPLE ............................................................ function (stringInfo) { // string is translatable if its key doesn't end with "$" return stringInfo.key.slice (-1) != '$'; } ............................................................ Following such a convention would provide the following benefits... - it becomes clear from the `codebase resource files` exactly which strings are not to be translated - the =isTranslatableString= test is simple, robust, and highly performant - it is easy to add new non-translatable strings of different types later to a project later, without needing to update the implementation of the =isTranslatableString= method Surveying Non-translatable Strings In order to verify that one's implementation of the =isTranslatableString= method for a project is doing the right thing, you can survey the strings that are determined to be non-translatable by using the =metrics= localization service method. When you execute the =metrics= method, one of its by-products will be the strings info files that are written to the "strings-info" folder. The strings info is obtained from the resource strings in the primary language resource files for the project and is written to two different format files: one JSON format file, and one CSV format file. - *.json* - The JSON format file contains an array of string info objects for all of the strings of the project, where each string info object contains properties such as =path=, =value=, =isTranslatable=, etc., and where the values for some properties (such as the =metrics= property) are sub-objects. - *.csv* - The CSV format file contains a table of rows for all of the strings of the project, where each row contains a number of columns that represent a flattened version of the string info object for a string. Both of these two strings info files contain the same information, but expressed slightly differently. The CSV file is particularly easy to "consume" and can be loaded up in any spreadsheet editor (such as Google Sheets) that correctly supports the CSV format's ability to have quoted field values that contain line breaks (some of your resource strings may contain line break characters). Once loaded into a spreadsheet tool, the =Translatable= column can be used to sort or filter the strings info rows based upon whether or not the strings are considered translatable. For non-translatable strings, the value in the =Translatable= column will be =false=. The spreadsheet can then be used to verify that there are no non-translatable strings being missed by the =isTranslatableString= method, and there are no translatable strings that are being incorrectly caught by it. Handling of Non-translatable Strings Non-translatable strings are handled according to the following rules... Never Pseudo-localized Non-translatable strings are never pseudo-localized when resource strings for the `pseudo-locale` are generated by the =export= localization service method. Instead, the values from the primary language will be used as is. Never Exported for Translation Non-translatable strings are never exported to `translation job files` by the =exportJobs= localization service method. This means that translators never have to deal with these strings, and there's never a possibility that translators will accidentally translate them, resulting in charges for wasted effort or even problems for the application. Initialized to Primary Language Values Whenever a non-translatable string is added or modified for the primary language of a project, the values for the string for the translatable languages are initialized to the value from the primary language. In this way, the values for the string in the resource files for the translatable languages don't need to be manually initialized by developers. Manually Modified Values are Respected While the values for non-translatable strings are `initialized to primary language values`, the values for the strings in the translatable languages can subsequently be manually modified by developers. Non-empty values for non-translatable strings in the resource files for the translatable languages of a project are respected throughout the localization automation process. Initializing Values of Non-translatable Strings The values of the non-translatable strings of a project can be initialized by executing the =export= command and specifying one of the supported initialization modes using the =initNonTranslatable= parameter. ........................................................................................................... node [pathToUize]build.js Uize.Build.Loc method=export project=[projectName] initNonTranslatable=[initMode] ........................................................................................................... The non-translatable strings initialized in this way will then be updated in the `codebase resource files` during the next =import= operation. Metrics Reports ~~ Metrics Report A metrics report can be produced for a project by using the =metrics= `localization service method`. EXAMPLE ............................................................................. node [pathToUize]build.js Uize.Build.Loc method=metrics project=[projectName] ............................................................................. The =metrics= method analyzes the primary language resource strings for a project and writes a =.json= file to the following path... ............................................................ [workingFolder]/[projectName]/metrics/[primaryLanguage].json ............................................................ The metrics report is a JSON file, so the data contained in it can be easily loaded by other scripts or an application and used to present a visualization of the metrics or to estimate cost for translation. Metrics Report Contents Total Word Count ~~ wordCount The total number of translatable words in all primary language resource files for the project is represented by the =wordCount= property. The value of the =wordCount= property is only an estimate and the accuracy of the value depends on how well the project is defined to understand the difference between `translatable words` and `non-translatable text`. The value of this property is useful when estimating the full translation cost for a project, since translators often charge for their services based upon word count and a cost per word. Total Character Count ~~ charCount The total character count for all the translatable words in all primary language resource files for the project is represented by the =charCount= property. The accuracy of the value of this property depends on the accuracy of the =wordCount= property, since this property is the sum of the character count for all `translatable words`. This value can be useful when estimating the full translation cost for a project, since translators may base cost estimates for their services partially upon total character count for all `translatable words`. Total Resource Files ~~ resourceFiles The total number of resource files for the language is represented by the =resourceFiles= property. The value of the =resourceFiles= property reflects the total number of codebase resource files for the language and not the total for all languages, so this value doesn't increase as you add support for new languages. Brand-specific Resource Files The total number of brand-specific resource files for the language is represented by the =brandSpecific= property of the =resourceFiles= object. What constitutes a brand-specific resource file will vary by project and depend on the project definition. Typically, brand-specific resource files provide overrides for certain resource strings that need to be different for specific brands, so the number of brand-specific resource files is usually smaller than the total number of resource files. Moreover, different brands may have a differing number of brand-specific resource files. Strings Info . Translation Audit Reports ~~ Translation Audit Report A translation audit report can be produced for a project by using the =auditTranslations= `localization service method`. EXAMPLE ....................................................................................... node [pathToUize]build.js Uize.Build.Loc method=auditTranslations project=[projectName] ....................................................................................... The =auditTranslations= method analyzes the resource strings for each of the `translatable languages` of the project, comparing them to the corresponding values from the `primary language` to detect potential issues of `inconsistent translations` and `identical translations`, and then writes a =.json= report file for each translatable language with a path of the form... ........................................................................... [workingFolder]/[projectName]/translation-audit/[translatableLanguage].json ........................................................................... Not a Guarantee of Problems It is important to note that the issues listed in the translation audit reports are not a guarantee of real problems in translation. For instance, there can be valid reasons for translations of any given `primary language` text to be inconsistent for a `translatable language`. For example, a single word that is used for different resource strings can have a different meaning based upon context In one context, the word may be used as a verb, while in another context the same word may be used as a noun. In the translatable language, the verb and noun forms may be different, while in the primary language they may be the same. This is the case with an English word like "call". There can also be valid reasons for translations of any given `primary language` text to be identical for a `translatable language`. This will, naturally, be the case for many resource strings when translating from US English to British English, but it can also be the case for specific English words whose usage has been adopted in other languages. Such is the case with a word like "OK" (typically used in the UI in OK buttons), which is considered acceptable (and even preferable) in French language UI. Generally, the longer the source text from the primary language, the more likely that an issue listed in the `translation audit report` is a real problem. The reports are provided as a useful guide to potential issues, and these potential issues should be assessed and investigated further, if necessary. Translation Audit Report Contents The translation audit report object contains the following sections, describing different potential translation issues... Identical Translations ~~ identicalTranslations An identical translation is a translation where the value of a resource string in a `translatable language` is identical to the corresponding value of that resource string in the `primary language`. Identical translations can be an indication of errors made during the translation process, where the source text is accidentally used unaltered as the translation. Information on all the resource strings that have identical translations for a given translatable language can be found in the =identicalTranslations= property in the `translation audit report` for the language. The value of the =identicalTranslations= property is an array of the form.. ............................ identicalTranslations:[ identicalTranslation0OBJ, identicalTranslation1OBJ, ... identicalTranslationNOBJ ] ............................ Each element of the =identicalTranslations= array is an object describing an instance of an identical translation, having the form... ........................ { path:stringPathARRAY, value:stringValueSTR } ........................ The value of the =path= property is a string path array that contains all the information necessary to identify the resource string in the project, starting with the resource file that contains the string as the first element, and ending with the string key as the last element. The value of the =value= property is a string, specifying the value (i.e. text) of the string. EXAMPLE ....................... identicalTranslations:[ { path:[ 'strings.xml', 'OkButton' ], value:'OK' }, { path:[ 'strings.xml', 'FaxButton' ], value:'FAX' } ] ....................... In the above example, two strings have translations that are identical to the source text from the primary language (English, in this case). This case can arise with a language like French, where it is acceptable to use "OK" and "FAX" in UI text. In fact, "OK" would even be considered preferable to using a stricter translation like "D'accord". So, in this case, the indentical translations are not an indication of a real problem. Inconsistent Translations ~~ inconsistentTranslations An inconsistent translation is a case in which some source text contained in resource strings of the `primary language` is translated in more than one way in the resource strings of a `translatable language`. Inappropriate inconsistent translations can arise in larger projects where translation jobs are distributed amongst multiple translators who translate in parallel and don't leverage shared translation memory, or when translation of future additions to an existing project are handed off to a new translation vendor without providing them with a translation memory or previous translations. Inconsistencies can also arise through basic mistakes in the translation process. Information on all inconsistent translations for a given translatable language can be found in the =inconsistentTranslations= property in the `translation audit report` for the language. The value of the =inconsistentTranslations= property is an array of the form... ............................... inconsistentTranslations:[ inconsistentTranslation0OBJ, inconsistentTranslation1OBJ, ... inconsistentTranslationNOBJ ] ............................... Each element of the =inconsistentTranslations= array is an object of the form... ........................ { source:sourceTextSTR, translations:[ translation0OBJ, translation1OBJ, ... translationNOBJ ] } ........................ The value of the =source= property is a string, specifying the source text from the `primary language`. The value of the =translations= property is an array of objects describing the different translations of the source text, where each object has the form... .................................. { translation:translationTextSTR, strings:[ stringPath0ARRAY, stringPath1ARRAY, ... stringPathNARRAY ] } .................................. The value of the =translation= property is a string, specifying the translated text from the `translatable language`. The value of the =strings= property is an array of string path arrays, identifying all the strings that have this translation for the `translatable language`. The value for each element of the =strings= array is a string path array that contains all the information necessary to identify the resource string in the project, starting with the resource file that contains the string as the first element, and ending with the string key as the last element. The structure of the =inconsistentTranslations= object is illustrated by the following example, in which there are two different translations for the source text "Call". EXAMPLE .................................... inconsistentTranslations:[ { source:'Call', translations:[ { translation:'Appeler', strings:[ [ 'strings.xml', 'CallButton' ], [ 'strings.xml', 'CallLink' ] ] }, { translation:'Appel', strings:[ [ 'strings.xml', 'CallHistoryLabel' ] ] } ] } ] .................................... In this example, the inconsistent French translations are legititate because the word "Call" is used as a verb in the first case (translated to "Appeler") and as a noun in the second (translated to "Appel"). Language Diff Reports ~~ Language Diff Report A language diff report can be produced for a project by using the =diffLanguages= `localization service method`. EXAMPLE ..................................................................................................... node [pathToUize]build.js Uize.Build.Loc method=diffLanguages project=... languageA=... languageB=... ..................................................................................................... The =diffLanguages= method loads the resource strings from the `master resource files` for each of the two specified languages, performs a diff comparison of the resource strings for the two languages, and then writes =.json= and =.csv= report files containing details regarding the differences, with paths of the form... ............................................................................ [workingFolder]/[projectName]/language-diffs/[languageA]-vs-[languageB].json [workingFolder]/[projectName]/language-diffs/[languageA]-vs-[languageB].csv ............................................................................ Investigating Drift Language diff reports may be useful in situations where you suspect that string values may have drifted in an undesirable way between two regional variants of a language, such as between US English (en-US) and British English (en-GB). In one example, undesirable drift may occur if process is not followed correctly and changes are made directly in the resource strings for one regional variant of a language and those changes are not replicated in other regional variants of that same language, particularly if the values were identical beforehand and no longer are after the change. Difference Types ~~ Difference Type Language diff reports are very straightforward and describe just three types of differences... - ='missing in A'= - the string is missing or empty for language A, but present and non-empty for language B - ='missing in B'= - the string is missing or empty for language B, but present and non-empty for language A - ='different'= - the string is present and non-empty for both language A and language B, but the values differ Language Diff Report Contents When you execute the =diffLanguages= method, two different format report files are written to the "language-diffs" folder: one JSON format file, and one CSV format file. - *.json* - The `JSON format diff report` contains an array of string diff objects for all strings that have differences, where each string diff object contains the properties =key=, =path=, =valueInA=, =valueInB=, and =difference=. - *.csv* - The `CSV format diff report` contains a table of rows for all of the strings that have differences, where each row contains columns that represent a flattened version of the string diff object for a string. Both of these two diff report files contain the same information, but expressed slightly differently. The CSV file is particularly easy to "consume" and can be loaded up in any spreadsheet editor (such as Google Sheets) that correctly supports the CSV format's ability to have quoted field values that contain line breaks (some of your resource strings may contain line break characters). Once loaded into a spreadsheet tool, the spreadsheet can be sorted by the "Difference" column or filtered by the `difference type` you are interested in. JSON Format Diff Report The JSON format diff report is a serialized array of string diff objects, of the form... .................. [ stringDiff0OBJ, stringDiff1OBJ, ..., stringDiffNOBJ ] .................. Each element of the string diff objects array is a string diff object of the form... ............................................................................. { key:keySTR, path:pathARRAY, valueInA:valueInASTR, valueInB:valueInBSTR, difference:differenceSTR // 'different' | 'missing in A' | 'missing in B' } ............................................................................. Each string diff object contains the following properties... - =key= - a string, specifying the key of the resource string (the last element of the string path array) - =path= - an array, specifying the full path of the resource string, where the first element is the path of the resource file containing the string, and where the last element is the key of the resource string - =valueInA= - a string, specifying the value of the resource string in language A, or an empty string if the string is missing in language A - =valueInB= - a string, specifying the value of the resource string in language B, or an empty string if the string is missing in language B - =difference= - a string, specifying the `difference type` for the resource string between language A and language B The structure of the JSON format language diff report is illustrated by the following example, in which there are two differences between the two languages compared in the diff... EXAMPLE .......................................... [ { key:'label', path:[ 'strings.js', 'colorPicker', 'label' ], valueInA:'color', valueInB:'colour', difference:'different' }, { key:'tooltip', path:[ 'strings.js', 'colorPicker', 'tooltip' ], valueInA:'Click to choose a color.', valueInB:'', difference:'missing in B' } ] .......................................... CSV Format Diff Report The CSV format diff report is a table containing the following columns... - *Resource File* - the path of the resource file containing the resource string - *Key* - the key of the resource string - *Path* - a JSON serialization of the full path of the resource string - *Difference* - the `difference type` for the resource string between language A and language B - *Value in A* - the value of the resource string in language A, or an empty string if the string is missing in language A - *Value in B* - the value of the resource string in language B, or an empty string if the string is missing in language B The structure of the CSV format language diff report is illustrated by the following example, in which there are two differences between the two languages compared in the diff... CSV EXAMPLE .................................................................................................. Resource File,Key,Path,Difference,Value in A,Value in B strings.js,label,"['strings.js','colorPicker','label']",different,color,colour strings.js,tooltip,"['strings.js','colorPicker','tooltip']",missing in B,Click to choose a color., .................................................................................................. Usage Reports ~~ Usage Report . Initializing Resource Strings The values of resource strings of a project can be initialized by executing the =export= command and specifying `value initialization options` using the =initValues= parameter. ..................................................................................................... node [pathToUize]build.js Uize.Build.Loc method=export project=[projectName] initValues=[initOptions] ..................................................................................................... The strings initialized in this way will then be updated in the `codebase resource files` during the next =import= operation. Value Initialization Options Value initialization options are specified using the =initValues= parameter, and these options determine which resource strings get initialized, when (under which conditions) they get initialized, and to what values they get initialized. The value specified for the =initValues= parameter should have the following syntax... ......................................... initValues=[initWhat],[initWhen],[initTo] ......................................... initWhat The first portion of the initialization options specifies what resource strings should be initialized, and supports the following values... - =none= - None of the resource strings will be initialized (this value can be used to defeat the feature). - =all= - All of the resource strings will be initialized (this includes translatable and non-translatable strings). - =translatable= - Only the translatable strings will be initialized. - =non-translatable= - Only the non-translatable strings will be initializd. initWhen The second portion of the initialization options specifies when (under which conditions) the strings should be initialized, and supports the following values... - =never= - The resource strings will never be initialized (this value can be used to defeat the feature). - =always= - The resource strings will always be initialized. - =blank= - The resource strings will only be initialized when they are blank. initTo The third portion of the initialization options specifies to what values the strings should be initialized, and supports the following values... - =primary= - The resource strings will be initialized to the values for the strings from the primary language. - =blank= - The resource strings will be initialized to blank values (empty strings). Examples In a simple example, the value initialization feature can be used to `initialize values of non-translatable strings`. To initialize the values of the non-translatable strings that are blank to the values from the primary language, one can use the following value for the =initValues= parameter... ......................................... initValues=non-translatable,blank,primary ......................................... In another example, the value initialization feature can be used to initialize all the strings for the =en-GB= (British English) language to the values from the primary language (presumably US English in this example), using the following parameters for the =export= method... ............................................. languages=en-GB initValues=all,always,primary ............................................. Note that we are using the =languages= parameter and specifying the value =en-GB= in order to perform this initialization during export for just the British English language (refer to the section `Performing Methods for a Filtered Set of Languages`). Different Classes of Files The localization automation scripts work with four main classes of files: `codebase resource files`, `master resource files`, `translation job files`, and various kinds of `report and info files`. Codebase Resource Files ~~ Codebase Resource File Codebase resource files are the files inside the codebase of a project that contain the resource strings for the project. The file format of the codebase resource files can vary from project to project and will depend on the technology used for the project. Common Resource File Formats Over the years, many resource string file formats have been standardized for different technologies. Some examples of common resource file formats include... - Android OS Strings Files (.xml) - iOS Strings Files (.strings) - GNU gettext PO / Portable Object Files (.po) - Java Properties files (.properties) - .NET Resource Files (.resx) - Ruby on Rails YAML String Files (.yml) Proprietary Resource File Formats In addition to the common (and somewhat standardized) resource file formats, projects can decide to use proprietary file formats. Master Resource Files ~~ Master Resource File . Translation Job Files ~~ Translation Job File . Report and Info Files Certain of the `localization service methods` (such as the =metrics= method, for example) produce report and info files. Metrics Report Files ~~ Metrics Report File `Metrics report` files are generated by executing the =metrics= `localization service method`. Strings Info Files ~~ Strings Info File `Strings info` files are generated by executing the =metrics= `localization service method`. Translation Audit Report File ~~ Translation Audit Report Files `Translation audit report` files are generated by executing the =auditTranslations= `localization service method`. Language Diff Report File ~~ Language Diff Report Files `Language diff report` files are generated by executing the =diffLanguages= `localization service method`. Usage Report Files ~~ Usage Report File `Usage report` files are generated by executing the =usage= `localization service method`. ### Use With Other Projects . ### TO DOCUMENT wordSplitter ~~ Word Splitter . tokenRegExp ~~ Token Regular Expression . ### Key Concepts Translatable Words . Non-translatable Text . examples... - HTML tags - substitution tokens Brand-specific Resource Strings .