Globalization(G11n) OpenSolaris 2010.03 Test Plan

Author: Petr Hruška (petr.hruska@sun.com)

Main G11n OpenSolaris binary distro test goal is to test entire distribution and ensure the system is working and is usable from global perspective, taking into account wide variations in regions, languages, and cultures. Localization (l10n) and internationalization (i18n) of OpenSolaris operating system introduces multilingual support to the product through the addition of locale support and offers language specific support for the GUI for pre-selected languages and regions. This can result in certain issues and non-expected behavior that needs to be tested

The objectives of this test plan are:

• Outline the tasks required to perform relevant system level testing for the each OpenSolaris binary distro release from g11n perspective.
• To address test strategies, tools, test environment and resources, configurations and detailed system test coverage as well as to address the testing outcomes (deliverables), dependencies and test execution criteria.
• Communicate to G11N community and other interested groups the test coverage planned on the OpenSolaris binary distro releases.

Test Strategy

G11n testing covers both localization and internationalization testing and also distinguishes between Solaris language and locale. The core language/locale support is delivered to OpenSolaris binary distro liveCD, while the full set of g11n features (fonts, input methods, locales, translations ..) is provided through additional packages, which can be installed from official network repositories. OpenSolaris G11n Test Group community is not applying regular build to build test cycles approach, but long-term testing across particular areas of OpenSolaris Binary Distro. Every time a new test build is released we upgrade our test environment to assure we are testing the latest bits available. OpenSolaris 2010.03 test execution will be done with strong focus on new features/projects and pre-integration testing.

Test Infrastructure

G11n test group infrastructure currently consists of this twiki site, test case management system (TCM), test repository and storage space for test data:

• twiki site - currently g11n test group pages are hosted on wikis.sun.com. We will move to OpenSolaris.org, once wiki is available there.
• test case management - TCM - for maintaining test cases and test scenarios, to create test assignments and store results/comments from testing we currently use tool called TCM. You can find TCM here.
• test repository - Our testing IPS repository is reachable at http://pkg.sunvirtuallab.com
• storage space/test data - To store e.g. test data samples, SVR4 test packages or simply any stuff needed for testing, there is some space available for it at sunvirtuallab - you can download test files here.

Test Automation

G11n test automation is built on Common Test Infrastructure (CTI) for Test Environment Toolkit (TET). G11n test suite will be executed regularly for every test build. Detailed test suite decription and coverage can be found here.

Test Execution

Test execution will be performed manually as well as using automation. While official builds will be tested during long term test campaign, test activities for new fixes and deliveries will be tested regularly in pre-consolidation test cycles.

• Pre-Consolidation Testing - this testing will be executed for new deliveries, before their integration to official build. Testing will be done on private test images assembled using distro constructor for testing LiveCD and install environment or using pkg update from test repositories.
• Official Build Testing - this testing will be executed on officially released test builds.

Language/Locale Coverage

Testing of particular languages is organized using tier model. There are three tiers of languages according their usage and support. All available languages/locales are supported and tested from i18n perspective, while full l10n support is granted only for Tier 1 and Tier2 languages.

New locales to be delivered within OpenSolaris 2010.03 release:

¹ English locales are tested just from i18n perspective

Testing Coverage

New features and New Localization Testing

• Goal: To test deeply new features delivered for a release. This area may overlap with other component specific testing other. In such case we are testing new deliveries with higher priority in scope of those particular areas.
  • Addition of new locales - 45+ new locales including new locales for India region, Arabic locales, English and Spanish locales and more .. for complete list see above Language/Locale Coverage section
  • Update CLDR (locale data) from 1.6 to 1.7
  • ibus integration - delivery of iBus input framework, including also m17n library
  • xkeyboard-config update from v1.4 to v1.6
  • New fonts - delivery of new font packages (Kacst fonts version 2.0, Gentium v. 1.02, GNU Unifont v. 5.1, MgOoenFonts, DejaVu, SIL fonts, BPG Fonts)
  • New localization testing - Test Matrix

LiveCD Sanity

• Goal: LiveCD works correctly and provide expected i18n and l10n for particular languages
  • user can boot and login to desktop environment for all available locales
  • I18n (keyboard setting, fonts, locale data ..) and l10n support (UI translations) is available and works correctly
  • Key desktop applications (Firefox, Thunderbird, DDU ...) are working as expected under available locales
  • Test Matrix

Install and Upgrade

• Goal: All g11n bits can be installed and upgraded without any issues
  • G11n packages are fully installable, having correct dependencies
  • Installed language support provides expected i18n and l10n support
  • Update of G11n packages is possible and does not break any functionality
  • GUI and text package manager work for all locales and are properly localized
  • Test Matrix

Keyboard Input

• Goal: All national specific characters can be correctly inputed
  • it is possible to select keyboard layout / input method for every language
  • Xserver configuration - keyboards map testing
  • Compose key testing
  • National keyboard input works correctly for different types of applications (gtk+, OpenOffice.org, Java, etc ..)
  • Test Matrix

Font Quality

• Main Goal: all national specific characters are correctly displayed with expected quality
  • Default fonts setting works for all locales and introduce required national specific support in proper quality
  • Closed fonts can be downloaded from extra repository and can be used after installation
  • Provided fonts are available and works correctly for different types of applications (gtk+, OpenOffice.org, Java, etc ..)
  • Test Matrix

Locale Data

• Main Goal: All environmental variables are correctly set, locale is having correct locale data (date formating, currency, sorting etc ..)
  • Date and time is displayed correctly in GUI and text interfaces (date and cal commands, Clock applet ..)
  • sorting works as expected and does not have negative performance impact
  • checking locale data content against CLDR project
  • Test Matrix

Desktop I18n & L10n Sanity

• Main Goal: Available applications are fully functional under different locales and localized (focusing on key applications e.g. Firefox/Thunderbird )
  • testing OpenSolaris Binary Distro specific components (Time slider, registration, system settings ...)
  • Gnome environment and gtk+ applications
  • Mozilla and email applications (Firefox, Thunderbird, Evolution ...)
  • OpenOffice.org
  • Test Matrix

User Based Scenarios

• Main Goal: Kind of random/user based testing with just rough guidelines to simulate usual end user tasks all done under particular locale/language environment
  • Office work - creating and printing documents, emails, etc ..
  • Multimedia and Communication - playing audio&video, browsing, burning, instant messaging
  • Administration and Development - basic admin tasks, system settings and devtools sanity
  • Test Matrix

Delivered Fixes Verification

• Main Goal: Assure all delivered fixes are verified in official builds
  • verify and close or reopen fixed bugs

Reporting issues

To report bug, use OpenSolaris Bugzilla tracking system. Please read and follow Bug Writting Guidelines when submitting a new issue.

OpenSolaris Bugzilla http://defect.opensolaris.org/bz/

Before reporting new bug, search for existing issues to avoid submitting duplicate bugs.

Pass Criteria

Final Go/NoGo assessment will be based on below criteria. The Go assessment will be given if the final build meets all criteria below:

1. Supported program UI for core locales must be translated.
2. SW Documentation (sw ui/messages, online help) for core locales must be translated (1).
3. Supported Install/Upgrade methods work correctly under all locales.
4. G11n packages are installed properly and do not cause major system defect (P1 or stopper bug) with installation/upgrade.
5. Locale support can be added or removed after installation. Adding/removing locales do not cause major system defect (P1 or stopper bug).
6. User can login into desktop session and all supported applications are usable under all locales.
7. User can input language specific characters.
8. No stopper bugs and all P1s and P2s must be resolved or triaged by the technical lead and determined not to be a stopper for the release.
9. G11n content does not cause any legal exposure.

(1) Note: Some projects can be delivered with translated program UI and not translated Online Help.

Links & Docs

• OpenSolaris Project - http://opensolaris.org/os/
• Internationalization and Localization OpenSolaris Community - http://www.opensolaris.org/os/community/int_localization/
• OpenSolaris defect tracking - http://defect.opensolaris.org/bz/
• Common Test Infrastructure (CTI) for Test Environment Toolkit (TET) - http://opensolaris.org/os/community/testing/testsuites/ctifortet/
• G11n new features - Solaris Engineering Programs: http://wikis.sun.com/display/g11n/Solaris+Engineering+Programs
• International Components for Unicode - Locale Explorer: http://demo.icu-project.org/icu-bin/locexp