Skip to main content

Internationalising Shell Scripts with Gettext

As part of my work for the F123 Project I've had to learn how to prepare bash shell scripts for translation into multiple languages using gettext.

This is a little workflow document I have written on the subject.

Shell Internationalisation Workflow

To provide translations for shell scripts we use the gettext package.

Preparing a Shell Script for Translation

Shell scripts contain echo commands to provide user feedback.

We need for the strings displayed by an echo command to be displayed in the language the user of the machine is using.

In the top of the script we put these lines:

export TEXTDOMAIN=myscript
export TEXTDOMAINDIR=/usr/share/locale

. gettext.sh

In the above example myscript is an example script name.

The TEXTDOMAINDIR above contains the usual path to localization for a typical Linux installation. It may be raplaced with:

export TEXTDOMAINDIR=$PWD/locale

For testing a script's translation.

The script author changes typical cho of this syntax:

echo 'Press any key to continue'

With:

echo $(gettext "Press any key to continue") ; echo

We place an extra echo with no arguments at the end because the gettext function does not add a line-feed.

The gettext function is contained in the gettext.sh script included at the top of the script.

If an echoed string contains any shell variables they must be replaced.

For example:

echo "Program name: $0"

Becomes:

PROGNAME=$0
echo $(eval_gettext "Prog name: \$PROGNAME") ; echo

The eval_gettext function is used instead of gettext and the dolla-sign is escaped with a back-slash.

Again an extra echo is put in after the command to echo a translated string.

Extracting Strings into a Portable Object Template (.pot)

Now we run xgettext against our script to create a file of strings to translate.

We do this in this fashion:

xgettext -L Shell -o myscript.pot myscript

This reads all the echo commands which contain references to the gettext and eval_gettext functions and writes them to a .pot (portable object template) file.

Here is a simple example of the contents of a .pot file for a script which contains only one echo command:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2017-04-19 22:52+0100\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
#
#: myscript:10
#, sh-format
msgid "Hello world: $PROGNAME"
msgstr ""

In the above example there are values in upper-case at the top in the comments which should be filled out with the details of the package-name, and the author details etc.

And this line:

"Content-Type: text/plain; charset=CHARSET\n"

Should be changed to read:

"Content-Type: text/plain; charset=UTF-8\n"

Or whichever character-set we are using.

The Translation Process

The first-cut .pot file should be passed to a translator with the name changed to drop the ending t:

cp myscript.pot myscript.po

Now give the .po (portable object) file to a translator.

After the line which reads:

  "Content-Transfer-Encoding: 8bit\n"

There will be entries of the form:

  msgid "Some string in the original language"
  msgstr ""

A translator places the translation of the string in the msgid entry in the msgstr entry between the quotes.

The translator MUST preserve any shell variable reference that is in the original string. For example:

msgid "Attention! $ERRORCODE"
      msgstr "Achtung! $ERRORCODE"

In this way a translator goes right through the file and translates the strings, leaving the msgid entries as-is, and placing their translations in the corresponding msgstr entries.

Creating Machine Object Files

The files which will actually be used when a script runs, to provide translated strings is a .mo file.

We create this from a .po file like this:

msgfmt -o myscript.mo myscript.po

Which will create myscript.mo from myscript.po.

Merging New Strings

Of course the development of a complex script is never static and it may be necessary to add new strings, or to remove old ones. We can use another utility to merge strings from a new .pot file into our .po file and re-create our .mo file.

For this we use msgmerge, like this:

msgmerge -U myscript.po myscript.pot

This will search the .pot file for new or deleted strings and merge them into the .po file. In this way previous translation work is not lost when things change.

And again we then create a new .mo file from the update .po file.

Installing a Translation

Using our myscript example from above, the .mo file is installed in:

  /usr/share/locale/<LANG>/LC_MESSAGES/myscript.mo

Where is the language into which the script has been translated.

For example, for Brazilian Portuguese:

/usr/share/locale/pt_BR/LC_MESSAGES/myscript.mo

The Worklow in Short

  1. Script is written and tested, including gettext support.
  2. Run xgettext to extract all echo strings into a .pot file.
  3. Rename the .pot file to .po and pass to translator.
  4. When step 3 is done, run msgfmt to create the .mo file.
  5. Install the .mo file (after testing).
  6. If something new is written in the script, go to step 2 and repeat to step 5.

Notes/Suggestions

The .po files to be translated should probably either be contained in a directory structure which clearly defines the language, or should contain the language in the file-name, although the completed .mo file cannot.

So, for myscript, perhaps:

myscript/
    en/myscript.en.po (English)
            fr/myscript.fr.po (French)
                        pt_BR/myscript.pt_BR.po (Brazilian Portuguese)

How to Generate a Table of Contents in a Kramdown Document

Here's how to generate a table of contents in kramdown.



# Heading
{:.no_toc} 

## Table of Contents
{:.no_toc}

* TOC 
{:toc} 

Introduction

The kramdown gem converts a kramdown file into html. It generates the id tags for each heading and uses them to generate a table of contents for a document. This becomes obvious if you take a file written in kramdown and convert it to html.

kramdown input-file.md > output-file.html

The ids are put on the generated html for all headings.

How To Automatically Generate A Table of Contents

Typically, you will want to have a numbered list for your table of contents or a bulleted list of entries. This is done by either writing 1. TOC or * TOC at the place in your file where you want the table of contents to be located. Underneath this line, specify the table of contents itself:



{:toc}

You can have either a numbered list of items or a bulleted list, in the normal way for lists:



1. TOC 
{:toc} 

* TOC 
{:toc} 

If you do not want a heading to be included in the table of contents, put:



`{:.no_toc}

Underneath the entry you want to exclude from the table of contents, often your top-level heading.

Notes

I have noticed that there is either a bug or an accidental omission in the generation of the toc. For example,, given headings like this:



# First level one heading
## First level two heading 
## Second level two heading 
## Third level two heading
# Second level one heading

I would expect the second level one heading to be a member of the list of level one headings, but it is left hanging in between two lists of second level headings, and is a member of no list. This bug makes sub-lists and then a return to a lower level further down impossible.

So you will need to have all headings which go to make up a table of contents the same level.

A bit annoying but it beats constantly scrolling up and down a document as you write it to try to remember the order of the headings and their names.

Tool for Installing Talking Arch

I've written a bash script that will install the Talking Arch distro non-interactively.

To use it follow these steps:

  1. Boot Talking Arch
  2. Set up your partitions
  3. `wget http://www.raspberryvi.org/Downloads/ta-tools-1.0.0.tar.gz
  4. tar zxf ta-tools-1.0.0.tar.gz
  5. Make sure the contained script is executable and run it.

It will do the first pre-chroot of the install and write a here script into the chroot and mark it executable.

After the chroot, run the newly created script to create the installation.

Here is the full script:



#!/bin/bash
#
# Talking Arch setup
#

# Change the commenting of the variables below
REGION=Europe
#REGION=America

CITY=London
#CITY=Chicago

LOCALE=en_UK
#LOCALE=en_US

KEYMAP=uk
#KEYMAP=us

DEVICE=${1}

HOSTNAME=arch

ROOTPASSWORD=root

USER=user
PASSWORD=password

# This MUST be a full path to where the script must be in the chroot
HERE_SCRIPT=/mnt/root/001-ta-tools.sh

# Crash out on any error
set -e

echo 'Mounting the disk partition(s)...'
mount ${DEVICE} /mnt

echo 'Pacstrapping the new system in what will be the chroot...'
pacstrap /mnt git base espeakup alsa-utils openssh acpi wget base-devel nmap lynx

echo 'Setting up fstab...'
genfstab -U -p /mnt >> /mnt/etc/fstab

#
# Write a 'here' script into the chroot before we chroot
#
cat  ${HERE_SCRIPT}
#!/bin/bash

set -e

echo 'Setting up localisation stuff...'
sed -i 's:#${LOCALE}.UTF-8 UTF-8:${LOCALE}.UTF-8 UTF-8:' /etc/locale.gen
echo "LANG=${LOCALE}.UTF-8" > /etc/locale.conf
export LANG=${LOCALE}.UTF-8
locale-gen

echo 'Removing /etc/localtime if it exists before replacing it...'
[ -f /etc/localtime ] && rm /etc/localtime
ln -s /usr/share/zoneinfo/${REGION}/${CITY} /etc/localtime

echo 'Setting up keymap...'
echo "KEYMAP=${KEYMAP}" > /etc/vconsole.conf

echo 'Setting the hardware clock...'
hwclock --systohc --utc

echo 'Setting the hostname...'
echo ${HOSTNAME} > /etc/hostname

echo 'Setting the password for root...'
echo -e "${ROOTPASSWORD}\n${ROOTPASSWORD}\n" | passwd root

echo 'Adding an ordinary user...'
useradd -m -g users -G wheel,storage,power,audio -s /bin/bash ${USER}
echo -e "${PASSWORD}\n${PASSWORD}\n" | passwd ${USER}

echo 'Editing /etc/sudoers to grant the usual privileges...'
sed -i 's:# %wheel ALL=(ALL) ALL: %wheel ALL=(ALL) ALL:' /etc/sudoers

echo 'Installing grub...'
pacman -S --noconfirm --noprogress grub-bios

echo 'Setting up grub...'
grub-install --target=i386-pc --recheck /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

echo 'Setting some stuff in /etc/hosts.allow...'
echo 'SSHD: ALL' >> /etc/hosts.allow

echo 'Enabling the sshd service...'
systemctl enable sshd.service

echo 'Enabling the dhcpcd service...'
systemctl enable dhcpcd.service

echo 'Enabling the espeakup service...'
systemctl enable espeakup.service

echo 'About to exit chroot...'
exit

echo 'un-mounting hard-disk partitions from virtual mount point...'
 umount /mnt/home
 umount /mnt

EOF

# Mark the created here script executable
chmod +x ${HERE_SCRIPT}


echo 'All done'
exit 0

Installing `yaourt` on Arch Linux

You can install yaourt from the AUR:

git clone https://aur.archlinux.org/package-query.git
cd package-query
makepkg -si
cd ..
git clone https://aur.archlinux.org/yaourt.git
cd yaourt
makepkg -si
cd ..

Examples

Search and install:

yaourt Sync database, upgrade packages, search aur and devel (all packages based on dev

version) upgrades:

yaourt -Syu --devel --aur Build package from source:
yaourt -Sb Check, edit, merge or remove *.pac* files:
yaourt -C Get a PKGBUILD (support splitted package):
yaourt -G Build and export package, its sources to a directory:
yaourt -Sb --export Backup database:
yaourt -B Query backup file:
yaourt -Q --backupfile

Notes

Yaourt gives an opportunity to edit the PKGBUILD and if the answer is 'y' it asks which editor to use if none is selected by your environment. Type in 'nano'. When getting tclx for use with emacspeak you will need to add armv7 if it is for a Pi2.

Nikola plugin for kramdown

I’m not surprised that markdown has become so successful and ubiquitous. But one of the things I think it lacks is syntax for creating tables.

For this reason, I have been using a txt2tags plugin for a while. My sites all have a lot of tabular data, both as a function of the nature of the sites and also because I just like tables.

But I recently discovered kramdown.

kramdown has a super-set of markdown syntax, including simple, but powerful table-creation syntax.

So I’ve written a plugin for Nikola along the lines of the txt2tags plugin to compile HTML posts and pages from kramdown files. For which I have started using the .kd extension.

This post is written in kramdown.

So this post is somewhat by way of a test of my kramdown plugin for Nikola.

Third-level Heading

The above heading is marked with two hash characters and in the same way that markdown posts have their heading demoted by one level I have done the same with kramdown.

Here is a table of silly data:

Date Weather Activity
01-Feb-2016 Rain Stayed in
02-Feb-2016 Dry and cloudy Stayed in again
03-Feb-2016 Cold, -4C Had to go out for most of the day
04-Feb-2016 Even colder than yesterday Stayed in and wrote a kramdown plugin for Nikola

Lists

Lists are the same as in markdown

  • unordered list item one
  • Unordered list item two

Links

The links in the above text are all done like references are done in markdown, with the actual URI of each link at the bottom of the text file.

Atom.io Editor Totally Inaccessible

I've just tried the atom.io editor.

I did this because there was to be a talk about it at the Portsmouth LUG, which I went to last Saturday.

It's totally inaccessible on Linux, Debian with the Mate desktop.

Apparently it is an editor written in Javascript and, if I have this correct, it renders the screen with the Chrome browser, or sub-system. There's also something called node.js involved.

These are technologies I know little about, except that it doesn't make a sound with Orca.

So there's another editor we can't use.

Back to good old Emacs.

Transmission Command-line Bit-torrent Programs

I was after a method of downloading and seeding bit-torrents exclusively from the command-line.

I tried aria2 but found it unreliable. I don't know if it was because I was doing something wrong but none of the torrent files I downloaded were not corrupted in some way.

And anyway I wanted something to run in the background.

I tried transmission-cli. Better results but again this was not something that just ran in the background.

So I installed a pair of related programs: transmission-daemon and transmission-remote.

The Daemon part of this program is exactly that, a daemon that runs, as all daemons do, in the background.

And the Remote part is a command-line program to control it.

The remote part is included in the transmission-cli package.

So:

$ sudo apt-get install transmission-daemon transmission-cli

Will install them both on Debian or Debian derivatives. Although why you're running a derivative when you could run Debian I don't know.

The default installation of transmission-daemon is configured for the user it runs to be set to transmission and the password is also set to transmission.

And by default it is only possible to access the daemon and the management web page from the local machine (localhost).

This led to a tedious situation, any command entered to make transmission-remote do anything with the daemon meant this was the command-line:

$ transmission-remote localhost --auth transmission:transmission <command>

I don't want to have to type that in every time.

So I set up an alias in .bash_aliases:

alias trem='transmission-remote localhost --auth transmission:transmission'

Now I can just type:

$ trem <command>

For example, to add a torrent:

$ trem --add <torrent>

To list all the current torrents and their status:

$ trem --list

To remove and delete a torrent:

$ trem -tN --remove-and-delete

(where N is the number in the list returned by the --list command).

To start a torrent:

$ trem -tN --start

And to stop one:

trem -tN --stop

There are loads of other commands but these are the ones I use the most, and mostly --add and --list.

Note also that transmission will handle the magnet links found on the Pirate Bay download site.

Capturing Regions of the Screen

Here are some nice keystrokes that work in Debian, and probably other distros as well as these are essentially Gnome actions:

  • PrintScreen -- Takes a screenshot of your entire desktop and saves it to the Desktop folder
  • Alt + PrintScreen -- Saves a screenshot of the focused window to the Desktop Folder
  • Shift + PrintScreen -- Lets you select an area of the screen, and saves to the Desktop Folder
  • Ctrl + PrintScreen -- Takes a screenshot of your entire desktop and copies it to the clipboard
  • Ctrl + Alt + PrintScreen -- copies a screenshot of the focused window to the clipboard
  • Ctrl + Shift + PrintScreen -- Lets you select an area of the screen, and copies it to the clipboard
  • Ctrl + Shift + Alt + R -- Records a Screencast) of your entire desktop and saves it to your Videos folder

The 'Desktop' folder may be different on other flavours, like 'Pictures' on Fedora.

The commands which allow selection of an area are not much use when you can't see but the others are nice. I particularly like the idea of a screen-cast.

Thanks to the guys on the Surrey Linux User Group email list for these.

More about WYSIWG GUI Designers and VI Programming

I've not always been blind.

For about twenty years I was a programmer, with a varying level of visual impairment.

Before I lost my sight very quickly in 2008 I had a level of sight which was the same for a long time, although it was gradually decreasing.

What this meant to me as a programmer in the early days of MS Windows was:

  • Keeping track of the mouse-pointer on the screen was tiring and not always easy
  • Using Windows graphical IDEs to design forms was hence not always comfortable for long periods
  • It was easier for me to rough-out a form, for example in MS Access with the IDE and then tweak the code with a text editor

For years I used QEdit, which was a very lovely text editor for MSDOS. I miss it.

Actually losing my sight totally, which made it necessary to embrace the use of a screen-reader made things a bit easier in some ways, and flipped things around a bit:

  • Now using the GUI on either MS Windows or Linux was easier, assuming the applications used are accessible

But, now using a drag-and-drop GUI designer was no longer possible, or at least not easy.

GUI Programming using Absolute Positioning

I used to write applications for MS Windows in vanilla C, using the native Win32 API calls.

That involved either generating controls with calls to CreateWindow and it's variants or by using resource (.rc files).

That made programming once I lost my sight difficult because:

  • I could no longer tell if my controls were all lined up correctly vertically and horizontally
  • My head was spinning with arithmetic every time I tried to size and position one control relative to the previous/next one

This approach, using absolute positioning also falls down when another user is not using the same sized font as you, or when the user clicks and drags the corner of your window to resize it. The formatting goes all to pot and looks like a dog's breakfast.

But more recent development tools have started using sizers.

In the sizer approach, controls are added to a kind of invisible net which controls the positioning.

Visualise it like this:

  1. There is a clear flat surface in front of you
  2. You need to place objects on that surface ina controlled way
  3. Imagine an invisible shield which is floating a couple of inches above the surface
  4. Cut into that shield are a number of holes through which you can place the objects on the surface
  5. When the objects are placed, changing the size of the surface will cause the invisible shield to change size as well and the objects will move and resize in a known and predictable way

I know of several GUI toolkits and languages that do this:

  • GTK, on Linux, and (I think) Windows and Mac
  • wxWidgets, multi-platform
  • wx.Python, multi platform
  • Java, multi platform

Varieties of sizer

Each of these tools has a variety of sizer types. Here is a by-no-means exhaustive list of them and a description of each:

Box Sizer

In either horizontal or vertical variants. Controls added either stack across (horizontal) or down/up (vertical).

Grid Sizer

A number of variants exist but essentially an X and Y grid of cells which are filled with controls by specifying their coordinates.

Sizers within Sizers

It is possible to add a sizer to another sizer to create complex GUIs.

For example if your form had a horizontal box sizer into which you have placed two vertical box sizer sizers, you will get left and right vertical sizers.

I know this explanation sounds complex but it really not.

One thing to remember is that the parent of a control is NOT the sizer into which it is placed but the object, usually termed something like panel, to which the sizer containing the control is added.

Using these sizer objects to position controls you will get:

  • A window which handles resizing and font changes correctly
  • A much easier life with no absolute-positioning headaches like arithmetic-overload

Sizers usually have some properties which control the way controls placed upon them will expand, contract and position themselves when the window is stretched or shrunken.

I still don't find WYSIWYG/drag-and-drop form designers very easy, or even possible to use, being totally blind, but the sizer approach has given me back a way to write complex and visually pleasing applications even though I can't see.

And of course the sighted authors of drag-and-drop IDEs tend to either just ignore accessibility, or assume that using them when you can't see is impossible. So keyboard shortcuts may not be included.

My personal favourite toolkit at the moment is wxWidgets, which is giving me a way to teach myself C++. Previously I stuck to C like a limpet.

I have yet to get to grips with:

  • GTK programming in C
  • Qt

Talking of Qt I believe accessibility is improved in version 5. It's been getting better and better. And now my C++ is coming on apace I will take a look at it.

All this is very exciting for the visually impaired programmer in a world where it is not as acceptable any more for an application to only run on one platform.