Revolution Newsletter

One of the most important parts of the upcoming 2.9 release is that it contains an updated Linux engine full to the brim with features that were previously only available on Windows and Mac OS X. In particular, the old UNIX external mechanism has been replaced by one using shared libraries - thus gaining the same performance and implementation ease as the other platforms.

In this article, we will introduce an updated version of the External Creator and show how to use it to produce a simple external that can be used on Linux.

Before we begin

Setting up the environment

As with the other platforms, before doing anything in a lower-level language on Linux you need to set up your build environment appropriately. On Windows and Mac OS X there are obvious choices for an IDE to build your external in, however on Linux there is no clear winner in this regard and so we have opted for a simple Makefile system.

To help with this, the new version of the Externals Environment and External Creator is able to generate Makefiles in a simple structure for use on Linux, and with a few simple terminal commands you can be up and running with Linux external development in no time.

So, before continuing take some time to unpack the ExternalsEnvironmentV3.zip file into a suitable place on your hard-drive and take a quick look through its contents. Also feel free to rename the unpacked folder ExternalsEnvironmentV3 to something more to your liking - but make a note of its location as this will be needed later when compiling and running your externals. (In future, we will refer to this folder as the environment folder).

A simple external - wrapping iconv

Setting up the project

As an introduction to external development for Linux, we are going to wrap a simple system library call (iconv) that is used for text conversions allowing access to its features inside Revolution.

Having unpacked the environment and put it in a suitable place, we now need to generate the skeleton files for our external. So, without further ado, load up your copy of Revolution on Linux and open the 'External Creator V3.rev' stack inside your environment folder.

fill in the Name field with rnaiconv (as in other external articles, we using the three-letter prefix rna to stand for Revolution Newsletter Article)

Once you are happy with your settings, just click Generate and the creator will chug away for a moment producing the necessary files.

Exploring our new project

Having generated the project, you should now find an rnaiconv folder within the environment, as well as two files in the root of the environment folder - Makefile and run (more on these later).

Taking a look in the rnaiconv folder should reveal a number of files and folders:

Now we have this basic structure we can go onto actually implementing some functionality...

All about iconv

The iconv command is ubiquitous on Linux systems. This flexible tool allows you to convert text files between different encodings on the command line (to find out more about this type man iconv at a terminal prompt).

As it turns out, this tool simply wraps a system library call that provides this ability - and our external will give us direct access to this ability without need to resort to the shell or open process Revolution syntax.

To use it is simple, you first open the converter with the appropriate source and target encodings, do your conversion, then close the converter. We will wrap this sequence of calls using an external command with the following prototype:

command rnaIconv pSrcEncoding, pDstEncoding, pInTextVar, pOutTextVar

Something to note here is that we have to pass a variable names to the function for both input and output - this is because we cannot pass arbitrary binary data to external commands. To get around this, we pass variable names and then use the GetVariableEx and SetVariableEx calls to get and set their values (for more details on these two calls see the previous external writing article: External Writing for the Unititiated - Part 2)

Before going any further, make sure you have your favourite linux text editor loaded (I tend to use gedit) and open the rnaiconv.cpp file.

Sorting out the declarations

Before actually implementing our function, we first need to sort out a couple of declarations - we need to import the declarations of the system functions we wish to use, as well as declare to Revolution that we are implementing an external function.

To do the former, we just need to include the relevent header files. To do this we use #include directives. The one to use to gain access to iconv is <iconv.h>. So, add the following lines just on the line after the BEGIN USER DEFINITIONS comment:

#include <cstdlib>

#include <cstring>

#include <cerrno>

#include <iconv.h>

To actually declare our external function to Revolution we need to add a line to the externals export table. In our environment this is easy, there are a number of macros we use to do this declared in <revolution/external.h>. In this case, we use the EXTERNAL_DECLARE_COMMAND one. Place the following line on the line after the BEGIN USER DECLARATIONS comment:

EXTERNAL_DECLARE_COMMAND("rnaiconv", rnaIconv)

Implementing the function

Although the method used to build externals differs between platforms, the actual Externals API is identical - a Linux external function or command is defined in exactly the same was as on the other two platforms, and such functions and commands have access to the same set of engine calls. As these details have been covered in previous articles, we'll instead go straight onto the function implementation.

Add the following function to the rnaiconv.cpp file after the #include <iconv.h> directive we've already added:

// Function:
// rnaIconv pSrcEncoding, pDstEncoding, pNameOfInTextVar, pNameOfOutTextVar
// Parameters:
// pSrcEncoding - the name of the source encoding
// pDstEncoding - the name of the destination encoding
// pNameOfInTextVar - the name of the variable containing the text we want
// to convert
// pNameOfOutTextVar - the name of the variable containing the text we want
// to place the converted text into
//
void rnaIconv(char *p_arguments[], int p_argument_count,

char **r_result, Bool *r_pass, Bool *r_err)

{

int t_success;

// First check we have been passed 4 arguments and throw a (Revolution)
// exception if not.
if (p_argument_count != 4)
{

*r_result = strdup("wrong number of parameters");
*r_err = True;
*r_pass = False;
return;

}

// Next we attempt to fetch the source text data we wish to use.
// To do this we use the 'GetVariableEx' call from the Externals API
ExternalString t_src_text;
GetVariableEx(p_arguments[2], "", &t_src_text, &t_success);
if (t_success == EXTERNAL_FAILURE)
{

*r_result = strdup("unknown input variable");
*r_err = True;
*r_pass = False;
return;

}

// Next attempt to create an iconv converter object using the given
// source and destination encodings.
// If creation fails, we throw a (Revolution) exception.
iconv_t t_converter;
t_converter = iconv_open(p_arguments[1], p_arguments[0]);
if (t_converter == NULL)
{

*r_result = strdup("unable to create converter");
*r_err = True;
*r_pass = False;
return;

}

// We now convert the source text we have into an output buffer
// that we extend as appropriate.
//
// iconv works by attempting to convert as much input as possible
// into an output buffer of a given size, therefore we loop,
// extending the buffer on each iteration until no input remains.
//
// At each iteration, we allocate twice the remaining input size
// which should be fine for most encoding conversions.

// If the variable is unknown, throw an error.
// As the iconv call modifies the input buffer and length parameters
// we make copies here.
char *t_in_text_ptr;
size_t t_in_text_left;
t_in_text_ptr = (char *)t_src_text . buffer;
t_in_text_left = t_src_text . length;

// Setup initial values for the out text buffer. These initial values
// are 0/NULL since the first thing we do in the loop is allocate
// memory for that iteration.
char *t_out_text_base;
size_t t_out_text_frontier;
size_t t_out_text_limit;
t_out_text_base = NULL;
t_out_text_frontier = 0;
t_out_text_limit = 0;

// Flags that indicate what error occured (if any)
bool t_memory_error;
t_memory_error = false;

bool t_conversion_error;
t_conversion_error = false;
while(t_in_text_left != 0)
{

// Extend our output buffer.
char *t_new_out_text_base;
t_new_out_text_base = (char *)realloc(t_out_text_base, t_out_text_limit + t_in_text_left * 2);
if (t_new_out_text_base == NULL)
{

t_memory_error = true;
break;

}

t_out_text_limit += t_in_text_left * 2;
t_out_text_base = t_new_out_text_base;

// Compute start pointer and bytes remaining for the out text buffer
char *t_out_text_ptr;
t_out_text_ptr = t_out_text_base + t_out_text_frontier;

size_t t_out_text_left;
t_out_text_left = t_out_text_limit - t_out_text_frontier;

// Attempt the conversion
size_t t_iconv_result;
t_iconv_result = iconv(t_converter, &t_in_text_ptr, &t_in_text_left, &t_out_text_ptr, &t_out_text_left);
if (t_iconv_result == (size_t)(-1) && errno != E2BIG)
{

t_conversion_error = true;
break;

}

// Update the output buffer's frontier (i.e. increase by the number
// of bytes newly converte).
t_out_text_frontier += (t_out_text_limit - t_out_text_frontier) - t_out_text_left;

}

if (t_memory_error)
{

// If a memory error occured, throw an error.
*r_result = strdup("out of memory");
*r_pass = False;
*r_err = True;

}
else if (t_conversion_error)
{

// If a conversion error occured, throw an error.
*r_result = strdup("conversion error");
*r_pass = False;
*r_err = True;

}
else
{

// No errors occured during conversion so attempt to write out result
// into the output variable
ExternalString t_out_text;
t_out_text . buffer = t_out_text_base;
t_out_text . length = t_out_text_frontier;
SetVariableEx(p_arguments[3], "", &t_out_text, &t_success);

// If the variable is unknown, throw an error.
if (t_success == EXTERNAL_FAILURE)
{

*r_result = strdup("unknown output variable");
*r_pass = False;
*r_err = True;

}
else
{

*r_result = strdup("");
*r_pass = False;
*r_err = False;

}

// Deallocate our out text buffer
free(t_out_text_base);

// Close the iconv converter
iconv_close(t_converter);

}

This function is basically an implementation of what we have outlined above - it opens an iconv converter, processes all the input, then closes the converter.

Building our external

Now that we've coded our function, we need to build and test it. As already mentioned, the External Creator sets up a simple Makefiles for building on Linux. A Makefile is simply a text file that describes the commands required to convert the source files of a project into an actual executable or shared library.

The ones the External Creator generates are predefined to make it easy for you to get started with external writing. Two Makefiles will have been generated, one in the environment and one in the rnaiconv folder. If you take a look at the latter, you should see the following:

NAME=rnaiconv
TYPE=library

SOURCES=rnaiconv.cpp

# To add your own source files put them into this list.
# Note that source files are searched for in src/
#
CUSTOM_DEFINES=

# Add any custom include directions needed by your
# external to this list
#
CUSTOM_INCLUDES=./src

# Add any custom static libraries needed by your external
# to this list
#
CUSTOM_STATIC_LIBS=external

# Add any custom dynamic libraries needed by your external
# to this list
#
CUSTOM_DYNAMIC_LIBS=

# Add any specific compiler options you need here
#
CUSTOM_CCFLAGS=-fno-exceptions -fno-rtti

# Add any specific link options you need here
#
CUSTOM_LDFLAGS=

include ../configurations/library.linux.makefile

As you can see from the comments, this file is ready to be added to as your project grows. The most important line here is the 'SOURCES' definition. To add more source files to your external, simply append the name of the file to this line (making sure they are separated by spaces). (Note that due to the design of the system, you need to make sure that your source files are all located in the src sub-folder).

The project Makefile is not designed to be invoked directly, instead it is called by the Makefile in the environment folder when all the project's dependencies are also built - this allows more complicated projects to be built up by decomposing them into smaller sub-projects present in separate folders with their own Makefiles. In this case, your external project will currently depend on libexternal - the standard glue-code required for a Revolution external.

Additionally, the environment Makefile offers three targets for each project - one to build in a debug configuration, one to build in a release configuration and one to clean the intermediate files thus enabling a full rebuild. To use these targets you would use the commands:

After a few moments, control should return to you without any error messages having been reported.

(If you do have problems and things don't seem to compile correctly, the NewsletterArticle3.zip archive you will have downloaded contains a pre-prepared set of files that should work straight out of the box. Simply unpack this archive, and copy the folder and other files into your environment folder. Compiling as above should then result in everything working as expected.)

Testing our external

Now that we've built our external, we need to test it. The External Creator will have already set up an empty test stack for us to use. To make it even easier, it will also have created a simple run script in the environment folder.

This run script combines both building and launching into a single command. To use it just type the comand ./run rnaiconv debug at your terminal and you should find Revolution popping up in due course with a blank stack.

Once this is done, put some text in the input field (I always find "Hello World!" a useful phrase) and just click the button. What you should see is "Hello World!" in the output field, except spaced out - this is because the output encoding we've chosen has two bytes per character, which for Roman script text will have one NUL byte per character.

To see something more interesting, try entering some input text with accented characters and use "ISO_8859-1" for the source encoding and "MACINTOSH" for the output encoding. In this case you should changes which comes from the fact that accented characters are in different places for these encodings.

To get a full list of the encodings that iconv supports you can use the command iconv --list at the terminal prompt.

Moving forward

This article has hopefully demonstrated how straightforward it can be to produce an external for Linux. You can use the External Creator as many times as you want inside a given environment folder to create additional externals, all of which can be built and run using the run script in a similar manner to that described above the rnaiconv project.

When you are happy with your external and want to install it permanently in your distribution, you just copy the shared library into the appropriate place inside your My Revolution <Edition> folder. Full details of this can be found in a previous article External Writing for the Uninitiated - Part 1. You can find the actual shared library file you need for this inside the _build/release/ or _build/debug/ folders in your environment folder.

Introduction