#contents

Introduction

This case study introduces how to componentize simple image processing.

Using an existing camera component and image display component, a component is created that performs horizontal (or vertical) image flipping on images from a camera, and a system is built to display the flipped camera image.

Although image flipping can be implemented easily, OpenCV is used here to simplify implementation further and create a more versatile RT Component.

What is OpenCV?

OpenCV is an open-source computer vision library that was formerly developed and released by Intel and is currently developed and maintained by Itseez.

Excerpted from Wikipedia.

RT Component to be Created

• Flip Component: An RT Component that performs image flipping using the cv::flip() function, one of the various image processing functions provided by the OpenCV library.

Converting the cv::flip Function into an RT Component

An RT Component that flips input images horizontally or vertically and outputs the result is created using the OpenCV library’s cv::flip function. The development and execution environment assumes Code::Blocks (gcc compiler) running on Ubuntu.

The target version of OpenRTM-aist is 1.1.2.

The overall procedure is as follows.

• Verify the runtime and development environments
• Review OpenCV and the cv::flip function
• Define the component specifications
• Generate a source code template using RTCBuilder
• Implement activity processing
• Verify component operation

About the cv::flip Function

The cv::flip function flips image data of type cv::Mat, which is commonly used in OpenCV, around the vertical axis (horizontal flip), horizontal axis (vertical flip), or both axes (horizontal and vertical flip).

The function prototype and the meanings of its input/output arguments are as follows.

 void flip(const Mat& src, Mat& dst, int flipMode)

 src       Input array
 dst       Output array. If dst=NULL, src is overwritten.
 flipMode  Specifies how the array is flipped:
   flipMode = 0: Flip around the X axis (vertical flip)
   flipMode > 0: Flip around the Y axis (horizontal flip)
   flipMode < 0: Flip around both axes (horizontal and vertical flip)

Component Specifications

The component to be created will be called the Flip Component.

This component has an image data input port (InPort) and an output port (OutPort) for outputting the flipped image.

The names of the ports are:

• Input Port (InPort): originalImage
• Output Port (OutPort): flippedImage

OpenRTM-aist includes sample vision-related components that use OpenCV.

The data ports of these components use the following CameraImage type for image input and output.

   struct CameraImage
     {
            /// Time stamp.
            Time tm;
            /// Image pixel width.
            unsigned short width;
            /// Image pixel height.
            unsigned short height;
            /// Bits per pixel.
            unsigned short bpp;
            /// Image format (e.g. bitmap, jpeg, etc.).
            string format;
            /// Scale factor for images, such as disparity maps,
            /// where the integer pixel value should be divided
            /// by this factor to get the real pixel value.
            double fDiv;
            /// Raw pixel data.
            sequence<octet> pixels;
     };

This Flip component also uses the same CameraImage type for its InPort and OutPort so that it can exchange data with these sample components.

The CameraImage type is defined in InterfaceDataTypes.idl, and in C++ it becomes available by including InterfaceDataTypesSkel.h.

In addition, there are three possible image flip directions: horizontal flip, vertical flip, and horizontal+vertical flip.

To allow this to be specified at runtime, the RT Component configuration mechanism will be used.

The parameter name will be flipMode.

To match the specification of the cv::flip function, flipMode is defined as type int, and the values 0, 1, and -1 are assigned to vertical flip, horizontal flip, and horizontal+vertical flip respectively.

The image processing behavior for each value of flipMode is shown below.

Image Flip Patterns of the Flip Component for Each flipMode Setting

The specifications of the Flip component are summarized below.

Component Name	Flip
InPort
Port Name	originalImage
Type	CameraImage
Description	Input image
OutPort
Port Name	flippedImage
Type	CameraImage
Description	Flipped image
Configuration
Parameter Name	flipMode
Type	int
Default Value	0
Constraint	(0,-1,1)
Widget	radio
Description	Flip Mode Vertical Flip: 0 Horizontal Flip: 1 Horizontal and Vertical Flip: -1

Runtime Environment and Development Environment

A development environment is built on Linux (Ubuntu 16.04 is assumed here).

Install using the installation script.

 $ wget http://svn.openrtm.org/OpenRTM-aist/tags/RELEASE_1_1_1/OpenRTM-aist/build/pkg_install_ubuntu.sh
 $ sudo sh pkg_install_ubuntu.sh

Installing OpenRTP

Download and install the Linux version of OpenRTP (integrated environment for component development tools and system development tools) from this URL.

Java is also required to run OpenRTP, so install the default-jre package.

 $ apt-get install default-jre
 $ wget http://openrtm.org/pub/openrtp/packages/1.1.2.v20160526/eclipse442-openrtp112v20160526-ja-linux-gtk-x86_64.tar.gz
 $ tar xvzf eclipse442-openrtp112v20160526-ja-linux-gtk-x86_64.tar.gz

After starting Eclipse, RTSystemEditor may not be able to connect to the Name Server. In that case, add your host name to the localhost entry in /etc/hosts.;

 $ hostname
 ubuntu1404 ← The host name is ubuntu1404
 $ sudo vi /etc/hosts

 127.0.0.1       localhost
 Change it as follows:
 127.0.0.1       localhost ubuntu1404

Installing CMake

 $ sudo apt-get install cmake cmake-gui

Installing OpenCV and OpenCV Components

Install OpenCV and the OpenCV components.

First, install the OpenCV packages provided by Ubuntu as follows.

 $ sudo apt-get install libopencv-dev libcv2.4 libcvaux2.4 libhighgui2.4

Next, check out the source code from the repository and compile it manually.

 $ svn  co http://svn.openrtm.org/ImageProcessing/trunk/ImageProcessing/opencv/
 $ cd opencv
 $ mkdir work
 $ cd work
 $ cmake ..
 $ make
 $ sudo make install
 AffineComp                       FlipComp               RockPaperScissorsComp
 Affine.so                        Flip.so                RockPaperScissors.so
 BackGroundSubtractionSimpleComp  HistogramComp          RotateComp
 BackGroundSubtractionSimple.so   Histogram.so           Rotate.so
 BinarizationComp                 HoughComp              ScaleComp
 Binarization.so                  Hough.so               Scale.so
 CameraViewerComp                 ImageCalibrationComp   SepiaComp
 CameraViewer.so                  ImageCalibration.so    Sepia.so
 ChromakeyComp                    ImageSubstractionComp  SubStractCaptureImageComp
 Chromakey.so                     ImageSubstraction.so   SubStractCaptureImage.so
 DilationErosionComp              ObjectTrackingComp     TemplateComp
 DilationErosion.so               ObjectTracking.so      Template.so
 EdgeComp                         OpenCVCameraComp       TranslateComp
 Edge.so                          OpenCVCamera.so        Translate.so
 FindcontourComp                  PerspectiveComp
 Findcontour.so                   Perspective.so

Installing Code::Blocks

Code::Blocks is an integrated development environment (IDE) that supports C/C++.

It can be installed using the following command.

 $ sudo apt-get install codeblocks

If you want to install the latest version, enter the following commands.

 $ sudo add-apt-repository ppa:damien-moore/codeblocks-stable
 $ sudo apt-get update
 $ sudo apt-get install codeblocks

Generating the Flip Component Template

The Flip component template is generated using RTCBuilder.

Starting RTCBuilder

In Eclipse, the folder used for development work is called the workspace, and in principle all generated files are stored under this folder.

The workspace can be created anywhere that is accessible, but this tutorial assumes the following workspace.

• /home//workspace

First, start Eclipse.

Move to the directory where OpenRTP was extracted and enter the following command.

 $ ./openrtp

You will first be prompted to specify the workspace location. Specify the workspace shown above.

A Welcome screen similar to the one below will appear.
Since the Welcome screen is not needed, click the [×] button in the upper-left corner to close it.

Initial Eclipse Startup Screen

Click the [Open Perspective] button in the upper-right corner.

Switching Perspectives

Select RTC Builder and click the [OK] button to start RTCBuilder.

Selecting a Perspective

Creating a New Project

To create the Flip component, you must create a new project in RTCBuilder.

Click the [Open New RTCBuilder Editor] icon in the upper-left corner.

Creating an RTC Builder Project

Enter the project name to create (Flip in this example) in the “Project Name” field and click [Finish].

A project with the specified name is generated and added to the Package Explorer.

Within the generated project, an RTC Profile XML file (RTC.xml) with default values is automatically generated.

Starting the RTC Profile Editor

When RTC.xml is generated, the RTCBuilder editor associated with this project should open automatically.

If it does not start, double-click RTC.xml in the Package Explorer.

Entering Profile Information and Generating Code

First, select the Basic tab on the far left and enter the basic information.

In addition to the Flip component specifications (name) defined earlier, enter information such as the description and version.

Items with red labels are required.

The remaining items may be left at their default values.

Module Name	Flip
Module Description	Flip image component
Version	1.0.0
Vendor Name	AIST
Module Category	Category
Component Type	STATIC
Activity Type	PERIODIC
Component Kind	DataFlowComponent
Maximum Number of Instances	1
Execution Type	PeriodicExecutionContext
Execution Period	0.0 (The figure below shows 1.0, but set this to 0.0.)

Entering Basic Information

Next, select the Activity tab and specify the action callbacks to use.

The Flip component uses the following callbacks:

• onActivated()
• onDeactivated()
• onExecute()

As shown in the figure below, first click ① onActivated, then select [ON] using radio button ②.

Perform the same operation for onDeactivated and onExecute.

Selecting Activity Callbacks

Next, select the Data Ports tab and enter the data port information.

Enter the values below according to the specifications defined earlier.

The variable names and display positions are optional and may be left unchanged.

InPort Profile

• Port Name: originalImage
• Data Type: RTC::CameraImage
• Variable Name: originalImage
• Display Position: left

OutPort Profile

• Port Name: flippedImage
• Data Type: RTC::CameraImage
• Variable Name: flippedImage
• Display Position: right

Entering Data Port Information

Next, select the Configuration tab and enter the configuration information based on the specifications defined earlier.

The constraint conditions and Widget settings are used when displaying component configuration parameters in RTSystemEditor, allowing values to be changed through GUI controls such as sliders, spin buttons, and radio buttons.

Since flipMode was defined earlier to take only three possible values (-1, 0, and 1), a radio button will be used.

flipMode

• Name: flipMode
• Data Type: int
• Default Value: 1
• Variable Name: flipMode
• Constraint: (-1, 0, 1) * (-1: horizontal and vertical flip, 0: vertical flip, 1: horizontal flip)
• Widget: radio

Entering Configuration Information

Next, select the Language / Environment tab and specify the programming language.

Here, select C++.

The language/environment fields have no default values. If you forget to specify the language, an error will occur during code generation, so be sure to specify it.

For C++, the default build system is CMake. If you want RTCBuilder to directly generate old-style Code::Blocks projects, check [Use old build environment].

Selecting the Programming Language

Finally, click the [Generate Code] button on the Basic tab to generate the component template.

Generating the Template (Generate)

&color(Red){ The generated code files are created in the workspace folder specified when Eclipse was started. You can check the current workspace from [File] → [Switch Workspace]. };

Generating Files Required for Building with CMake

The code generated by RTC Builder includes a CMakeLists.txt file for generating the various files required for building with CMake.

By using CMake, project files, solution files for Visual Studio, Makefiles, and other build-related files can be generated automatically from CMakeLists.txt.

Editing CMakeLists.txt

Open and edit src/CMakeLists.txt using gedit or another editor.

Alternatively, in Eclipse, you can edit it by double-clicking src/CMakeLists.txt in the Package Explorer on the left side of the screen, or by dragging and dropping it into the editor.

Editing CMakeLists.txt

Since this component uses OpenCV, it is necessary to specify the OpenCV header include paths, libraries, and library search paths.

Fortunately, OpenCV supports CMake, so simply adding/modifying the following two lines enables linking and use of the OpenCV libraries.

• Modify src/CMakeLists.txt
  • Double-click src/CMakeLists.txt in Eclipse’s Package Explorer.
• Add find_package(OpenCV REQUIRED)
• Add ${OpenCV_LIBS} to the first target_link_libraries
  • There are two target_link_libraries entries. The upper one specifies the DLL library, and the lower one specifies the executable library.

 set(comp_srcs Flip.cpp )
 set(standalone_srcs FlipComp.cpp)
 
 find_package(OpenCV REQUIRED) ← Add this line
   ：omitted
 add_dependencies(${PROJECT_NAME} ALL_IDL_TGT)
 target_link_libraries(${PROJECT_NAME} ${OPENRTM_LIBRARIES} ${OpenCV_LIBS}) ← Add OpenCV_LIBS
   ：omitted
 add_executable(${PROJECT_NAME}Comp ${standalone_srcs}
   ${comp_srcs} ${comp_headers} ${ALL_IDL_SRCS})
 target_link_libraries(${PROJECT_NAME}Comp ${OPENRTM_LIBRARIES} ${OpenCV_LIBS}) ← Add OpenCV_LIBS

Using CMake (cmake-gui)

Use CMake to configure the build environment.

First, start CMake (cmake-gui).

 $ cmake-gui

Starting CMake GUI and Specifying Directories

At the top of the screen there are text boxes similar to the following. Specify the location of the source code (where CMakeLists.txt is located) and the build directory.

• Where is the source code
• Where to build the binaries

The source code location is the directory where the Flip component source code was generated and where CMakeLists.txt exists.

By default, this is:

<workspace directory>/Flip

The build directory is where project files, object files, and binaries used for building are stored.

The location can be anywhere, but in this case it is recommended to specify a subdirectory of Flip with an easy-to-understand name, such as:

<workspace directory>/Flip/build

Where is the source code	/home/<username>/workspace/Flip
Where to build the binaries	/home/<username>/workspace/Flip/build

After specifying the directories, click the Configure button at the bottom.

A dialog like the one shown below will appear, where you specify the type of project to generate.

This time, select CodeBlocks - Unix Makefiles.

If you are not using Code::Blocks, select Unix Makefiles.

Selecting the Type of Project to Generate

If you do not use cmake-gui, you can generate the files using the following commands.

 $ mkdir build
 $ cd build
 $ cmake .. -G "CodeBlocks - Unix Makefiles"

When you click [Finish] in the dialog, Configure begins.

If there are no problems, “Configuring done” will be displayed in the log window at the bottom. Then click the [Generate] button.

When “Generating done” is displayed, generation of the project file (.cbp), Makefile, and related files is complete.

Note that CMake generates cache files during the Configure stage. Therefore, if you change settings or modify the environment while troubleshooting, delete the cache via [File] > [Delete Cache] and rerun Configure from the beginning.