image processing component development (windows 10, openrtm-aist-2.0.0, openrtp-2.0.0, cmake-3.19.8, vs2019)

✏️ Edit this page on GitHub

#contents

Introduction

This case study introduces how to convert a simple image processing function into a component.

Using an existing camera component and image viewer component, we will create a component that flips images horizontally (or vertically), and build a system that displays the flipped camera image.

Although image flipping is easy to implement, we will use the OpenCV library to simplify the implementation and create a highly reusable RT Component.

What is OpenCV?

OpenCV (Open Source Computer Vision Library) is an open-source computer vision library originally developed by Intel and currently maintained by Itseez.

Excerpted from Wikipedia.

RT Component to be Created

  • Flip Component: An RT Component that flips images using the cv::flip() function provided by the OpenCV library.

Converting the cv::flip Function into an RT Component

We will create an RT Component that receives an image, flips it horizontally or vertically, and outputs the processed image using the OpenCV cv::flip() function.

The development and execution environment assumes Visual Studio running on Windows.

The target OpenRTM-aist version is 1.1.2.

The development process is roughly as follows:

  • Confirm the runtime and development environment
  • Understand OpenCV and the cv::flip function
  • Define the component specifications
  • Generate source code templates using RTCBuilder
  • Implement activity processing
  • Verify component operation

About the cv::flip Function

The cv::flip function flips image data stored in OpenCV’s standard cv::Mat format around a vertical axis (horizontal flip), horizontal axis (vertical flip), or both axes.

The function prototype and argument descriptions are as follows.

void flip(const Mat& src, Mat& dst, int flipMode)
src Input array
dst Output array
flipMode Flip method:
flipMode = 0: Flip around the X-axis (vertical flip)
flipMode > 0: Flip around the Y-axis (horizontal flip)
flipMode < 0: Flip around both axes

Component Specifications &aname(flip_info);

We will call the component being created the Flip Component.

This component has an input port (InPort) for image data and an output port (OutPort) for outputting the flipped image.

The port names are:

  • InPort: originalImage
  • OutPort: flippedImage

OpenRTM-aist includes sample vision components that use OpenCV.

These components use the following CameraImage type for image input/output.

struct CameraImage
{
    /// Time stamp.
    Time tm;
    /// Image pixel width.
    unsigned short width;
    /// Image pixel height.
    unsigned short height;
    /// Bits per pixel.
    unsigned short bpp;
    /// Image format (e.g. bitmap, jpeg, etc.).
    string format;
    /// Scale factor for images.
    double fDiv;
    /// Raw pixel data.
    sequence<octet> pixels;
};

To maintain compatibility with these sample components, this Flip Component will also use the CameraImage type for both InPort and OutPort.

The CameraImage type is defined in InterfaceDataTypes.idl.

In C++, it becomes available by including InterfaceDataTypesSkel.h.

The image can be flipped in three ways:

  • Vertical
  • Horizontal
  • Both vertical and horizontal

To allow runtime selection, the RT Component configuration mechanism will be used.

The parameter name will be flipMode.

The type is int, matching the cv::flip specification:

  • 0 : Vertical flip
  • 1 : Horizontal flip
  • -1 : Both axes

The image processing behavior for each value is shown below.

Image Flip Patterns for Flip Component flipMode Settings

The specifications of the Flip Component are summarized below.

Component Name Flip
InPort
Port Name originalImage
Type RTC::CameraImage
Description Input image
OutPort
Port Name flippedImage
Type RTC::CameraImage
Description Flipped image
Configuration
Parameter Name flipMode
Type int
Default Value 0
Constraint (0, -1, 1)
Widget radio
Description Flip Mode
Vertical Flip: 0
Horizontal Flip: 1
Vertical + Horizontal Flip: -1

Runtime and Development Environment

Let’s first verify the runtime and development environment.

Since OpenRTM-aist 1.1, CMake has been used to build components.

Generating the Flip Component Template

The Flip component template is generated using RTCBuilder.

Starting RTCBuilder

In Eclipse, the folder used for development work is called a “workspace”, and in principle all generated files are stored under this folder.

The workspace can be located anywhere that is accessible. In this tutorial, the following workspace is assumed:

  • C:\workspace

First, launch Eclipse.

In Windows 10, double-click the following shortcut on the desktop.

You will first be prompted for the workspace location. Specify the workspace above and click [OK].

The following Welcome page will appear.

The Welcome page is not needed, so click the [×] button in the upper-left corner to close it.


Initial Eclipse Startup Screen

Click the [Open Perspective] button in the upper-right corner.

Switching Perspectives

Select “RTC Builder” and click the [OK] button.

RTCBuilder will start.

Selecting a Perspective

Creating a New Project

To create the Flip component, you need to create a new project in RTCBuilder.

Click the [Open New RTCBuilder Editor] icon in the upper-left corner.

Creating an RTC Builder Project

Enter the project name to create (here, Flip) in the “Project Name” field and click [Finish].

A project with the specified name will be generated and added to the Package Explorer.

An RTC profile XML file (RTC.xml) with default values will automatically be generated in the project.

Starting the RTC Profile Editor

When RTC.xml is generated, the RTCBuilder editor associated with the project should automatically open.

Entering Profile Information and Generating Code

First, select the leftmost “Basic” tab and enter the basic information.

In addition to the Flip component name defined earlier, enter a description, version, and other information.

Items shown in red are required fields. All others can remain at their default values.

  • Module Name: Flip
  • Module Description: Optional (Flip image component)
  • Version: Optional (1.0.0)
  • Vendor Name: Optional
  • Module Category: Optional (ImageProcessing)
  • Component Type: STATIC
  • Activity Type: PERIODIC
  • Component Kind: DataFlowComponent
  • Maximum Number of Instances: 1
  • Execution Type: PeriodicExecutionContext
  • Execution Rate: 1000.0


Entering Basic Information


Next, select the “Activity” tab and specify the action callbacks to use.

The Flip component uses the onActivated(), onDeactivated(), and onExecute() callbacks.

As shown below, click on onActivated and then select [ON] using the radio button.

Perform the same operation for onDeactivated and onExecute.


Selecting Activity Callbacks


Next, select the “Data Port” tab and enter the data port information.

Based on the specifications defined earlier, enter the following information.

Variable names and display positions are optional and do not need to be changed.


  • InPort Profile:
    • Port Name: originalImage
    • Data Type: RTC::CameraImage
    • Variable Name: originalImage
    • Display Position: left


  • OutPort Profile:
    • Port Name: flippedImage
    • Data Type: RTC::CameraImage
    • Variable Name: flippedImage
    • Display Position: right


Entering Data Port Information


Next, select the “Configuration” tab and enter the configuration information according to the specifications defined earlier.

The Constraint and Widget settings are used by RTSystemEditor to determine how configuration parameters are displayed and edited in the GUI (for example, sliders, spin buttons, or radio buttons).

In this case, flipMode can only take the values -1, 0, and 1, so we will use radio buttons.


  • flipMode
    • Name: flipMode
    • Data Type: int
    • Default Value: 0
    • Constraint: (0, -1, 1)
      • ※ (-1: vertical & horizontal flip, 0: vertical flip, 1: horizontal flip)
    • Widget: radio


Entering Configuration Information


Next, we will select the programming language and generate the code.

The tool specifications differ between OpenRTM-aist 1.2 and 2.0.

First, the procedure for version 1.2.2 and earlier is explained.

Select the “Language / Environment” tab and choose the programming language.

Here, select C++.

Since no default language is set, forgetting to specify the language will result in an error during code generation.

Be sure to select a language.

Selecting the Programming Language


Finally, click the “Generate Code” button on the “Basic” tab to generate the component template.


Generating the Template (Generate)


Next, the procedure for version 2.0 and later is explained.

Select the “Basic” tab and scroll down to the “Language” section.

Select C++.

Finally, click the Generate Code button to generate the component template.

After code generation is complete, right-click the Flip project and select:

“Show In” → “System Explorer”

to open the workspace folder in Windows Explorer.

Verify that the necessary files have been generated in the Flip folder.

Generating Files Required for Building with CMake

The code generated by RTC Builder includes a CMakeLists.txt file for generating the various files required for building with CMake.

By using CMake, Visual Studio project files, solution files, Makefiles, and so on can be automatically generated from CMakeLists.txt.

Editing CMakeLists.txt

Open and edit src/CMakeLists.txt in the Flip folder using Notepad or another editor.

Editing CMakeLists.txt

This component uses OpenCV, so it is necessary to configure the OpenCV header include paths, libraries, and library search paths.

OpenCV supports CMake, so OpenCV libraries can be linked and used by adding and modifying only the following two lines.

  • Modify src/CMakeLists.txt
    • Double-click src/CMakeLists.txt in Explorer and open it with Notepad 1. Add find_package(OpenCV REQUIRED) 1. Add ${OpenCV_LIBS} to the first target_link_libraries
    • There are two target_link_libraries entries. The upper one specifies the library for the DLL, and the lower one specifies the library for the executable file.
 set(comp_srcs Flip.cpp )
 set(standalone_srcs FlipComp.cpp)
 
 find_package(OpenCV REQUIRED) # <- Add this line
   ：Omitted
 add_dependencies(${PROJECT_NAME} ALL_IDL_TGT)
 target_link_libraries(${PROJECT_NAME} ${OPENRTM_LIBRARIES} ${OpenCV_LIBS}) # <- Add OepnCV_LIBS
   ：Omitted
 add_executable(${PROJECT_NAME}Comp ${standalone_srcs}
   ${comp_srcs} ${comp_headers} ${ALL_IDL_SRCS})
 add_dependencies(${PROJECT_NAME}Comp ALL_IDL_TGT)
 target_link_libraries(${PROJECT_NAME}Comp ${OPENRTM_LIBRARIES} ${OpenCV_LIBS})  # <- Add OepnCV_LIBS

Operating CMake (cmake-gui)

Use CMake to configure the build environment.

First, start CMake (cmake-gui). In Windows 10, enter “CMake” in the search box at the lower-left of the screen.

Starting CMake GUI and Specifying Directories

There are text boxes at the top of the screen as shown below. Specify the source code location (where CMakeLists.txt is located) and the build directory.

  • Where is the source code
  • Where to build the binaries

The source code location is the directory where the Flip component source code was generated and where CMakeLists.txt exists. By default, this is <workspace directory>/Flip.

If you drag and drop CMakeLists.txt in the Flip folder into cmake-gui, the path to the Flip folder will be entered automatically.

The build directory is the location where project files, object files, and binaries required for building are stored.

The location can be arbitrary, but in this case it is recommended to specify a subdirectory of Flip with an easy-to-understand name, such as <workspace directory>/Flip/build.

**Where is the source code** C:\workspace\Flip
**Where to build the binaries** C:\workspace\Flip\build

After specifying the directories, click the Configure button at the bottom.

If the Create Directory screen appears, click YES.

Then a dialog like the one below will appear. Specify the type of project you want to generate.

Here, select Visual Studio 16 2019.

If you are using VS2017 or VS2022, select the corresponding version instead. If Visual Studio 2022 does not appear in the list, your cmake-gui version is old, so install the latest version.

Specifying the Type of Project to Generate

Click [Finish] in the dialog to start Configure.

If there are no problems, “Configuring done” will appear in the log window at the bottom. Then click the [Generate] button.

When “Generating done” is displayed, output of the project files, solution files, and related files is complete.

Click the “Open Project” button to start Visual Studio 2019 and open Flip.sln in the build directory specified earlier.

CMake generates cache files during the Configure step. If you change settings or modify the environment due to trouble or other reasons, delete the cache from [File] > [Delete Cache] and restart from Configure.

Editing Headers and Source Files

Edit the header file (include/Flip/Flip.h) and source code (src/Flip.cpp).

Click Flip.h and Flip.cpp from Visual Studio’s Solution Explorer to open the editing screen.

Implementing Activity Processing

In the Flip component, the image received from the InPort is stored in an image buffer. The stored image is then converted using OpenCV’s cv::flip() function.

After that, the converted image is sent from the OutPort.


The processing performed in onActivated(), onExecute(), and onDeactivated() is shown below.

Overview of Activity Processing


The processing performed in onExecute() is shown below.


Processing Performed in onExecute()


Editing the Header File (Flip.h)

Since OpenCV libraries are used, include the OpenCV header files.

 #include <rtm/DataInPort.h>
 #include <rtm/DataOutPort.h>
 
 #include <opencv2/opencv.hpp> // Add this line
 
 
 // <rtc-template block="component_description">

Add member variables for storing the flipped image.

 private:
   // <rtc-template block="private_attribute">
  
   // </rtc-template>
  
   // <rtc-template block="private_operation">
  
   // </rtc-template>
  cv::Mat m_imageBuff; // Add this line
  cv::Mat m_flipImageBuff; // Add this line

Editing the Source File (Flip.cpp)

Implement onActivated(), onDeactivated(), and onExecute() as shown below.

 RTC::ReturnCode_t Flip::onActivated(RTC::UniqueId ec_id)
 {
 
 	   // Set the OutPort image size to 0
 	   m_flippedImage.width = 0;
 	   m_flippedImage.height = 0;
 
 	   return RTC::RTC_OK;
 }

 RTC::ReturnCode_t Flip::onDeactivated(RTC::UniqueId ec_id)
 {
 	   if (!m_imageBuff.empty())
 	   {
 		   // Release image memory
 		   m_imageBuff.release();
 		   m_flipImageBuff.release();
 	   }
 
 	   return RTC::RTC_OK;
 }

 RTC::ReturnCode_t Flip::onExecute(RTC::UniqueId ec_id)
 {
 	   // Check for new data
 	   if (m_originalImageIn.isNew()) {
 		   // Read InPort data
 		   m_originalImageIn.read();
 
 		   // Handle image size changes and allocate image memory
 		   if (m_originalImage.width != m_flippedImage.width || m_originalImage.height != m_flippedImage.height)
 		   {
 			   m_flippedImage.width = m_originalImage.width;
 			   m_flippedImage.height = m_originalImage.height;
 
 			   m_imageBuff.create(cv::Size(m_originalImage.width, m_originalImage.height), CV_8UC3);
 			   m_flipImageBuff.create(cv::Size(m_originalImage.width, m_originalImage.height), CV_8UC3);
 
 			
 		   }
 
 		   // Copy InPort image data to m_imageBuff
 		   memcpy(m_imageBuff.data, (void *)&(m_originalImage.pixels[0]), m_originalImage.pixels.length());
 
 		   // Flip the image received from InPort.
 		   // m_flipMode: 0 = around X-axis, 1 = around Y-axis, -1 = around both axes
 		   cv::flip(m_imageBuff, m_flipImageBuff, m_flipMode);
 
 		   // Get image data size
 		   int len = m_flipImageBuff.channels() * m_flipImageBuff.cols * m_flipImageBuff.rows;
 		   m_flippedImage.pixels.length(len);
 
 		   // Copy the flipped image data to OutPort
 		   memcpy((void *)&(m_flippedImage.pixels[0]), m_flipImageBuff.data, len);
 
 		   // Output the flipped image data from OutPort
 		   m_flippedImageOut.write();
 	   }
 
      return RTC::RTC_OK;
 }

Building with Visual Studio

Running the Build

Select Build → Build Solution from the Visual Studio menu to build the project.


Running the Build


Verifying the Operation of the Flip Component

In this section, we will verify the operation by connecting the camera component (OpenCVCameraComp) and viewer component (CameraViewerComp) that have been included with OpenRTM-aist since version 1.1.

First, start RTSystemEditor. From the OpenRTP perspective selection screen, open RTSystemEditor.

Starting the Name Service

Start the Name Service used to register component references.


Click the Start Name Service button at the top of the Name Service View in RTSystemEditor.

Starting the Flip Component

Start the Flip component.

Run the FlipComp.exe file located in the build/src/Debug (or build/src/Release) folder.

Starting the Camera Component and Viewer Component

Start OpenCVCameraComp, which outputs images captured from a USB camera through an OutPort, and CameraViewerComp, which displays images received through an InPort.

These two components can be started using the following procedure.

  • For Windows 10, type C++_OpenCV-Examples into the search box at the lower-left corner of the screen.
  • When Explorer opens, click and run both OpenCVCameraComp.bat and CameraViewerComp.bat.

Connecting the Components

When the components are started, the Flip, CameraViewer, and OpenCVCamera components will appear in the Name Service View as shown below.

Click the Open New System Editor button (the ON button) in the upper-left corner to open the editor, then drag and drop the components into the editor.

Connect OpenCVCameraComp, Flip, and CameraViewerComp in RTSystemEditor as shown below.

Connecting Components

Port connections are created by dragging from one port and dropping onto the other port.

Activating the Components

Click the All Activate icon located at the top of RTSystemEditor to activate all components.

If activation succeeds, the components will be displayed in light green as shown below.


Activating Components


Verifying Operation

As shown below, configuration parameters can be modified in the Configuration View.

Click the Flip component in the editor and press the Edit button.

Change the Flip component configuration parameter flipMode to values such as 0 or -1, and verify that the image is flipped correctly.


Changing Configuration Parameters


To terminate the components, click the All Exit button. All components displayed in the editor will be shut down.