creating an image processing component (openrtm-aist-1.1, cmake, vc2010)
-
Converting the cvFlip Function into an RT-Component
- About the cvFlip Function
- Component Specifications
- Runtime Environment and Development Environment
- Generating the Flip Component Skeleton
- Editing the Header and Source Files
- Generating Files Required for CMake Build
- Building with Visual C++
- Verifying Operation of the Flip Component
- Connecting the Components
#contents
Introduction
In this case study, we introduce how to componentize a simple image-processing function.
Using an existing camera component and image display component, we create a component that performs image flipping (horizontal or vertical) and build a system that displays images captured by the camera after flipping them.
Although image flipping can be implemented easily, in this tutorial we use the OpenCV library to create a highly reusable RT-Component with minimal effort.
What is OpenCV?
OpenCV (Open Source Computer Vision Library) is an open-source computer vision library originally developed by Intel and currently maintained and distributed by Willow Garage.
Excerpt from Wikipedia.
RT-Component to Be Created
- Flip Component: An RT-Component that performs image flipping using the
cvFlip()function, one of the many image-processing functions provided by the OpenCV library.
Converting the cvFlip Function into an RT-Component
We will create an RT-Component that flips input images horizontally or vertically and outputs the result using the OpenCV library’s cvFlip function.
The development and execution environment assumes Visual C++ on Windows. The target version of OpenRTM-aist is 1.1.0.
The overall procedure is as follows:
- Confirm the runtime and development environments
- Review OpenCV and the
cvFlipfunction - Define the component specifications
- Generate a source-code skeleton using RTCBuilder
- Implement the activity processing
- Verify component operation
About the cvFlip Function
The cvFlip function flips image data of type IplImage, the standard image format used by OpenCV, around the vertical axis (horizontal flip), horizontal axis (vertical flip), or both axes.
The function prototype and argument meanings are as follows.
void cvFlip(IplImage* src, IplImage* dst=NULL, int flipMode=0);
#define cvMirror cvFlip
src Input array
dst Output array. If dst=NULL, src is overwritten.
flipMode Specifies the flipping method:
flipMode = 0: Flip around the X-axis (vertical flip)
flipMode > 0: Flip around the Y-axis (horizontal flip)
flipMode < 0: Flip around both axes (horizontal and vertical flip)
Component Specifications
The component we are about to create will be called the Flip component.
This component has an image-data input port (InPort) and an output port (OutPort) that outputs the flipped image.
The port names are:
- Input Port (InPort): originalImage
- Output Port (OutPort): flippedImage
OpenRTM-aist includes sample vision-related components that use OpenCV.
These components use the following CameraImage type for image input/output through data ports.
struct CameraImage
{
/// Time stamp.
Time tm;
/// Image pixel width.
unsigned short width;
/// Image pixel height.
unsigned short height;
/// Bits per pixel.
unsigned short bpp;
/// Image format (e.g. bitmap, jpeg, etc.).
string format;
/// Scale factor for images, such as disparity maps,
/// where the integer pixel value should be divided
/// by this factor to get the real pixel value.
double fDiv;
/// Raw pixel data.
sequence<octet> pixels;
};
To allow data exchange with these sample components, the Flip component also uses the CameraImage type for both its InPort and OutPort.
The CameraImage type is defined in InterfaceDataTypes.idl and can be used in C++ by including InterfaceDataTypesSkel.h.
The image can be flipped in three ways:
- Horizontal flip
- Vertical flip
- Horizontal and vertical flip
To allow the direction to be specified at runtime, we use the RT-Component configuration feature.
The parameter name is flipMode.
Following the cvFlip specification, flipMode is of type int, with the following values assigned:
- Vertical flip:
0 - Horizontal flip:
1 - Horizontal and vertical flip:
-1
Figure 1 shows the image-processing behavior for each value of flipMode.
The specifications of the Flip component are summarized below.
| Component Name | Flip |
|---|---|
| InPort | |
| Port Name | originalImage |
| Type | CameraImage |
| Description | Input image |
| OutPort | |
| Port Name | flippedImage |
| Type | CameraImage |
| Description | Flipped image |
| Configuration | |
| Parameter Name | flipMode |
| Type | int |
| Description | Flip Mode Vertical flip: 0 Horizontal flip: 1 Horizontal and vertical flip: -1 |
Runtime Environment and Development Environment
Let’s first confirm the runtime and development environments.
- OS: Windows XP SP3 (Vista and Windows 7 are also supported)
- RTSystemEditor 1.1
- RTCBuilder 1.1
- Eclipse3.4.2+RTSE+RTCB(1.1.0-RC2) Complete Windows Package(No corresponding page available.)
- Doxygen Required for documentation generation
- Archive Extraction Tool (Lhaplus)
Since OpenRTM-aist 1.1, CMake has been used for component builds.
RTCBuilder can also automatically generate component manuals by passing entered documentation to Doxygen.
Therefore, Doxygen is required during the CMake configuration process and must be installed in advance.
Generating the Flip Component Skeleton
The Flip component skeleton is generated using RTCBuilder.
Starting RTCBuilder
When Eclipse is started with a new workspace specified, the Welcome screen shown below appears.
Since the Welcome screen is not needed, close it by clicking the [×] button in the upper-left corner.
Click the [Open Perspective] button in the upper-right corner and select “Other…” from the pull-down menu.
Select “RTC Builder” and click the [OK] button.
RTCBuilder starts.
Creating a New Project
To create the Flip component, you must first create a new project in RTCBuilder.
Select:
[File] > [New] > [Project]
from the menu.
In the displayed “New Project” dialog, select:
[Other] > [RTC Builder]
and click [Next].
Enter the project name (Flip in this example) in the “Project Name” field and click [Finish].
The specified project is generated and added to the Package Explorer.
An RTC profile XML file (RTC.xml) with default values is automatically generated inside the project.
Launching the RTC Profile Editor
When RTC.xml is generated, the RTCBuilder editor associated with the project workspace should open automatically.
If it does not open, click the [Open New RtcBuilder Editor] button on the toolbar or select:
[File] > [Open New Builder Editor]
from the menu.
Entering Profile Information and Generating Code
First, select the leftmost “Basic” tab and enter the basic information.
In addition to the Flip component name defined earlier, enter information such as the summary and version.
Fields displayed in red are required. Other fields may be left at their default values.
- Module Name: Flip
- Module Description: Flip image component
- Version: 1.0.0
- Vendor: AIST
- Module Category: Category
- Component Type: STATIC
- Activity Type: PERIODIC
- Component Kind: DataFlowComponent
- Maximum Number of Instances: 1
- Execution Type: PeriodicExecutionContext
- Execution Period: 0.0 (Figure 13 shows 1.0, but please set it to 0.0.)
- Output Project: Flip
Next, select the “Activity” tab and specify the action callbacks to be used.
The Flip component uses the following callbacks:
- onActivated()
- onDeactivated()
- onExecute()
As shown in Figure 14, click onActivated in section ① and then enable the [ON] radio button in section ②.
Repeat the same procedure for onDeactivated and onExecute.
Next, select the “Data Ports” tab and enter the data port information.
Based on the specifications defined earlier, enter the following values.
Variable names and display positions are optional and may remain unchanged.
- InPort Profile:
- Port Name: originalImage
- Data Type: CameraImage
- Variable Name: originalImage
- Position: left
- OutPort Profile:
- Port Name: flippedImage
- Data Type: CameraImage
- Variable Name: flippedImage
- Position: right
Next, select the “Configuration” tab and enter the Configuration information according to the specifications.
Constraints and Widgets determine how configuration parameters are displayed in RTSystemEditor, allowing values to be modified using GUI controls such as sliders, spin buttons, and radio buttons.
Since flipMode is restricted to the three values defined earlier (-1, 0, and 1), we use radio buttons.
- flipMode
- Name: flipMode
- Data Type: int
- Default Value: 1
- Variable Name: flipMode
- Constraint: (-1, 0, 1) * (-1: Horizontal and vertical flip, 0: Vertical flip, 1: Horizontal flip)
- Widget: radio
Next, select the “Language / Environment” tab and choose the programming language.
In this example, select C++ as the programming language.
Note that no default value is set for the Language / Environment field. If you forget to specify it, code generation will fail, so be sure to select a language.
For C++, the default build environment uses CMake. However, if you want RTCBuilder to generate legacy Visual C++ project or solution files directly, check Use old build environment.
Finally, click the [Generate Code] button in the “Basic” tab to generate the component skeleton.
&color(Red){* The generated source files are created inside the workspace folder specified when Eclipse was started. You can check the current workspace via [File] > [Switch Workspace].};
Preliminary Build
At this point, the Flip component source code has been generated.
Since the processing logic has not yet been implemented, nothing will be output even if image data is supplied to the InPort. However, the generated source code can still be compiled and executed.
- For components with service ports and providers, some implementations may fail to build unless the service implementation is completed.
First, configure the build environment using CMake.
On Linux, run the following commands in the directory where the Flip component source code was generated:
$ cmake .
$ make
This should complete both configuration and build.
For Windows, use the GUI version of CMake.
Launch CMake (cmake-gui) from the Start menu.
At the top of the window, you will find the following text boxes.
Specify the source-code directory (where CMakeLists.txt exists) and the build directory.
- Where is the source code
- Where to build the binaries
The source-code directory is the location where the Flip component source code was generated and where CMakeLists.txt exists.
By default, this will be:
<workspace directory>/Flip
The build directory is where project files, object files, and binaries generated during the build process are stored.
Although any location can be used, it is recommended to specify an easy-to-understand subdirectory such as:
<workspace directory>/Flip/build
After specifying the directories, click the [Configure] button.
A dialog similar to Figure 20 will appear.
Specify the type of project to generate.
In this example, select Visual Studio 10.
If you are using VS8 or VS9, select the corresponding version.
Click [Finish] to start configuration.
If successful, the log window at the bottom will display:
Configuring done
Then click the [Generate] button.
When:
Generating done
is displayed, generation of project files and solution files is complete.
Since CMake creates cache files during configuration, if you change settings or modify the environment due to troubleshooting, delete the cache via:
[File] > [Delete Cache]
and rerun the configuration process.
Next, double-click Flip.sln inside the build directory specified earlier to launch Visual Studio 2010.
After Visual Studio starts, right-click ALL_BUILD in the Solution Explorer and select Build.
If there are no problems, the build should complete successfully.
Editing the Header and Source Files
Implementing the Activity Processing
In the Flip component, images received through the InPort are stored in an image buffer.
The stored image is then processed using OpenCV’s cvFlip() function.
Finally, the converted image is sent through the OutPort.
The processing performed by onActivated(), onExecute(), and onDeactivated() is shown in Figure 22.
The processing performed in onExecute() is shown in Figure 23.
Editing the Header File (Flip.h)
To use the OpenCV library, include the OpenCV header files.
// Include files for OpenCV
#include<cv.h>
#include<cxcore.h>
#include<highgui.h>
This cvFlip component performs image-buffer allocation, image flipping, and image-buffer release.
These processes are handled by the callback functions onActivated(), onDeactivated(), and onExecute() respectively.
/***
*
* The activated action (Active state entry action)
* former rtc_active_entry()
*
* @param ec_id target ExecutionContext Id
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onActivated(RTC::UniqueId ec_id);
/***
*
* The deactivated action (Active state exit action)
* former rtc_active_exit()
*
* @param ec_id target ExecutionContext Id
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onDeactivated(RTC::UniqueId ec_id);
/***
*
* The execution action that is invoked periodically
* former rtc_active_do()
*
* @param ec_id target ExecutionContext Id
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onExecute(RTC::UniqueId ec_id);
Add member variables for storing flipped images.
IplImage* m_imageBuff;
IplImage* m_flipImageBuff;
Editing the Source File (Flip.cpp)
Implement onActivated(), onDeactivated(), and onExecute() as shown below.
RTC::ReturnCode_t Flip::onActivated(RTC::UniqueId ec_id)
{
// Initialize image memory
m_imageBuff = NULL;
m_flipImageBuff = NULL;
// Initialize OutPort image size
m_flippedImage.width = 0;
m_flippedImage.height = 0;
return RTC::RTC_OK;
}
RTC::ReturnCode_t Flip::onDeactivated(RTC::UniqueId ec_id)
{
if(m_imageBuff != NULL)
{
// Release image memory
cvReleaseImage(&m_imageBuff);
cvReleaseImage(&m_flipImageBuff);
}
return RTC::RTC_OK;
}
RTC::ReturnCode_t Flip::onExecute(RTC::UniqueId ec_id)
{
// Check for new data
if (m_originalImageIn.isNew()) {
// Read InPort data
m_originalImageIn.read();
// Process InPort and OutPort image size and allocate image memory
if( m_originalImage.width != m_flippedImage.width || m_originalImage.height != m_flippedImage.height)
{
m_flippedImage.width = m_originalImage.width;
m_flippedImage.height = m_originalImage.height;
// If the InPort image size has changed
if(m_imageBuff != NULL)
{
cvReleaseImage(&m_imageBuff);
cvReleaseImage(&m_flipImageBuff);
}
// Allocate image memory
m_imageBuff = cvCreateImage(cvSize(m_originalImage.width, m_originalImage.height), IPL_DEPTH_8U, 3);
m_flipImageBuff = cvCreateImage(cvSize(m_originalImage.width, m_originalImage.height), IPL_DEPTH_8U, 3);
}
// Copy InPort image data to IplImage imageData
memcpy(m_imageBuff->imageData,(void *)&(m_originalImage.pixels[0]),m_originalImage.pixels.length());
// Flip image data from InPort. m_flipMode 0: X-axis, 1: Y-axis, -1: both axes
cvFlip(m_imageBuff, m_flipImageBuff, m_flipMode);
// Obtain image data size
int len = m_flipImageBuff->nChannels * m_flipImageBuff->width * m_flipImageBuff->height;
m_flippedImage.pixels.length(len);
// Copy flipped image data to OutPort
memcpy((void *)&(m_flippedImage.pixels[0]),m_flipImageBuff->imageData,len);
// Output flipped image data through OutPort
m_flippedImageOut.write();
}
return RTC::RTC_OK;
}
Generating Files Required for CMake Build
Editing CMakeLists.txt
Since this component uses OpenCV, you must provide the OpenCV include path, library path, and libraries to the build system.
Fortunately, OpenCV supports CMake, so simply adding the following lines will enable OpenCV libraries to be linked and used.
# check doxygen installed
find_package(Doxygen)
if(DOXYGEN_FOUND STREQUAL "NO")
message(FATAL_ERROR "Doxygen not found.")
endif()
# Add the following five lines to CMakeLists.txt
cmake_policy(SET CMP0015 NEW)
find_package(OpenCV REQUIRED)
list(APPEND INCLUDE_DIRS ${OpenCV_INCLUDE_DIRS})
list(APPEND LIBRARY_DIRS ${OpenCV_LIB_DIR})
list(APPEND LIBRARIES ${OpenCV_LIBS})
Building with Visual C++
Executing the Build
Since CMakeLists.txt has been modified, run Configure and Generate again using CMake GUI.
After confirming that CMake generation completed successfully, double-click Flip.sln and start Visual C++ 2010.
Once Visual C++ 2010 has started, build the component as shown in Figure 24.
Verifying Operation of the Flip Component
Here, we verify operation by connecting the camera component (OpenCVCameraComp or DirectShowCamComp) and the viewer component (CameraViewerComp) included with OpenRTM-aist 1.1 and later.
Starting the Name Service
Start the Name Service used to register component references.
Select:
[Start]
> [All Programs]
> [OpenRTM-aist]
> [C++]
> [tools]
and click Start Naming Service.
&color(RED){* If omniNames does not start when you click “Start Naming Service”, verify that the full computer name is set to 14 characters or fewer.}
Creating rtc.conf
RT-Components require a file called rtc.conf that specifies information such as the Name Server address and registration format.
Save the following contents as a file named rtc.conf and place it in:
Flip\FlipComp\Debug
(or the Release folder).
* If the workspace was left at the default location when Eclipse was started, the Flip folder path will be:
C:\Documents and Settings<login user name>\workspace
corba.nameservers: localhost
naming.formats: %n.rtc
Starting the Flip Component
Start the Flip component.
Execute FlipComp.exe located in the folder where you placed rtc.conf.
The FlipComp.exe window will appear.
Starting the Camera Component and Viewer Component
Start OpenCVCameraComp or DirectShowCamComp, which outputs USB camera images through an OutPort, and CameraViewerComp, which displays images received through an InPort.
These components can be started as follows.
Select:
[Start]
> [All Programs]
> [OpenRTM-aist]
> [components]
> [C++]
> [examples]
> [opencv-rtcs]
and execute OpenCVCameraComp and CameraViewerComp.
The OpenCVCameraComp and CameraViewerComp windows will appear.
In some cases, OpenCVCameraComp may fail to recognize the camera.
If this occurs, try using DirectShowCamComp instead.
Connecting the Components
As shown in Figure 25, connect the USBCameraAqcuireComp, Flip, and USBCameraMonitorComp components using RTSystemEditor.
Activating the Components
Click the ALL icon located at the top of RTSystemEditor to activate all components.
If activation is successful, the components will be displayed in light green as shown in Figure 26.
Verifying Operation
As shown in Figure 27, configuration parameters can be modified in the Configuration View.
Change the Flip component configuration parameter flipMode to values such as 0 or -1, and verify that the image is flipped correctly.
Complete Source Code for the Flip Component
Flip Component Source File (Flip.cpp)
// -*- C++ -*-
/*!
* @file Flip.cpp
* @brief Flip image component
* @date $Date$
*
* $Id$
*/
#include "Flip.h"
// Module specification
static const char* flip_spec[] =
{
"implementation_id", "Flip",
"type_name", "Flip",
"description", "Flip image component",
"version", "1.0.0",
"vendor", "AIST",
"category", "Category",
"activity_type", "PERIODIC",
"kind", "DataFlowComponent",
"max_instance", "1",
"language", "C++",
"lang_type", "compile",
// Configuration variables
"conf.default.flipMode", "1",
// Widget
"conf.__widget__.flipMode", "radio",
// Constraints
"conf.__constraints__.flip_mode", "(-1,0,1)",
""
};
/*!
* @brief constructor
* @param manager Maneger Object
*/
Flip::Flip(RTC::Manager* manager)
: RTC::DataFlowComponentBase(manager),
m_originalImageIn("originalImage", m_originalImage),
m_flippedImageOut("flippedImage", m_flippedImage)
{
}
/*!
* @brief destructor
*/
Flip::~Flip()
{
}
RTC::ReturnCode_t Flip::onInitialize()
{
// Registration: InPort/OutPort/Service
// Set InPort buffers
addInPort("originalImage", m_originalImageIn);
// Set OutPort buffer
addOutPort("flippedImage", m_flippedImageOut);
// Bind variables and configuration variable
bindParameter("flipMode", m_flipMode, "1");
return RTC::RTC_OK;
}
RTC::ReturnCode_t Flip::onActivated(RTC::UniqueId ec_id)
{
// Initialize image memory
m_imageBuff = NULL;
m_flipImageBuff = NULL;
// Initialize OutPort image size
m_flippedImage.width = 0;
m_flippedImage.height = 0;
return RTC::RTC_OK;
}
RTC::ReturnCode_t Flip::onDeactivated(RTC::UniqueId ec_id)
{
if(m_imageBuff != NULL)
{
// Release image memory
cvReleaseImage(&m_imageBuff);
cvReleaseImage(&m_flipImageBuff);
}
return RTC::RTC_OK;
}
RTC::ReturnCode_t Flip::onExecute(RTC::UniqueId ec_id)
{
// Check for new data
if (m_originalImageIn.isNew()) {
// Read InPort data
m_originalImageIn.read();
// Process InPort and OutPort image size and allocate image memory
if( m_originalImage.width != m_flippedImage.width || m_originalImage.height != m_flippedImage.height)
{
m_flippedImage.width = m_originalImage.width;
m_flippedImage.height = m_originalImage.height;
// If the InPort image size has changed
if(m_imageBuff != NULL)
{
cvReleaseImage(&m_imageBuff);
cvReleaseImage(&m_flipImageBuff);
}
// Allocate image memory
m_imageBuff = cvCreateImage(cvSize(m_originalImage.width, m_originalImage.height), IPL_DEPTH_8U, 3);
m_flipImageBuff = cvCreateImage(cvSize(m_originalImage.width, m_originalImage.height), IPL_DEPTH_8U, 3);
}
// Copy InPort image data to IplImage imageData
memcpy(m_imageBuff->imageData,(void *)&(m_originalImage.pixels[0]),m_originalImage.pixels.length());
// Flip image data from InPort. m_flipMode 0: X-axis, 1: Y-axis, -1: both axes
cvFlip(m_imageBuff, m_flipImageBuff, m_flipMode);
// Obtain image data size
int len = m_flipImageBuff->nChannels * m_flipImageBuff->width * m_flipImageBuff->height;
m_flippedImage.pixels.length(len);
// Copy flipped image data to OutPort
memcpy((void *)&(m_flippedImage.pixels[0]),m_flipImageBuff->imageData,len);
// Output flipped image data through OutPort
m_flippedImageOut.write();
}
return RTC::RTC_OK;
}
extern "C"
{
void FlipInit(RTC::Manager* manager)
{
coil::Properties profile(flip_spec);
manager->registerFactory(profile,
RTC::Create<Flip>,
RTC::Delete<Flip>);
}
};
Flip Component Header File (Flip.h)
// -*- C++ -*-
/*!
* @file Flip.h
* @brief Flip image component
* @date $Date$
*
* $Id$
*/
#ifndef FLIP_H
#define FLIP_H
#include <rtm/Manager.h>
#include <rtm/DataFlowComponentBase.h>
#include <rtm/CorbaPort.h>
#include <rtm/DataInPort.h>
#include <rtm/DataOutPort.h>
#include <rtm/idl/BasicDataTypeSkel.h>
#include <rtm/idl/ExtendedDataTypesSkel.h>
#include <rtm/idl/InterfaceDataTypesSkel.h>
// Include files for OpenCV
#include<cv.h>
#include<cxcore.h>
#include<highgui.h>
using namespace RTC;
/*!
* @class Flip
* @brief Flip image component
*
*/
class Flip
: public RTC::DataFlowComponentBase
{
public:
/*!
* @brief constructor
* @param manager Maneger Object
*/
Flip(RTC::Manager* manager);
/*!
* @brief destructor
*/
~Flip();
/***
*
* The initialize action (on CREATED->ALIVE transition)
* formaer rtc_init_entry()
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onInitialize();
/***
*
* The activated action (Active state entry action)
* former rtc_active_entry()
*
* @param ec_id target ExecutionContext Id
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onActivated(RTC::UniqueId ec_id);
/***
*
* The deactivated action (Active state exit action)
* former rtc_active_exit()
*
* @param ec_id target ExecutionContext Id
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onDeactivated(RTC::UniqueId ec_id);
/***
*
* The execution action that is invoked periodically
* former rtc_active_do()
*
* @param ec_id target ExecutionContext Id
*
* @return RTC::ReturnCode_t
*
*
*/
virtual RTC::ReturnCode_t onExecute(RTC::UniqueId ec_id);
protected:
// Configuration variable declaration
/*!
*
* - Name: flipMode
* - DefaultValue: 1
*/
int m_flipMode;
// DataInPort declaration
CameraImage m_originalImage;
/*!
*/
InPort<CameraImage> m_originalImageIn;
// DataOutPort declaration
CameraImage m_flippedImage;
/*!
*/
OutPort<CameraImage> m_flippedImageOut;
private:
// Image buffers used for processing
IplImage* m_imageBuff;
IplImage* m_flipImageBuff;
};
extern "C"
{
DLL_EXPORT void FlipInit(RTC::Manager* manager);
};
#endif // FLIP_H
Complete Source Code for the Flip Component
The complete source code for the Flip component is attached below.

























