Main Content

Enhanced Color Replacement with Customized Code Using Android Devices

This example uses:

Open Model

This example shows how to use FromApp block from Simulink® Support Package for Android® Devices to receive data and add a touch interface in an Android application. This example demonstrates a workflow to customize the Simulink generated Android project using Android Studio.

Introduction

A color detection algorithm identifies pixels in an image that match a specified color or color range. The detected pixels in the original image are replaced by the pixels from a different image. This process is known as color replacement or Chroma Key.

This model internally uses the same sub-system as the Color Detection and Color Replacement examples.

In this example, you will choose the reference color for detection from selected position on your Android device screen. You will configure the model and develop this user interface by updating code to get the touched location and corresponding pixel info from a video frame.

This example shows:

  • How to configure a Simulink model for receiving data from the application.

  • An example for importing the generated Android application into Android Studio.

  • How to add custom code for developing advanced features in an application.

Prerequisites

Required Hardware

  • Android device, configured using androidhwsetup

  • USB cable to connect the device to host computer

Task 1 - Build Simulink Model and Import Generated Project

In this task, you will build a pre-configured model. You will import generated project in Android Studio for further development.

1. Open the androidcolorreplacementcustom Simulink model.

2. Load the template image into the matrix RGB_background by executing the following command in the MATLAB® command window:

RGB_background = imread('image_640_480.png');

3. In the model, double-click the FromApp block. Note the field Method name is pickColorFromDeviceScreen. In the next tasks you will implement this method.

4. Connect your configured Android device to your host computer. If you have not previously configured your device, run androidhwsetup from the MATLAB command prompt.

5. On the Modeling tab of the toolstrip, select Model Settings.

6. Select the Hardware Implementation pane and set the Hardware board to Android Device.

7. On the Hardware tab of the Simulink model, in the Mode section, select Run on board and then click Build, Deploy & Start.

8. Locate the generated Android project folder androidcolorreplacementcustom_ert_rtw in your current working directory.

9. In Android Studio, import your project from the androidcolorreplacementcustom subfolder. For detailed steps refer to Import Project to Google Android Studio.

Task 2 - Run Base Application

In this task you will modify the generated app code to run a simple case of color replacement detecting a fixed color.

1. In your Android project, open androidcolorreplacementcustom.java file located at androidcolorreplacementcustom > src > main > java > com > example > androidcolorreplacementcustom

2. In the androidcolorreplacementcustom class, add the following implementation of pickColorFromDeviceScreen method. This method returns RGB components of fixed color as an output of FromApp block in the Simulink model. Note that a method declaration can be added anywhere inside the class as long as it does not overlap with other declarations.

public int[] pickColorFromDeviceScreen()
{
    int []rgb_send = {211,27,10};
    /* R, G, B components of the color Red */
    return rgb_send; //reference color for detection
}

3. In Android Studio IDE, click the Run Button to build the updated application on device.

4. In the Choose Device pane, select your connected device. Press OK to deploy and run the app on the selected device.

5. The color replacement camera app appears, detecting red colors in the scene.

Task 3 - Update Code to Use Touched Position

In this task, you will update code in the application to get screen touch location and choose the reference color at that location. You will add extra code to calculate the pixel location on the video frame and get color components for detection. You will run the updated application to see result using touch interface.

1. At the top section in the androidcolorreplacementcustom.java file, add the following lines of code after other import declarations.

import android.view.MotionEvent;
import android.view.Window;
import android.util.DisplayMetrics;
import android.graphics.Rect;
import android.view.View;

2. In the androidcolorreplacementcustom class implementation at the starting of class body, add the following code to declare member variables after other member variable declarations.

int x = -1;  //X Co-ordinate of touched position
int y = -1;  //Y Co-ordinate of touched position
int [] rgb_send = new int[3];
// to store RGB of reference color last at touched location
DisplayMetrics metrics;

3. Add the following code at the end of registerCamera method present in androidcolorreplacementcustom class, to get touched location value into x, y member variables.

  mCameraView.setOnTouchListener(new View.OnTouchListener() {
    public boolean onTouch(View v, MotionEvent event) {
     x = ((int) event.getX());
     y = ((int) event.getY());
        return false;
 }
  });

4. In the same class, add the following methods to find view offsets in the layout of application. These offsets will be used to establish correspondence between touch location and pixel position in the image.

/** Get Height of the Status Bar from the Layout for the device. */
public int getStatusBarHeight() {
     Rect r = new Rect();
     Window w = getWindow();
     w.getDecorView().getWindowVisibleDisplayFrame(r);
     return r.top;
 }
/** Get Height of the Title Bar from the app title name */
public int getTitleBarHeight() {
    int viewTop =
        getWindow().findViewById(Window.ID_ANDROID_CONTENT).getTop();
    return (viewTop - getStatusBarHeight());
}

5. Replace the existing onCameraFrame method with following implementation

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
  mCameraBufInput = inputFrame.rgba();
  metrics = new DisplayMetrics();
  getWindowManager().getDefaultDisplay().getMetrics(metrics);
    if (x != -1 && y != -1) { // For Latest touched position
    //if true, touch detected
      if ((y < metrics.heightPixels) && (x < metrics.widthPixels))
      {
        int h_off, rel_h;
        int w_off, rel_w;
        /** calculation of offsets and scaling parameters
        * for application window to original video frame */
        h_off = getTitleBarHeight() + getStatusBarHeight();
        rel_h = (metrics.heightPixels - h_off);
        rel_w = ((mCameraBufInput.width() * rel_h)
              / mCameraBufInput.height());
        w_off = (metrics.widthPixels - rel_w) / 2;
        /**  Calculating  correspondence for touch position
        *  to the pixel co-ordinates in the video frame */
        int posY = ((y - h_off) * mCameraBufInput.height())
              / rel_h;
        int posX = ((x - w_off) * mCameraBufInput.width())
              / rel_w;
        /* get the pixel info for calculated offset in the image */
        double []rgb = mCameraBufInput.get(posY , posX );
        if(rgb != null) {
              rgb_send[0] = (int)rgb[0];
              rgb_send[1] = (int)rgb[1];
              rgb_send[2] = (int)rgb[2];
        }
    }
    x = -1;  /* Resetting x,y for the next touch event */
    y = -1;
  }
  return mCameraBufOutput;
}

6. Replace the pickColorFromDeviceScreen method with the following implementation to return RGB components of selected reference color as an output of FromApp block in the Simulink model.

public int[] pickColorFromDeviceScreen()
{
/* Reference color from the video frame */
  return rgb_send;
}

7. In Android Studio IDE, click the Run button to build the updated application on device.

8. In the Choose Device pane, select your connected device. Press OK to deploy and run the app on the selected device.

9. Once the updated application appears on device, touch at different regions on the screen image to observe the color replacement.

The following screen shows replacement of the orange color wall with the background image.

The following screen shows replacement of the white color with background image.

Other Things to Try

  • Modify the Simulink model to receive reference color data from the FromApp block. Use the ToApp block from Simulink Support Package for Android Devices to receive the data for customization. In the application, implement a method and add customized code to print the received data in the log.