July 24, 2020 THETA Plug-in OpenCV

Running OpenCV in Your RICOH THETA - Get Preview Edition

Running OpenCV in Your RICOH THETA - Get Preview Edition

Originally published in Japanese here.

Introduction

Hello, this is @roohii_3 from RICOH.

I previously wrote an article called “Running OpenCV in Your RICOH THETA.” This time, I would like to introduce how to use the get preview command to get video from the THETA and process it in real time using OpenCV.

I got the preview screen and turned it into black and white in real time, as seen below.

black and white

The code is available on GitHub. Please note, though, in this code post image processing data is not saved. https://github.com/theta-skunkworks/theta-plug-in-opencv-preview-sample

Use Vysor to confirm what’s being displayed on the preview screen.

Development Environment

  • OpenCV Android pack ver. 3.4.5
  • Android Studio ver. 3.3+
  • Gradle ver. 5.1.1
  • RICOH THETA V Firmware ver. 2.50.1

Preparation

RICOH THETA V

Set THETA to developer mode by referring to the links below.

THETA Plugin SDK (Android Studio project file)

Use the THETA Plugin SDK as a base. Download from here:

https://github.com/ricohapi/theta-plug-in-sdk

OpenCV Android pack

From the OpenCV Releases page, download the latest version of 3.x “Android pack”. OpenCV4 is available, but the API level is not compatible with THETA, so it may not work on THETA.

Vysor

Vysor is used to confirm what is seen on the preview screen. Install from here: https://www.vysor.io/

Preparing the project file

Opening project file

Start Android Studio and open the downloaded THETA Plugin SDK from “File > Open”.

Change the application ID or package・plug-in names to your liking (reference in Japanese).

Importing OpenCV into the project

In my previous article, I introduced how to use NDK (C/C++) in OpenCV. In this article, I will use OpenCV with Java.

This corresponds to “1.2. How to link the library statically” of my previous article (See the section “OpenCV for Android”).

Procedure

  1. Select “File > New > Import module”, and input “C:/(where OpenCV was placed)/sdk/java” in “Source directory”.

import module

  1. Create “jniLibs” directory, under “app” directory, then copy and paste OpenCV library (.so file)

copy source: C:/(where open CV was placed)/sdk/native/libs/arm64-v8a

copy destination: C:/(where project file is located)/app/src/main/jniLibs/arm64-v8ad

jnilibs dir

  1. Open ”File > Project Structure”, select ”app”→”Dependencies” tab→”+”→ select “openCVLibrary345” with ”Module dependency” → “OK”

project structure

To call OpenCV using the method in this article, use OpenCVLoader.initDebug()

MainActivity.java

protected void onResume() {
    super.onResume();
    if (!OpenCVLoader.initDebug()) {
        Log.d(TAG, "Internal OpenCV library not found.");
    } else {
        Log.d(TAG, "OpenCV library found inside package.");
        mLoaderCallback.onManagerConnected(LoaderCallbackInterface.SUCCESS);
    }
}

To use OpenCV Manager, use OpenCVLoader.initAsync(). (reference)

Implementation

Key Points when using OpenCV with THETA

OpenCV Android pack has a number of sample apps. However, if you use them as-is with THETA, the following errors will occur.

error

Within the OpenCV sample apps, Camera API is used to control the camera. However, the Camera API is locked by default in the THETA plug-in, so an error message like this appears.

To successfully use OpenCV with THETA, the following points need to be considered:

Enable the Camera API to be used in the plug-in

It is convenient to use the Camera API to get the preview screen of THETA. According to the documentation, using Broadcast Intent with “com.theta360.plug-in.ACTION-MAIN-CAMERA-CLOSE” will allow you to use the Camera API in your plug-in.

Ultimately, it is necessary to free the camera resource by using “com.theta360.plug-in.ACTION-MAIN-CAMERA-OPEN”.

In actuality, in the class that inherits PluginActivity, if you use notificationCameraOpen() at the beginning and notificationCameraClose() at the end, these Intents will be performed.

Use THETA’s Original Parameters

OpenCV has a class called JavaCameraView, and it appears that various settings and operations are performed internally by the Camera API. However, to use this with THETA, it is necessary to set THETA’s original parameters. THETA’s original parameters are summarized here: https://api.ricoh/docs/theta-plug-in-reference/camera-api/

For example, in the OpenCV sample application, setting “RicMoviePreview1920” in the “RIC_SHOOTING_MODE” parameter when starting the camera will work.

Source code

Only the important parts are pulled out here. The outline is made in reference to “tutorial-x-xxx” of OpenCV sample code. See GitHub for source code details.

Enabling・Disabling the Camera API

As mentioned in the key points section, calling notificationCameraClose()when you start up the plug-in and calling close() when you close the plug-in requires enabling or disabling the Camera API.

Calling close() inside onPause() may cause a failure, so call close() when “long press Mode button (signaling the end of the plug-in)”.

MainActivity.java

@Override
protected void onCreate(Bundle savedInstanceState) {

    // ...Abbreviated...

    // Set a callback when a button operation event is acquired.
    setKeyCallback(new KeyCallback() {
        @Override
        public void onKeyDown(int keyCode, KeyEvent keyEvent) {}

        @Override
        public void onKeyUp(int keyCode, KeyEvent keyEvent) {}

// called at the end (long press Mode button)

        @Override
        public void onKeyLongPress(int keyCode, KeyEvent keyEvent) {
            if (keyCode == KeyReceiver.KEYCODE_MEDIA_RECORD) {
                if (!isEnded) {
                    // ...Abbreviation...

                    // Disable Camera API
                    close();
                    isEnded = true;
                }
            }
        }
    });

    // Enable Camera API
    notificationCameraClose();

    // ...Abbreviated...
}

THETA’s Original Camera Control Class

I have added a class called ThetaView, which is JavaCameraView optimized for THETA use. Except for the initializeCamera() method, it is pretty much what JavaCameraView is. The only changes are “setting the value “RicMoviePreview1920” in a parameter named “RIC_SHOOTING_MODE” and omitting unnecessary descriptions.

ThetaView.java

public class ThetaView extends CameraBridgeViewBase implements PreviewCallback {

// according to the "RicMoviePreviewXXXX" parameter value define the frame size.

    private static final int PREVIEW_SIZE_WIDTH = 1920;
    private static final int PREVIEW_SIZE_HEIGHT = 960;
    private static final String PREVIEW_SIZE_PARAM = "RicMoviePreview1920";

    protected boolean initializeCamera(int width, int height) {
        boolean result = true;
        synchronized (this) {
            mCamera = null;

            mCamera = Camera.open();
            if (mCamera == null)
                return false;

            /* Now set camera parameters */
            try {
                Camera.Parameters params = mCamera.getParameters();

                // Set frame size
                Size frameSize = new Size(PREVIEW_SIZE_WIDTH, PREVIEW_SIZE_HEIGHT);
                Log.d(TAG, "Set preview size to " + Integer.valueOf((int) frameSize.width) + "x" + Integer.valueOf((int) frameSize.height));
                params.setPreviewSize((int) frameSize.width, (int) frameSize.height);

                // Set shooting mode
                params.set("RIC_SHOOTING_MODE", PREVIEW_SIZE_PARAM);

                mCamera.setParameters(params);

                // ...Abbreviated...

                /* Finally we are ready to start the preview */
                Log.d(TAG, "startPreview");
                mCamera.startPreview();

            } catch (Exception e) {
                result = false;
                e.printStackTrace();
            }
        }

        return result;
    }

Image Processing Section

You can get one frame of preview screen with the onCameraFrame() method of the class that implements CvCameraViewListener2. I was able to get RGBA with rgba() and grayscale with gray(). Here, grayscale was used to get the frame, and OpenCV was used to process binarizing.

MainActivity.java

public class MainActivity extends PluginActivity implements CvCameraViewListener2 {

    private ThetaView mOpenCvCameraView;
    private Mat mOutputFrame;

    // ...Abbreviated...

    public void onCameraViewStarted(int width, int height) {
        mOutputFrame = new Mat(height, width, CvType.CV_8UC1);
    }

    public void onCameraViewStopped() {
        mOutputFrame.release();
    }

    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

// RGBA can be obtained with rgba() and 1frame of preview can be obtained in grayscale with gray().

        Mat gray = inputFrame.gray();

        // do some image processing

// Only binarizing process here

        Imgproc.threshold(gray, mOutputFrame, 127.0, 255.0, Imgproc.THRESH_BINARY);

        return mOutputFrame;
    }

Execution

Execute by connecting THETA with UBS and starting Visor.

Setting permissions

The first time you start the plug-in, the plug-in will drop immediately because there are no permissions set in “Permissions”.

Open Settings app, set “Camera” permission to ON from “OpenCV Preview Sample > Permissions”.

permissions

The following screen will appear when you set permissions and re-execute the plug-in.

success

Conclusion

A simple process like this time worked smoothly without taking a lot of time. I am curious to see what would happen when the process is a little more complicated. This time I only got the preview screen, so I would like to try using the picture taking process. Here is one such article: Running OpenCV in THETA Get Preview Frame Edition (added June 5, 2019, in Japanese).

About the RICOH THETA Plug-in Partner Program

If you are interested in THETA plug-in development, please register for the partner program!

Please be aware that the THETA with its serial number registered with the program will no longer be eligible for standard end-user support.

For detailed information regarding the partner program please see here.

The registration form is here.