April 29, 2021 plug-ins OpenCV

Taking Pictures Using Ghost Town Effect (THETA plug-in development example)

Taking Pictures Using Ghost Town Effect (THETA plug-in development example)

Introduction

Original article appeared in Japanese here.

This is Yuuki_S from Ricoh. RICOH makes a camera called RICOH THETA that can shoot 360 degree images and videos. The RICOH THETA V and Z1 use Android as the OS, so you can customize your THETA as if you were making an Android application. The customization function is called a “plug-in.” (See the end of this article for details).

The Ghost Town effect, a unique photography technique similar to long exposure, is interesting.

I wanted to take a picture like this myself, so I built a THETA plug-in to do it.

ghosttown

What do you think? The people disappear and it appears to be a ghost town.

I will show you how to make this THETA plug-in. If you want to just jump to the final image instead of how to make it, please see the examples at the end.

Introduction to OpenCV

If you want to perform image processing efficiently, using OpenCV is the quickest route. OpenCV can be used with THETA plug-ins, and past articles have covered how to install it:

Running OpenCV in Your RICOH THETA. This article introduces how to handle OpenCV in NDK written in C ++. In this article, I will introduce another method, statically linking the library and handling OpenCV in Java.

It was roughly divided into three steps, and it took about 5 minutes to introduce. It’s simple.

  1. Getting OpenCV Android Pack
  2. Importing OpenCV into your project
  3. Editing Source Code (Implementing OpenCV Interface)

1. Getting OpenCV Android Pack

Download the latest version of OpenCV version 3 “Android pack” from the OpenCV Releases page.

opencv

Note that version 4 does not match the required Android API level with THETA, so there may be many features that do not work. This time, I used 3.4.12.

Extract the downloaded Zip file and place the folder in any location. I’ll link to this folder in Android Studio later, so I think it’s a good place to find it. I created a folder called OpenCV directly under the C drive and put it in it.

location

2. Importing OpenCV into your project

After opening the project in Android Studio where you want to import OpenCV, select “File”-> “New”-> “Import Module” from the menu, and select “C: / (where OpenCV is located) / sdk / java” in “Source directory”. Hit enter.

import

After adding it, the integrity of the set of project files is checked. You will be notified that there is a problem with the following line in AndroidManifest.xml for “openCV3412”, so delete this line. (It is line 7. In the figure below it has already been deleted.)

fix

In addition, please modify “compileSdkVersion” and “targetSdkVersion” of build.gradle (Module: openCVLibrary3412) to “25”.

android {
   compileSdkVersion 25
   ~shortened~
   defaultConfig {
   ~shortened~
   targetSdkVersion 25
   }
   ~shortened~
  }

Some Android Camera2 APIs used by some OpenCV version 3 functions are not supported by THETA. Here’s how to ignore the inconsistencies and let the build go through. I did this because it does not affect the basic image processing functions and is simpler than modifying the library side to resolve inconsistencies.

Next, create a folder called “jniLibs” in the specified location in the project, copy the file under “arm64-v8a” (the library with the extension so) from the OpenCV library, and save it there.

Copy source: C: / (where OpenCV is placed) / sdk / native / libs / arm64-v8a

Copy to: C: / (where the project files are) / app / src / main / jniLibs / arm64-v8a

arm64-v8a

Next, configure the settings to enable this file. From the Android Studio menu, open “File”-> “Project Structure” and from the left, select the “Dependencies” tab, then the “app” tab, and then click the “+” under the word “Declared Dependencies”. Choices will be displayed, click “Module Dependency”.

module-dependency

Check “openCV Library 3412” in the displayed dialog and press OK. This completes the importing OpenCV into your project.

3. Editing Source Code (Implementing OpenCV Interface)

Finally, I will describe loading OpenCV in the THETA plug-in. Add the following to onResume so that it will be loaded when the plug-in starts.

protected void onResume() {
 super.onResume();
 if (!OpenCVLoader.initDebug()) {
   Log.d(TAG, "Internal OpenCV library not found.");
 } else {
   Log.d(TAG, "OpenCV library found inside package.");
 }
 ~shortened~
}

You have now completed handling OpenCV from the Java side.

Implementing Ghost Town Effect

This time, I wanted to check the result in real time when shooting outside, so I based it on the project from the following article where images can be previewed on a smartphone.

Make live previews easier to handle with THETA plug-ins (in Japanese).

The method is to apply processing to the image acquired by live preview. Although it is easy to check on the spot and easy to implement, it has the disadvantage that the resolution is about full HD (1920x960) because the image is processed using live preview. I decided on this method to prioritize convenience, but I feel that it is actually more normal to take time-lapse photography and process it.

However, I think that the image processing of the effect will be the same even if you use time lapse.

The implementation can actually be done with one OpenCV function.

Imgproc.accumulateWeighted

This function can obtain the moving average of continuous images and is used to extract the background image by detecting moving objects based on background subtraction. (Initially, I completely forgot about the existence of this function and tried to implement it from scratch. I’m glad I noticed!)

The code that uses this is as follows. The actual effect processing is only the one line mentioned above, and before and after that is only the conversion of the image format.

   public void drawGTEffectThread() {;
        new Thread(new Runnable() {
            @Override
            public void run() {
                android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND);
                Mat averageImage = new Mat(960, 1920, CvType.CV_32FC(4));//background image
                Bitmap back_frame = Bitmap.createBitmap(1920,960, Bitmap.Config.ARGB_8888);
                while (mFinished == false)//live preview state
                {
                    byte[] jpegFrame = latestLvFrame;
                    if ( jpegFrame != null )
                    {
                        if(toggle_GTEffect == true)
                        {
                            Bitmap frame = BitmapFactory.decodeByteArray(jpegFrame, 0, jpegFrame.length);//Bitmap conversion of preview image
                            matFrame =  new Mat(back_frame.getHeight(), back_frame.getWidth(), CvType.CV_8UC3);
                            matFrameBG = new Mat(back_frame.getHeight(), back_frame.getWidth(), CvType.CV_8UC3);

                            Utils.bitmapToMat(frame,matFrame);//Mat conversion of preview image
                            Mat matFrame_f = new Mat(960, 1920, CvType.CV_32FC(3));//Prepare float Mat
                            matFrame.convertTo(matFrame_f,CV_32FC4, 1/255.0);//Convert preview image to float

                            Imgproc.accumulateWeighted(matFrame_f,averageImage,0.03);

                            averageImage.convertTo(matFrameBG,CvType.CV_8UC3,255);//Convert background image from float to Mat
                            Utils.matToBitmap(matFrameBG,back_frame);//Convert background image from Mat to Bitmap
                            ByteArrayOutputStream baos = new ByteArrayOutputStream();
                            back_frame.compress(Bitmap.CompressFormat.JPEG, 100, baos);//Jpeg conversion for preview display

                            latestFrame_Result = baos.toByteArray();

                            Map<TextArea, String> output = new HashMap<>();//Display status on Z1 OLED
                            output.put(TextArea.MIDDLE, "");
                            output.put(TextArea.BOTTOM, "Processing...");
                            notificationOledTextShow(output);
                        }else{
                                Map<TextArea, String> output = new HashMap<>() ;
                                output.put(TextArea.MIDDLE, "");
                                output.put(TextArea.BOTTOM, "Stopping...");
                                notificationOledTextShow(output);
                                latestFrame_Result = latestLvFrame;
                            }
                    }else{
                        try {
                            Thread.sleep(125);
                        } catch (InterruptedException e) {e.printStackTrace();}
                    }}}}).start();
    }

After that, call this thread with onResume and switch between false and true for toggle_GTEffect from the button, and you’re done.

One thing to note is that this implementation has a fixed input resolution for the live preview, so an error will occur if different resolutions are entered. Therefore, you need to specify the preview format as follows.

protected void onResume() {
       ~shortened~
        //Start LivePreview
        previewFormatNo = GetLiveViewTask.FORMAT_NO_1920_8FPS;
    ~shortened~
    }

In addition, change the JavaScript on the web display side. In the initial state, 2 (960x480) is selected.

javascript

var PREVIEW_FORMAT = 5;

With this, you’re all set!

Checking Operation & Examples

Set up your THETA outdoors and connect to your smartphone.

setup

If you access 192.168.1.1:8888 from your smartphone browser, the preview will be displayed as shown below.

preview

After that, if you turn on the effect, you can get a photo with only the background. I really like how it emerges slowly.

background

Save the image from the browser. Just press and hold (on iPhone) and select “Add to Photos.”

Here is an example taken in a place with a little more people.

more-people more-people

People in motion are nicely hidden. However, due to this technique, people who have stopped for a long time show up.

Since it does not look great in equirectangular, I edited it with THETA + app.

final final final

Conclusion

I was able to make a THETA plug-in that can easily apply the Ghost Town Effect. It’s fun to manipulate effects (image processing) that you come up with and see the results on the spot.

Since OpenCV can be used quite easily, please try to make a THETA plug-in that uses your own favorite effects.

About the RICOH THETA Plug-in Partner Program

If you are interested in THETA plug-in development, please register for the partner program!

Please be aware that the THETA with its serial number registered with the program will no longer be eligible for standard end-user support.

For detailed information regarding the partner program please see here.

The registration form is here.