Control THETA from Bluetooth Audio Devices

Original article: https://qiita.com/mShiiina/items/4b9f74625deeb43763e9

Introduction

This is RICOH’s @mShiiina

RICOH sells a camera called RICOH THETA that can shoot in all directions in 360 degrees.

The RICOH THETA V and RICOH THETA Z1 use Android for OS. THETA can be customized just like making Android apps, and this customization function is called “plug-in.” (See the end of this article for details.)

With the firmware update of June 2020, Bluetooth Classic devices can be used with plug-ins.

This time, I made a plug-in that enables remote shooting using Bluetooth earphones and Bluetooth Audio. Here’s the video:

https://www.youtube.com/watch?v=NN33SM5_SJQ

This section introduces how to connect THETA with the following Bluetooth Audio devices and operate THETA remotely.

・Bluetooth Earphones: Anker Soundcore Liberty Air

・Bluetooth Earphone：JPRiDE TWS-520

・Bluetooth Audi：Anker Soundcore Icon Mini

In addition to the above, I have checked a number of other operations and posted a list in the last section. As you will see in the last section, compatibility (connectable/not connectable) and usability seem to differ depending on the specific audio device. It is impossible for us to try all the audio devices out there. Please bear in mind that your earphones might not work. (That said, even Apple’s AirPods Pro worked, so the most should work!)

The following THETA functions can be assigned to the operation of the Bluetooth Audio device.

1. Still image shooting

2. Exposure compensation setting

3. Volume setting for normal notification sound

When setting the exposure compensation, audio play and set value reading were done following this article (in Japanese).

The firmware must be newer than below versions. Don’t forget to update to the latest firmware!

・THETA V: 3.40.1

・THETA Z1: 1.50.1

Bluetooth Profile

This plug-in can use Bluetooth devices that support the following profiles.

・Headset Service Profile (HSP) - Microphone cannot be used

・Audio/Video Remote Control Profile (AVRCP)

Bluetooth Earphones Connection and Operation

I tried out Anker Soundcore Liberty Air and JPRiDE TWS-520.

Anker Soundcore Liberty Air

Corresponding profile: HSP, AVRCP

JPRiDE TWS-520

Corresponding profile: HSP, AVRCP

Connection procedure:

1. Set Soundcore Liberty Air or TWS-520 in pairing mode.

2. Launch the plug-in (search for Bluetooth Classic device for up to 12 seconds).

3. Wait until connected.

THETA action assignment for earphone operation:

Bluetooth earphones have touch-type multifunction buttons that allow users to operate the devices they are connected to. Here’s how THETA actions were assigned.

Bluetooth Audio Connection and Operation

Here I used Anker Soundcore Icon Mini.

Corresponding profile: Audio/Video Remote Control Profile (AVRCP)

THETA action assignment by audio operations:

Bluetooth Audio has buttons to control connected devices. Here’s how THETA actions were assigned.

Connection procedure:

1. Set Soundcore Icon Mini into pairing mode.

2. Launch the plug-in (search for Bluetooth Classic device for up to 12 seconds).

3. Wait until connected.

Operation of THETA camera

THETA actions by THETA operations:

LED Display

With the THETA V, LED3 shows the searching, connection and disconnection status of Bluetooth devices.

OLED Display

With the THETA Z1, the Bluetooth device’s searching, connection, and disconnection status are displayed in the third row of the OLED display on the front of the camera.

Displaying webUI’s keyCode

The keyCode received by a plug-in is displayed using a webUI.

Limitations

When a newly added option [_bluetoothRole](https://api.ricoh/docs/theta-web-api-v2.1/options/_bluetooth_role/) is set to Central or Central_Peripheral (where remote control function is ON in normal operation), it cannot be used with THETA. This is because the THETA side repeats the execution/stop/connection process of the Bluetooth Classic search process.

It is possible to save when the plug-in starts and recover when it ends, but that implementation is omitted here. The method to restart and restore is mentioned in the second half of the “Suppressing the remote control search function on the THETA app side” section. Please add an implementation if you like.

Program Description

This is the source code.

General Description

Using the THETA plug-in SDK as a base, files in ~\app\src\main\java\com\theta360\pluginapplication and MainActivity have been modified.

File description

Manifest FIle

This plug-in adds permissions and services to AndroidManifest.xml.

Permission

The following permission is defined in order to use Bluetooth Classic devices.

<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN" />
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />

When running the program as a developer, manually enable the “Location” permission in Settings->App->HEADSET Remote from Visor.

This plug-in is scheduled to be published in the official plug-in store.

If you installed from the official plug-in store, the permission will be enabled during installation and no manual configuration is necessary.

Service

Service to handle Bluetooth Classic devices is added.

<service android:name="com.theta360.pluginapplication.bluetooth.BluetoothClientService" />

Main processing (MainActivity.java)

Existing template part (keyCode interpretation part)

Basically, just execute the action corresponding to the received keyCode with the following code.

execKeyProcess(keyCode2KeyProcess(keyCode));

Bluetooth Classic Enable/Disable Switch

To enable Bluetooth Classic, first create EnableBluetoothClassicTask when the plug-in starts.

       new EnableBluetoothClassicTask(getApplicationContext(),EnableBluetoothClassicTask).execute();

Within Task, configure option [_bluetoothClassicEnable](https://api.ricoh/docs/theta-web-api-v2.1/options/_bluetooth_classic_enable/) to true, and enable Bluetooth Classic.

       HttpConnector camera = new HttpConnector("127.0.0.1:8080");
        String errorMessage = camera
                .setOption("_bluetoothClassicEnable", Boolean.toString(Boolean.TRUE));

If Bluetooth module power source option [_bluetooth_power](https://api.ricoh/docs/theta-web-api-v2.1/options/_bluetooth_power/) is OFF, switch it to ON to complete.

      String bluetoothPower = camera.getOption("_bluetoothPower");
        if (bluetoothPower.equals("OFF")) {
            errorMessage = camera.setOption("_bluetoothPower", "ON");
            if (errorMessage != null) { // If parameter setting fails, an error message is displayed
                return "NG";
            }
        }

To reflect enable/disable change in Bluetooth Classic, it is necessary to OFF/ON the power of Bluetooth module.

If you change _bluetoothClassicEnable while _bluetoothClassicPower is ON, the THETA app side will automatically operate Bluetooth module’s OFF/ON power.

Bluetooth Classic Device’s Search and Connection

After enabling Bluetooth Classic, proceed with Bluetooth Classic device search and connect. Android’s Bluetooth API is used.

Here are the steps:

1. Generate service

2. Start search

3. Pair the found devices

4. Connect the paired devices

First, generate the service.

   private EnableBluetoothClassicTask.Callback mEnableBluetoothClassicTask = new EnableBluetoothClassicTask.Callback() {
        @Override
        public void onEnableBluetoothClassic(String result) {
            getApplicationContext().startService(new Intent(getApplicationContext(), BluetoothClientService.class));
        }
    };

Next, start Bluetooth Classic device search. This will stop 12 seconds after starting, according to Android specs. If still searching, stop once, then start again.

   private void startClassicScan() {
        if (mBluetoothAdapter.isDiscovering()) {
            mBluetoothAdapter.cancelDiscovery();
        }
        boolean isStartDiscovery = mBluetoothAdapter.startDiscovery();
        Log.d(TAG, "startClassicScan :" + isStartDiscovery);
    }

If the connection target is found, proceed with pairing. Connection target device is decided by CoD. CoD is a value that identifies the intended use of the device specified by Bluetooth SIG.

       @Override
        public void onFound(BluetoothDevice bluetoothDevice,
                BluetoothClass bluetoothClass,
                int rssi) {
            String name = bluetoothDevice.getName();
            Log.d(TAG, "name" + name);
            if (name != null) {
                int type = bluetoothDevice.getType();
                Log.d(TAG, "type" + String.valueOf(type));
                if (type == bluetoothDevice.DEVICE_TYPE_CLASSIC || type == bluetoothDevice.DEVICE_TYPE_DUAL) {
                    int classNo = bluetoothClass.getDeviceClass();
                    Log.d(TAG, "class" + classNo);
                    if ((classNo == COD_AUDIO_VIDEO_HEADPHONES) || (classNo
                            == COD_AUDIO_VIDEO_WEARABLE_HEADSET_DEVICE)) {
                        stopClassicScan();
                        bluetoothDevice.createBond();
                    }
                }
            }
        }

By targeting “DEVICE_TYPE_DUAL” for pairing, it was possible to support Apple AirPods Pro.

Connection after pairing is complete. Pairing is completed when bondState becomes BOND_BONDED.

       public void onBondStateChanged(BluetoothDevice bluetoothDevice, int bondState) {
            Log.d(TAG, "onBondStateChanged");
            if (bondState == BluetoothDevice.BOND_BONDED) {
                connect(bluetoothDevice);
            }
        }

Specify a profile using getProfileProxy, and associate it with service listener.

BluetoothProfile.HEADSET is used for the profile.

   private void connect(BluetoothDevice device) {
        int state = mBluetoothAdapter.getProfileConnectionState(BluetoothProfile.HEADSET);
        if (state == BluetoothProfile.STATE_DISCONNECTED) {
            int type = device.getType();
            if (type == BluetoothDevice.DEVICE_TYPE_CLASSIC || type == BluetoothDevice.DEVICE_TYPE_DUAL) {
                int classNo = device.getBluetoothClass().getDeviceClass();
                Log.d(TAG, "class" + classNo);
                boolean isGetProfile = false;
                if ((classNo == COD_AUDIO_VIDEO_HEADPHONES) || (classNo
                        == COD_AUDIO_VIDEO_WEARABLE_HEADSET_DEVICE)) {
                    {
                        isGetProfile = mBluetoothAdapter
                                .getProfileProxy(mContext, mServiceListener,
                                        BluetoothProfile.HEADSET);
                    }
                }

                Log.d(TAG, "isGetProfile :" + isGetProfile);
                if (isGetProfile) {
                    mBluetoothDevice = device;
                }
            }
        }
    }

By targeting “DEVICE_TYPE_DUAL” for pairing, it was possible to support Apple AirPods Pro.

Perform connect() of BluetoothHeadset class inside onServiceConnected ().

       public void onServiceConnected(int i, BluetoothProfile bluetoothProfile) {
            try {
                if (i == BluetoothProfile.HEADSET) {
                    Class bluetoothHeadset = Class
                            .forName("android.bluetooth.BluetoothHeadset");
                    Object object = bluetoothHeadset.cast(bluetoothProfile);
                    Method getConnectionState = bluetoothHeadset
                            .getDeclaredMethod("getConnectionState", BluetoothDevice.class);
                    int connectionState = (int) getConnectionState
                            .invoke(object, mBluetoothDevice);
                    if (connectionState == BluetoothProfile.STATE_DISCONNECTED) {
                        Method connect = bluetoothHeadset
                                .getDeclaredMethod("connect", BluetoothDevice.class);
                        boolean isConnected = (boolean) connect
                                .invoke(object, mBluetoothDevice);
                        Log.d(TAG, "isConnected : " + isConnected);
                        if (isConnected) {
                            mProfile = bluetoothProfile;
                        }
                    }
                }
            } catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException |
                    InvocationTargetException e) {
                e.printStackTrace();
            }
        }

If the status acquired by BluetoothAdapter.EXTRA_CONNECTION_STATE becomes BluetoothAdapter.STATE_CONNECTED, the connection is complete.

       public void onConnectionStateChanged(int connectionState) {
            Log.d(TAG, "onConnectionStateChanged");
            if (Build.MODEL.equals("RICOH THETA Z1")) {
                Intent intent = new Intent(Constants.ACTION_OLED_TEXT_SHOW);
                if (connectionState == BluetoothAdapter.STATE_DISCONNECTED) {
                    intent.putExtra(Constants.TEXT_BOTTOM, "disconnected");
                } else if (connectionState == BluetoothAdapter.STATE_CONNECTED) {
                    intent.putExtra(Constants.TEXT_BOTTOM, "connected");
                }
                sendBroadcast(intent);
            } else {
                //LED3点灯
                Intent intentLedShow = new Intent("com.theta360.plugin.ACTION_LED_SHOW");
                if (connectionState == BluetoothAdapter.STATE_DISCONNECTED) {
                    intentLedShow.putExtra("color", LedColor.BLUE.toString());
                } else if (connectionState == BluetoothAdapter.STATE_CONNECTED) {
                    intentLedShow.putExtra("color", LedColor.WHITE.toString());
                }
                intentLedShow.putExtra("target", LedTarget.LED3.toString());
                mContext.sendBroadcast(intentLedShow);
            }
        }

Volume Adjustment

There are two types of volume data handled by this plug-in.

1. Stream Volume (Android’s AudioManager’s parameter, STREAM_MUSIC is used for StreamType.)

Android is managing this. This is the actual output volume.

1. Shutter Volume (Using API Options)

The THETA app is managing this. This is not the actual output volume. In order to change the actual output volume, stream volume change is required.

When outputting audio with the THETA application using Intent from the plug-in, the value of the stream volume is calculated from the shutter volume and the audio is output after setting the stream volume.

In other words, when playing audio with the THETA app, the stream volume is adjusted to the volume of the shutter volume when the Intent was sent.

Shutter volume can be changed with option [_shutter_volume](https://api.ricoh/docs/theta-web-api-v2.1/options/_shutter_volume/)

There are two ways to control volume with this plug-in.

・Bluetooth Audio operation (AVRCP)

・THETA camera button operation

Volume is changed using ChangeVolumeTask.java.

When the volume is changed by Bluetooth Audio operation, stream volume is changed on the Android side and the plug-in is notified of Intent: “android.media.VOLUME_CHANGED_ACTION” indicating that the stream volume has been changed.

At this point, shutter volume hasn’t changed, so it is changed in the plug-in. A task is created by passing the stream volume and ACTION_TYPE_SET_VOL to ChangeVolumeTask.

   private MediaReceiver.Callback mMediaReceiverCallback = new MediaReceiver.Callback() {
        @Override
        public void onChangeVolume(int stream_type, int prev_vol, int vol) {
            if (stream_type == AudioManager.STREAM_MUSIC) {
                if (vol != prev_vol) {
                    new ChangeVolumeTask(getApplicationContext(), vol,
                            ChangeVolumeTask.ACTION_TYPE_SET_VOL).execute();
                }
            }
        }
    };

When the volume is changed by pressing the buttons on the THETA camera, the stream volume is calculated from the shutter volume at that time, and the volume increased or decreased by one step is set for the shutter volume and stream volume, respectively.

          case SET_VOL_PLUS:
                new ChangeVolumeTask(getApplicationContext(), 0,
                        ChangeVolumeTask.ACTION_TYPE_UP_VOL).execute();
                break;
            case SET_VOL_MINUS:
                new ChangeVolumeTask(getApplicationContext(), 0,
                        ChangeVolumeTask.ACTION_TYPE_DOWN_VOL).execute();
                break;

The sound notification when the volume is changed follows SoundManagerTask here (in Japanese). Use AudioManager in the plug-in to play audio. StreamType during playback is set to STREAM_MUSIC to output audio from the connected Bluetooth device.

Suppressing Remote Control Search Function on THETA App Side

Although not implemented in this plug-in, in order to use Bluetooth Client function in the plug-in, Bluetooth remote control function has to be turned OFF on the THETA app side.

Bluetooth remote control function can be switched with option _bluetoothRole.

For example, option _bluetoothRole was acquired with onCreate() when starting the plug-in, then return it to the acquired Role with onPause(), it is possible to turn OFF Bluetooth remote control function of THETA camera only while the plug-in is starting.

Bluetooth Audio Devices Confirmed to Work

We have checked the operation of some other Bluetooth Audio devices. Here is the list of summarized results.

・ N/A indicates the earphone side has no function.

・ The manual for JPRiDE TWS-520 (PDF, in Japanese) has an explanation that the volume can be adjusted with a single tap, but since it could not be recognized by smartphones (iOS or Android), it was treated as N/A.

・ With Anker Liberty Air, it was difficult to operate the earphones themselves because of the touch recognition position and the touch sensitivity.

・ The volume control of VANKYO X100 requires a long tap. Since the key code is sent continuously during tapping, it is difficult to set the target volume. However, other operations are easy to perform, and you may be able to purchase it for about 3000 yen with an Amazon sale. Perhaps, you could purchase it exclusively for THETA use. This is recommended.

・ When pairing an Apple AirPods Pro with an Android OS device, you need to press the button on the case to create the pairing state. Please be careful.

Summary

In this article, remote shooting using a Bluetooth device has been introduced. Please try some of these easy remote operations using Bluetooth earphones! Voice assistant activation of earphones was also tested. However, it was hard to grasp the behavior as notification was sometimes recognized and sometimes not.

The main point this time was the audio playback when volume was adjusted. Sometimes the stream volume change event notification did not stop when the volume change on the plug-in side was requested to THETA by Intent.

A good one to notice here was the audio playback during volume change. When notification sound by plug-in volume change was requested with Intent on THETA side, stream-volume’s volume change event notice did not stop.

The reason is that the shutter volume is adjusted to the stream volume when the stream volume changes in order to capture the AVRCP volume change event.

When playing audio on THETA side, adjust the stream volume to the shutter volume before performing audio playback.

For this reason, the plug-in side shutter volume change and THETA side stream volume change ended up adjusting with a time lag endlessly.

As a countermeasure, AudioManager was prepared on the plug-in side and audio playback was performed. As a result, stream volume adjustment and shutter volume adjustment were synced and the problem was solved.

About the RICOH THETA Plug-in Partner Program

If you are interested in THETA plug-in development, please register for the partner program!

Please be aware that the THETA with its serial number registered with the program will no longer be eligible for standard end-user support.

For detailed information regarding the partner program please see here.

The registration form is here.