Would it be possible to control a PTZ USB webcam movement from TD?
I’m looking at Obsbot Tiny. Alternatively I’m considering a 360 camera that has enough resolution to zoom and pan on the input image, given they could stream it, which I’m not sure.
Any ideas?
We don’t have native support for controlling these cameras as far as I know, but I do see they have an SDK on their website.
You need an account to download the SDK, so I’m not entirely sure how it works but if there is a python API than that should be quite easy to control in TouchDesigner, and if it’s c++ based than you could theoretically make a custom c++ operator that could control the camera from inside TouchDesigner as well.
Hope that helps.
Thank you! I’ll investigate the SDK once It arrives. I may have some questions down the line since it’s a bit out of my scope.
Yeah, we’d be happy to answer any general questions about how to integrate a third party API into TouchDesigner. There’s also a chance someone else in the community has worked with these cameras before who may be able to give you more specific suggestions.
If you have a Pro license we can also potentially look into more direct support using pro support hours if you are interested.
It looks there could be an easier first approach through the OBSBOT Center app:
It has OSC control over connected devices.
The ideal scenario would be to skip the app entirely, so I’ll try my hand at the SDK.
I managed to get OSC control, however there’s a clear limitation: there’s only one address reading at a time for the gimbal movement, impossible to make smooth transitions as with their control software.
The preset selection works fine, you can get 3 predefined positions, although it transitions at maximum speed, forget about anything smooth with OSC (at least for now, I contacted the devs).
All in all I think this is a very good PTZ camera for its cost and brings some interesting uses with TD.
I wonder if the SDK would be an improvement over OSC control, no idea how to even start with it though. There’s this:
typedef struct
{
float roll_euler; /// roll euler angle
float pitch_euler; /// pitch euler angle
float yaw_euler; /// yaw euler angle
float roll_motor; /// roll motor angle
float pitch_motor; /// pitch motor angle
float yaw_motor; /// yaw motor angle
float roll_v; /// roll angular velocity
float pitch_v; /// pitch angular velocity
float yaw_v; /// yaw angular velocity
} AiGimbalStateInfo;
Edit: The SDK is too much for me to handle at the moment. I don’t have Pro support hours unfortunately. I’ll stick with OSC for now and see what comes next.
Thanks for the update and analysis. That’s useful information for the community.
It’s hard to say for sure whether the SDK would improve the behaviour compared with the OSC implementation. I’d need to spend some time with the documentation to say, and with some devices you don’t really know until you actually try it.
Just as a general reference point, it’s usually at least a few days worth of pro hours to add new device support but it can go longer depending on how complicated the SDK is and whether we run into problems.
Yes, I understand. There’s a sample cpp file that looks that could ease the work.
#include <iostream>
#include <vector>
#include <thread>
#include <codecvt>
#include <dev/devs.hpp>
using namespace std;
/// device sn list
std::vector<std::string> kDevs;
std::shared_ptr<Device> dev;
/// call when detect device connected or disconnected
void onDevChanged(std::string dev_sn, bool in_out, void *param)
{
cout << "Device sn: " << dev_sn << (in_out ? " Connected" : " DisConnected") << endl;
auto it = std::find(kDevs.begin(), kDevs.end(), dev_sn);
if (in_out)
{
if (it == kDevs.end())
{
kDevs.emplace_back(dev_sn);
}
}
else
{
if (it != kDevs.end())
{
kDevs.erase(it);
}
}
cout << "Device num: " << kDevs.size() << endl;
}
/// call when camera's status update
void onDevStatusUpdated(void *param, const void *data)
{
auto *status = static_cast<const Device::CameraStatus *>(data);
switch (dev->productType())
{
/// for tiny series
case ObsbotProdTiny:
case ObsbotProdTiny4k:
case ObsbotProdTiny2:
{
cout << dev->devName().c_str() << " status update:" << endl;
cout << "zoom value: " << status->tiny.zoom_ratio << endl;
cout << "ai mode: " << status->tiny.ai_mode << endl;
break;
}
/// for meet series
case ObsbotProdMeet:
case ObsbotProdMeet4k:
{
cout << dev->devName().c_str() << " status update:" << endl;
cout << "zoom value: " << status->meet.zoom_ratio << endl;
cout << "background mode: " << status->meet.bg_mode << endl;
break;
}
/// for tail air
case ObsbotProdTailAir:
{
cout << dev->devName().c_str() << " status update:" << endl;
cout << "zoom value: " << status->tail_air.digi_zoom_ratio << endl;
cout << "ai mode: " << status->tail_air.ai_type << endl;
break;
}
default:;
}
}
/// call when device event notify
void onDevEventNotify(void *param, int event_type, const void *result)
{
cout << "device event notify, event_type: " << event_type << endl;
}
/// call when file download finished
void onFileDownload(void *param, unsigned int file_type, int result)
{
cout << "file download callback, file_type: " << file_type << " result: " << result << endl;
}
std::string getProductNameByType(ObsbotProductType type)
{
std::string productName = "UnKnown";
switch (type)
{
case ObsbotProdTiny:
productName = "Tiny";
break;
case ObsbotProdTiny4k:
productName = "Tiny4K";
break;
case ObsbotProdMeet:
productName = "Meet";
break;
case ObsbotProdMeet4k:
productName = "Meet4K";
break;
case ObsbotProdMe:
productName = "Me";
break;
case ObsbotProdTailAir:
productName = "TailAir";
break;
case ObsbotProdTiny2:
productName = "Tiny2";
break;
case ObsbotProdHDMIBox:
productName = "HDMIBox";
break;
case ObsbotProdButt:
productName = "Butt";
break;
default:
break;
}
return productName;
}
int main(int argc, char **argv)
{
cout << "Hello World" << endl;
kDevs.clear();
/// register device changed callback
Devices::get().setDevChangedCallback(onDevChanged, nullptr);
std::this_thread::sleep_for(std::chrono::milliseconds(3000));
/// select the first device
int deviceIndex = 0;
string cmd;
cout << "please input command('h' to get command info): " << endl;
while (cin >> cmd)
{
if (cmd == "h")
{
cout << "==========================================" << endl;
cout << "q: quit!" << endl;
cout << "p: printf device info!" << endl;
cout << "s: select device!" << endl;
cout << "1 set status callback!" << endl;
cout << "2 set event notify callback!" << endl;
cout << "3 wakeup or sleep!" << endl;
cout << "4 control the gimbal to move to the specified angle!" << endl;
cout << "5 control the gimbal to move by the specified speed!" << endl;
cout << "6 set the boot initial position and zoom ratio and move to the preset position!"
<< endl;
cout << "7 set the preset position and move to the preset positions!" << endl;
cout << "8 set ai mode!" << endl;
cout << "9 cancel ai mode!" << endl;
cout << "10 set ai tracking type!" << endl;
cout << "11 set the absolute zoom level!" << endl;
cout << "12 set the absolute zoom level and speed!" << endl;
cout << "13 set fov of the camera!" << endl;
cout << "14 set media mode!" << endl;
cout << "15 set hdr!" << endl;
cout << "16 set face focus!" << endl;
cout << "17 set the manual focus value!" << endl;
cout << "18 set the white balance!" << endl;
cout << "19 start or stop taking photos!" << endl;
cout << "21 download file!" << endl;
cout << "==========================================" << endl;
cout << "please input command('h' to get command info): ";
continue;
}
if (cmd == "q")
{ exit(0); }
if (kDevs.empty())
{
cout << "No devices connected" << endl;
cout << "please input command('h' to get command info): ";
continue;
}
/// print device's info
if (cmd == "p")
{
int index = 0;
cout << "Current connected devices:" << endl;
auto dev_list = Devices::get().getDevList();
for (auto &item : dev_list)
{
cout << "---------------------------------------------------" << endl;
cout << "Device SN: " << item->devSn() << endl;
cout << " index: " << index++ << endl;
cout << " deviceName: " << item->devName().c_str() << endl;
cout << " deviceVersion: " << item->devVersion().c_str() << endl;
#ifdef _WIN32
if (item->devMode() == Device::DevModeUvc)
{
std::wstring_convert<std::codecvt_utf8<wchar_t>> converter;
cout << " videoDevPath: " << converter.to_bytes(item->videoDevPath()).c_str() << endl;
cout << " videoFriendlyName: " << converter.to_bytes(item->videoFriendlyName()).c_str() << endl;
cout << " audioDevPath: " << converter.to_bytes(item->audioDevPath()).c_str() << endl;
cout << " audioFriendlyName: " << converter.to_bytes(item->audioFriendlyName()).c_str() << endl;
}
#endif
std::string strProductType = getProductNameByType(item->productType());
cout << " product: " << strProductType << endl;
/// network mode
if (item->productType() == ObsbotProductType::ObsbotProdTailAir &&
item->devMode() == Device::DevModeNet)
{
cout << " deviceBluetoothMac: " << item->devBleMac().c_str() << endl;
cout << " deviceWifiMode: " << item->devWifiMode().c_str() << endl;
if (item->devWifiMode() == "station")
{
cout << " deviceWifiSsid: " << item->devWifiSsid().c_str() << endl;
cout << " deviceWiredIp: " << item->devWiredIp().c_str() << endl;
cout << " deviceWirelessIp: " << item->devWirelessIp().c_str() << endl;
}
}
}
cout << "please input command('h' to get command info): ";
continue;
}
/// select the first device
dev = Devices::get().getDevBySn(kDevs[deviceIndex]);
/// update selected device
if (cmd == "s")
{
cout << "Input the index of device:";
cin >> deviceIndex;
if (deviceIndex < 0 || deviceIndex >= kDevs.size())
{
cout << "Invalid device index, valid range: 0 ~ " << kDevs.size() - 1 << endl;
cout << "please input command('h' to get command info): ";
continue;
}
dev = Devices::get().getDevBySn(kDevs[deviceIndex]);
cout << "select the device: " << dev->devName().c_str() << endl;
cout << "please input command('h' to get command info): ";
continue;
}
/// control the device to do something
int cmd_code = atoi(cmd.c_str());
switch (cmd_code)
{
/// set status callback
case 1:
{
dev->setDevStatusCallbackFunc(onDevStatusUpdated, nullptr);
dev->enableDevStatusCallback(true);
break;
}
/// set event notify callback, only for tail air
case 2:
{
if (dev->productType() == ObsbotProdTailAir)
{
dev->setDevEventNotifyCallbackFunc(onDevEventNotify, nullptr);
}
break;
}
/// wakeup or sleep
case 3:
{
dev->cameraSetDevRunStatusR(Device::DevStatusRun);
break;
}
/// control the gimbal to move to the specified angle, only for tiny2 and tail air
case 4:
{
if (dev->productType() == ObsbotProdTiny2 || dev->productType() == ObsbotProdTailAir)
{
dev->aiSetGimbalMotorAngleR(0.0f, -45.0f, 90.0f);
}
break;
}
/// control the gimbal to move by the specified speed, the gimbal will be stop if the speed is 0
case 5:
{
dev->aiSetGimbalSpeedCtrlR(-45, 60, 60);
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
dev->aiSetGimbalSpeedCtrlR(0, 0, 0);
break;
}
/// set the boot initial position and zoom ratio and move to the preset position
case 6:
{
Device::PresetPosInfo BootPosPresetInfo;
BootPosPresetInfo.id = 0;
std::string BootPosPresetPosName = "BootPresetInfoZero";
memcpy(BootPosPresetInfo.name, BootPosPresetPosName.c_str(), BootPosPresetPosName.length());
BootPosPresetInfo.name_len = BootPosPresetPosName.length();
BootPosPresetInfo.zoom = 1.4;
BootPosPresetInfo.yaw = 45.0f;
BootPosPresetInfo.pitch = 0.0f;
BootPosPresetInfo.roll = 90.0f;
BootPosPresetInfo.roi_cx = 2.0;
BootPosPresetInfo.roi_cy = 2.0;
BootPosPresetInfo.roi_alpha = 2.0;
dev->aiSetGimbalBootPosR(BootPosPresetInfo);
std::this_thread::sleep_for(std::chrono::milliseconds(100));
dev->aiTrgGimbalBootPosR();
break;
}
/// set the preset position and move to the preset position
case 7:
{
Device::PresetPosInfo presetInfo;
presetInfo.id = 0;
std::string PresetPosName = "PresetInfoZero";
memcpy(presetInfo.name, PresetPosName.c_str(), PresetPosName.length());
presetInfo.name_len = PresetPosName.length();
presetInfo.zoom = 1.6;
presetInfo.yaw = 25.0f;
presetInfo.pitch = 45.0f;
presetInfo.roll = 60.0f;
presetInfo.roi_cx = 2.0;
presetInfo.roi_cy = 2.0;
presetInfo.roi_alpha = 2.0;
dev->aiAddGimbalPresetR(&presetInfo);
std::this_thread::sleep_for(std::chrono::milliseconds(100));
dev->aiTrgGimbalPresetR(presetInfo.id);
}
/// set ai mode
case 8:
{
if (dev->productType() == ObsbotProdTiny || dev->productType() == ObsbotProdTiny4k)
{
dev->aiSetTargetSelectR(true);
}
else if (dev->productType() == ObsbotProdTiny2)
{
dev->cameraSetAiModeU(Device::AiWorkModeHuman, Device::AiSubModeUpperBody);
}
else if (dev->productType() == ObsbotProdTailAir)
{
dev->aiSetAiTrackModeEnabledR(Device::AiTrackHumanNormal, true);
}
break;
}
/// cancel ai mode
case 9:
{
if (dev->productType() == ObsbotProdTiny || dev->productType() == ObsbotProdTiny4k)
{
dev->aiSetTargetSelectR(false);
}
else if (dev->productType() == ObsbotProdTiny2)
{
dev->cameraSetAiModeU(Device::AiWorkModeNone);
}
else if (dev->productType() == ObsbotProdTailAir)
{
int ai_type = dev->cameraStatus().tail_air.ai_type;
if (ai_type == 5)
{ dev->aiSetAiTrackModeEnabledR(Device::AiTrackGroup, false); }
else
{ dev->aiSetAiTrackModeEnabledR(Device::AiTrackNormal, false); }
}
break;
}
/// set ai tracking type
case 10:
{
dev->aiSetTrackingModeR(Device::AiVTrackStandard);
break;
}
/// set the absolute zoom level
case 11:
{
dev->cameraSetZoomAbsoluteR(1.5);
break;
}
/// set the absolute zoom level and speed
case 12:
{
dev->cameraSetZoomWithSpeedAbsoluteR(150, 6);
break;
}
/// set fov of the camera
case 13:
{
dev->cameraSetFovU(Device::FovType86);
break;
}
/// set media mode, only for meet and meet4K
case 14:
{
if (dev->productType() == ObsbotProdMeet || dev->productType() == ObsbotProdMeet4k)
{
dev->cameraSetMediaModeU(Device::MediaModeBackground);
dev->cameraSetBgModeU(Device::MediaBgModeReplace);
}
break;
}
/// set hdr
case 15:
{
dev->cameraSetWdrR(Device::DevWdrModeDol2TO1);
break;
}
/// set face focus
case 16:
{
dev->cameraSetFaceFocusR(true);
break;
}
/// set the manual focus value
case 17:
{
dev->cameraSetFocusAbsolute(50, false);
break;
}
/// set the white balance
case 18:
{
dev->cameraSetWhiteBalanceR(Device::DevWhiteBalanceAuto, 100);
break;
}
/// start or stop taking photos, only for tail air
case 19:
{
if (dev->productType() == ObsbotProdTailAir)
{
dev->cameraSetTakePhotosR(0, 0);
}
break;
}
/// download file, only for meet, meet4k and tiny2
case 21:
{
uint8_t enable = 2;
dev->aiGetLimitedZoneTrackAutoSelectR(enable);
printf("============================ enable: %u", enable);
if (dev->productType() == ObsbotProdMeet || dev->productType() == ObsbotProdMeet4k
|| dev->productType() == ObsbotProdTiny2)
{
std::string image_mini = "C:/obsbot/image";
std::string image = "C:/obsbot/image";
dev->setLocalResourcePath(image_mini, image, 0);
dev->setFileDownloadCallback(onFileDownload, nullptr);
dev->startFileDownloadAsync(Device::DownloadImage0);
}
break;
}
default:;
cout << "unknown command, please input 'h' to get command info" << endl;
}
cout << "please input command('h' to get command info): ";
}
return 0;
}
I believe the camera is worth to integrate on TD, given the price point and tracking that works quite well. I’ll think about commissioning the work if I get stuck with the SDK.
It seems that when limiting the control to pan or tilt independently, OSC can achieve a smooth enough movement.
Unfortunately there are always more devices then we have resources to support, but thanks for bringing this one to our attention and sharing the code. We’ll keep an eye on them as we make our plans.
Just adding info for those looking for other PTZ cameras. Panasonic has a nice line of PTZ that take FreeD protocol to control the PTZ, these work great with TouchDesigner. They are high-end so a bit pricey, but they work.
Check “Standard Model” and up on this page and search their specs for FreeD.
I’ve received feedback from OBSBOT devs:
OSC control is indeed different from the one in OBSBOT Center. As you mentioned, it only allows movement in one direction at a time. Our team will discuss and evaluate whether we can add more functionality to make it closer to the control in OBSBOT Center.
The SDK allows for smoother control and can bypass OBSBOT Center (since OBSBOT Center uses SDK for device control). If you have the capability, you can try it out.
I’ve tested having independent tilt and pan and the result is very positive, so horizontal and vertical movements are smooth, diagonal not really.
Also saw that there’s a zoom speed parameter (it’s on the same OSC address as zoon, so I have to set a OSC DAT with a CHOP Execute, I haven’t been able to do that with CHOPs alone), at minimum (1-11) the zoom transition is also very smooth.
As of now and to me, the only pros to pursue the SDK implementation are:
So I’m happy with the current state of things. Very good PTZ camera for the cost, specially seeing the Panasonic ones being x30 of its cost (although more feature packed). Thinking on getting another for live use.