OpenCV background subtraction

General TouchDesigner Discussion
1

I have an AMD GPU so I am unfortunately unable to make use of the Nvidia Background TOP. However, I have used OpenCV a fair amount in the past, so I figured I would use OpenCV in a script TOP to produce a foreground mask.

Here is the source code of the script of the TOP:

import numpy as np
import cv2

fgbg = cv2.createBackgroundSubtractorMOG2()

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return

def onCook(scriptOp):
	input = scriptOp.inputs[0].numpyArray(delayed=True)

	if not input is None:
		fgMask = fgbg.apply(input, learningRate=0.01)
		
		# Using this next line instead of the other two lines below works
		# and correctly displays a grayscale version of my incoming video feed.
		#scriptOp.copyNumpyArray(cv2.cvtColor(cv2.cvtColor(input, cv2.COLOR_BGR2GRAY), cv2.COLOR_GRAY2BGR))
		
		colorizedMask = cv2.cvtColor(fgMask, cv2.COLOR_GRAY2BGR)
		scriptOp.copyNumpyArray(colorizedMask)

	return

This should in theory produce a black image with white pixels where the foreground has been detected. Instead, I simply get all black pixels. If I comment out the last two lines and uncomment the commented code, I do see a grayscale version of my input video, so I’m not sure what I’m doing wrong here.

My only guess is maybe it has to do with instantiating fgbg in global scope? Is it alright for me to instantiate fgbg in global scope or is there some context I should keep that in when running in touch designer? The object needs to persist across frames… I’m not sure what the lifecycle of this script looks like.

2

Some further context:

I am able to run the following code (outside of Touch Designer):

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

fgbg = cv2.createBackgroundSubtractorMOG2()

while(1):
    ret, frame = cap.read()

    fgmask = fgbg.apply(frame)

    cv2.imshow('frame',fgmask)
    k = cv2.waitKey(30) & 0xff
    if k == 27:
        break

cap.release()
cv2.destroyAllWindows()

and get the desired white mask on black background… So I don’t think it’s an openCV problem… it seems more like I’m doing something wrong w.r.t. TouchDesigner.

3

Hi @lyet

Apologies for the delay.

I’m not an OpenCV genius but I think this should get you started:
OpenCVMog.toe (4.7 KB)

The main thing I did is move the instance of the MOG2 Substractor to an extension, and multiply my array values by 255, and converted to gray before sending to the apply method of the substractor.

Best,
Michel

1 Like
4

@JetXS thank you so much! I am definitely going to read up on Extensions to get a better idea of what’s going on there. I am so happy this is working now.