Hi…
I’m working on a project that will combine :
1 Hand drawn images, which will be processed by a TouchDesigner. then the values I will send to bitwig.
2 Music, for the creation and control of which I will need values that I would like to get when processing drawings (i.e. to turn a drawing into music).
Since this project is conceived in the format of audiovisual performance, we would like to process images in real time.I really like the capabilities of the GSL language and the fact that (I read, I may be wrong) it is less resource-intensive (when analyzing an image (roughly speaking)) than the same Python.
I am currently assembling a setup (with nodes) based on FFT. The main task is to create a tool that will “sonify” an image. But I have not come to truly satisfying results. I feel as if I am missing something and cannot fully understand it.
Maybe some materials on this topic?
I would be very grateful for any information? and perhaps such a tool already exists, then I would be glad to know about it too
sonification of images needs an underlying concept of what attribute of an image you would like to use for sonification.
Is it the average color of the whole image or all color values of a single row of pixels? What element in the image might have most numeric meaning to influence a waveform? You could be scanning through the image horizontally, interpreting certain color values as notes on a musical scale for example.
So first I would look at it from the conceptual part and then investigate the techniques involved.
I would like to analyze the process of drawing:
Appearance of a point, line, spot( how big or small the spot is). intersection of lines.
Most likely, I want to subdivide the canvas into sectors (maybe, which will be in motion).
I definitely don’t want to extract notes from the drawing.
In the process of analysis I will send the data to bitwig to create an impulse and change the characteristics of the wave and instruments.
I want to transfer the movement of the line and the appearance of the point in the drawing to the transformation of the sound wave and its characteristics.
Thank you very much for your reply, Marcus.
I am immensely grateful
detecting this kind of features is not a trivial task and I can only give you pointers. Intersecting lines in a image might be possible to identify with openCV using the HoughLines - maybe a first video to watch in this direction would be
For size aproximation this video might give an idea:
All of this requires a bit of programming in TouchDesigner using it’s python interface. Can’t think of any build in - ready to go solution right now.
