Pixel to MIDI.toe optimization

Hellow ,

hope to get some hints and help for this Project.
Its made for converting lights in night shots of buildings , stars or train-rides to MIDI - Notes.
Its wokin now but i have some questions and problems…for shure :unamused:
I prepared and cleaned the Project to talk about with you :slight_smile:

MIDI goes out to LoopMIDI Device(Win10) and then to ableton live. A midi track with loopmidi port Ch1 as Input and a piano as instrument plays the incoming notes.the notes are scaled with midi effect “scale” in ableton.

Questions :

  1. I loose Notes in the combination of countCHOP to eventCHOP , i think i use it in a wrong way? the problem is that i get repeating glitchy MIDI notes wihout decreasing the notes with this chops. is there any other way to increase midi notes played at the same time ?

2.also i convert the video to 127x127 pixels ( 127 is midi maximum value ),bigger resolution freezes the cpu.is there a more cpu friendly version to convert the pixels to a table ?
or no table at all ?

  1. in the MUTEK sequencer ( https://www.youtube.com/watch?v=fn7U6VFN1_E )
    the data is writen to a table,this is fancy,is it a better way to use the pixel values ?
    i tried to change the sequencer to my pixel-patch,but failed.

  2. i level the incoming picture to get more defined light points.is there an other way to redefine the lights/points ? blob tracking ? ( no license ) tensor flow ?opencv ?
    i have tensorflow in TD installed and working,but no clue how to step further.

Here is the Project on Github

screenshot.tiff (412 KB)
PixelToMusic.15.toe (23.6 KB)

Hi there,

I had a look at your network; thanks for sharing it, but I’m sorry I won’t be able to help with specific tips as the method you are using follows a workflow that I am not very familiar with.

More broadly speaking, in terms of optimising your network, you can use two tools In TD: Probe and Performance Monitor.
Probe is in Palette > Tools, whereas Performance Monitor is under dialogs.

As Elburz mentions in his amazing workshop Project Optimisation, “optimising is a game of millimetres”; meaning that there isn’t one simple/single step solution to optimising a network. but instead, optimising it’s an analytical approach that will help you saving handful of milliseconds here and there across many operators.
Also, minimising the usage or repositioning in the network those operators that always cook (like a lag CHOP for example), or re-thinking the overall order and signal flow of your operators helps a lot.

I hope that helps

I try to understand your network but its very difficult to distinguish the flow and perhaps a little bit of cleaning and using subnetwork will help to understand. Renaming the operators to show their real function can help us (and you too) follow the process. Clearly separate the process: command input, image transforming, image analysis will also help a lot.
But to my understanding, transforming TOP to CHOP and then analyse the flow its a very demanding task and I would try to do it in GPU instead of CPU with GLSL TOP. I would try to make all the analysis in the shader, outputting a 127 x 1 image for the 127 midi note with r for velocity and g, b and a for various parameters. Only there I would analyze the image to obtain the midi. Its not an easy task but I think is doable.
At the moment I just begin a collaboration with a musician who was coming with a video process linked to music (Max/Msp sending OSC) made with Processing and the image was limited to 64 x 300 particules outputted on 2400 x 300. Rewriting the process with GPU, my first test on the same machine made possible 64 x 10000 particules on 9600 x 1200. Its difficult to have exactly the same result but the efficiency difference is huge.
Hope that helps, Jacques