I am trying to help a musician friend get his excellent music more widely known by making some visuals for him for music videos he can post online. We recorded one song with three different systems last week: stereo audio in Ableton Live 12, 4K iPhone video with sound and Kinect V2 streams in Touch Designer. We used a head sync clap with slate and to be safe a tail sync clap after his song.
I know how to slip sync in Final Cut Pro between audio and video by aligning claps. But in searching forum I found this note by Ben Forest I found intriguing:
As a challenge to myself to get as comfortable as possible in Touchdesigner, I decided to do 100% of the video production in TD. The “screen” behind us shows an audio synced montage of the 5 scenes of live visuals I built for our performance which were rendered and exported in real time. The kinect geometry of us performing was captured with a Kinect v2 in touch. I then created the environment you see in the video, integrating the synced kinect footage and live visuals footage which was rendered, again in touch (not in real time though).
I also did some searching on net and learned the audio and video recording are asynchronous in Azure Kinect (which I don’t have) and get out of sync a few seconds in a long recording (which was my rationale for recording a tail clap too). Presumably Kinect V2 is no better - probably worse.
We are not doing this live yet in real time, and in past I have used delays and caches on either audio or video to get td Movie File Out recordings in perfect sync with each other for videos.
I would also like to do all the syncing in Touch Designer like Ben did but was not able to see any of his older ARIADNE videos or techniques. My questions for all TD gurus are:
Q1: What is a suggested workflow or approach in TouchDesigner to actually view and sync the Kinect TOP and CHOP channels (depth, color, player index etc) with Kinect audio, iphone video and audio, and Ableton Live Audio?
Q2: I have experimented a bit in the past with trail chops, Movie File In TOP trims etc but am still very green with those. Nothing as simple as lining up clap visuals and sound waveforms in Final Cut Pro. Should I just do all the syncing in FCP or can TD do it somehow cleverly with enough work?
Thanks for any advice. Not urgent but important.