088: Map an object with CamSchnappr

Note: The latest CamSchnappr moved to the palette starting with TouchDesigner version 088.50180. For older builds the last working version is attached at the bottom of this post.

CamSchnappr has its own wiki page.

CamSchnappr is a interactive mapping application entirely inspired by MAPAMOK, created by Kyle McDonald at the YCAM Interlab. With Kyle publishing the source code for MAPAMOK, we had a chance to look at it and convert it into TouchDesigner 088. For the original tool and source have a look here: https://github.com/YCAMInterlab/ProCamToolkit/wiki/mapamok-%28English%29

Kyle describes the significance of the tool as the use of openCVs cameraCalibrate to calibrate a projector via a model of the to be mapped structure instead of using a checkerboard. With TouchDesigner 088’s integration of Python, we were able to implement the same functionality into a standalone tool which lets you map complex structures in a small amount of time.

[vimeo]http://www.vimeo.com/57081301[/vimeo]

What it does:
If you have a physical 3D structure plus a virtual 3D model of that structure, you will be able to project a rendered virtual model onto that physical structure perfectly-aligned. You will be selecting points of your 3D Model and align them with the real world positions as you see them on the projector output.

Using your 6 or more alignment points, CamSchnappr will compute the position, rotation, scale and viewing angle of a specified TouchDesigner Camera component used for the rendering. Internally, an openCV function called calibrateCamera will be run and the “intrinsics and extrinsics” of the camera will be calculated and stored in the Camera component.

What you need:

  • TouchDesigner 088
  • a recent version of CamSchnappr
  • a projector
  • a physical structure to project on
  • a 3D Model of the to-be-mapped structure

What’s in the package:
The attached zip file contains:

  • a recent version of CamSchnappr complete with an integrated test model and render setup
  • the documentation as pdf file
  • a cutout plan of the integrated model as pdf file

Interface:
Open the latest CamSchnappr.toe, right-click on the CamSchnappr Component and select “View”:

basicInterface.jpg
The initial interface will show an interactive viewport with a default geometry and some input elements at the bottom.

The Viewport:
You can interact with the viewport as follows:

  • left-mouse click and drag to tumble
  • middle-mouse click and drag to zoom
  • right-mouse click and drag to pan
  • h to home the geometry
  • select points by left-mouse clicking on the blue spheres (blue sphere turns yellow, inactive selected points are orange)
  • make selected points active by left-mouse clicking on orange spheres
  • de-select points by right-mouse clicking onto spheres
  • tab to cycle through selected points

When you select points a little flag with the point number will be displayed:

selectedPointsViewport.jpg
Below the viewport are separate controls to specify

an external Camera COMP

  • Initially an internal default camera will be used for the mapping, yet normally you would want to use your own Camera COMP. Be advised that all of your camera settings will be overwritten by the application.

    an external Render TOP
  • You will want to also drag in your own Render TOP to be used by the CamSchnappr. The application will change the resolution of the your Render TOP to the projectors output resolution.

an external SOP

  • Drag your own SOP (surface operator) into this field. From testing on a decent machine good results were achieved with geometries of up to 200 points but a considerable slowdown when using the viewport occurred. Yet, if you minimize the number of points of the geometry you want to map, it all should be fine. In the end the result is saved in the camera and you can disable or delete the CamSchnappr Component entirely.

an output Monitor selector

  • select the output Monitor number here. Depending on which Monitor you choose, the resolution of your specified render will be adjusted to match that resolution.

  • a button to enable the fullscreen Monitor output and Mapping Viewport.

The Mapping Viewport
Once you selected the correct output Monitor on the Monitor Selector, click the Fullscreen Output Button

mapperInterface.jpg
The interface is quite simple:

  • a pink crosshair indicates your current cursor position on the screen
  • a blue crosshair with number indicates an inactive point previously selected to be mapped in the main Viewport
  • a yellow crosshair with number indicates an active point previously selected to be mapped in the main viewport

The numbers beside each point correspond to the numbers of the selected points in the main viewport

You can drag points to their real-world position on the screen or use the arrow keys on the keyboard. For faster movement, use shift+arrowkey to move points.

Tab will cycle through the available points.

A faint outline will show the virtual model representation.

Workflow

  • Create a simple render setup with the model of the object you want to map onto.
  • Right-click on CamSchnappr and select view.
  • Drag your Camera COMP onto the Camera COMP field at the bottom of the CamSchnappr Viewport.
  • Drag your Render TOP to onto the Render TOP field at the bottom of the CamSchnappr Viewport.
  • Drag the SOP containing the geometry of the to be mapped object onto the Geometry SOP field at the bottom of the CamSchnappr Viewport.
  • Select the number of the output your Projector is connected to via the Monitor selector at the bottom of the CamSchnappr Viewport.
  • Open the Mapping Viewport by clicking the open projection output button, if somehow the output now overlays your main viewport, just hit escape and select a different Monitor Output number via the Monitor selector at the bottom of the CamSchnappr Viewport.
  • In the main viewport align the geometry to the camera so it’s similar to what the projector sees.
  • Select a point in the main viewport and move it in the mapper viewport to the corresponding position on the real world object.
  • Repeat the last step for at least 6 points total.
  • After you have aligned 6 points, the camera should be calibrated and you should see the projection mapped onto the object.
  • If everything is correct you can disable the CamSchnappr or delete it. The Camera Calibration values are saved inside your Camera COMP as 2 Table DATs.
  • CamSchnappr can be run on multiple render setups making it possible to calibrate multiple projectors or multiple objects.

mapperInterfaceMapped.jpg
CamSchnappr.zip (283 KB)

Awesome. Thx markus!!!

Sweet!!! now we don’t have to build our own. Thanks for the extremely useful component.

cheers
Keith

YAY! it’s like my dreams came true! :smiley:

Wow, very nice work! Markus strikes again!

this made my day, thanks for putting the work in!

just tested out the isam model with a virtual scenario, works like a charm, just had to reduce the map point model so it had only the points i’d consider using to keep the UI speed up

very excellent :slight_smile:

Legend! that takes the sting outta mapamok

chrs!

Sweet! Great job man. This is amazing!

Thank you for using your amazing skills to help us lesser mortals Markus. Your awesome as usual.

Hi Peter,

out of curiosity, how many points did the original model have?

cheers
Markus

Hi archo-p,
Any chance you can share some of your insight on your workflow to reduce points in the model?

Currently I am using meshmixer to remove unused geometry in my 3D scene, and netfabb studio to repair and reduce the number of polygons in my mesh. This greatly helps but the scenes I am practicing on are still barely manageable. Any advice is greatly appreciated :slight_smile:

Thanks!

Hi emintzer,

how many points does your test scene have? Would like to get a sense of what size of models we are dealing with…

would a plausible solution be to use the Delete SOP to remove points you possibly won’t be using for the mapping process?

cheers
Markus

Markus,

The original mapping model was 240 points, after consolidating, and only dealing with the faces to be projected on, so as few as possible really given the complexity.

Testing out camSchnapper, I just isolated the few primitives that had points I knew would be useful for targetting, down to 104 total points. I’ve been doing this for other projects too, pre-camschnapper, in order to only give my self points that would really work (ie. corner points of sliced curved surfaces and such, where an internal surface point would be of no use).

It actually still ran decently leaving all 240 points, but definitely sped up and more straight forward with less than half as many. I’m sure for stuff like car mapping, people will really have to isolate those points!

Emintzer,

As Markus said, I think you can literally just go in and pick the points that you care about. I isolated certain primitives with good points in houdini and grouped them, so i could just delete non-selected. As long as the points/prims you are isolating remain exactly where they are on the full model it doesn’t really matter how chopped up and disconnected that point picking geometry becomes, as long as you yourself know which points correspond to the full model. Does that make sense?

-Peter

Markus,
just about 18,000 points! lol :stuck_out_tongue:

Peter,
Thank you for the tip. This makes sense and worked great! In fact, I found it easier to view points in TD and use the delete SOP to isolate my points as opposed to using 3rd party software as I first tried.

Oh im going to have fun with this. Thanks both^ :slight_smile:

Awesome!

First, thx for this! Use it a couple of weeks and verry happy with it…

But I have problems using CamSchnappr in aktuell TouchDesigner versions (since 6360). It dont catch all points from given SOP and the flags with the Numbers are not working correctly…

Also the calibration process seems broken… When i look to the network, everything seems ok, nor errors or something. Any solutions?

fHainer

P.S. Sorry for my english, i know it could be better :confused:

Looks like CamShnappr is partially broken in build 6360 (I’m not sure about build 6580 because it is crashing projects from older builds).

Basically I’ve found two things so far:

  • not all the points are showing for the selected SOP
  • there is a rectangle artifact appearing in the render view similar that looks the same as the grid that appears when a GLSL shader is not working.

I’ve tracked down where the points are being rendered be there doesn’t seem to be anything wrong except they aren’t displaying.

The point problem looks like it’s happening in the sort CHOP at with path - CamSchnappr/pointDisplayPicking/sort1 . It seems to many points are being removed. I’ve disabled it to see how that will work for now.

It looks like it’s more than points just not showing up. It’s not making a matrix at all for the camera.I haven’t been able to fully track it down but it looks like it has something to do with instancing. I’m going to roll back to an earlier version of Touch just for this project for now but I think something is broken in the geometry comp instancing section.

cheers
Keith