Hello all,
I am trying to rotate my instances on the y axis so the front face will face the camera while staying upright. If I tranform my camera up, for example, I don’t want my instance to rotate on it’s local x axis.
The normals will also need to be deformed.

I got this function from an old post as a starting point -

mat3 calcLookAtMatrix(in vec3 camPosition, in vec3 camTarget, in float roll) {

This currently just rotates the geometry towards the camera entirely and has an issue while the vectors are colinear as malcom points out in that post.
This is my attempt as using that function-

void main()

{

mat3 lookAtMatrix = calcLookAtMatrix(P,uCamPosition.xyz, 0);
mat3 lookAtMatrixN = calcLookAtMatrix(N,uCamPosition.xyz, 0);
vec3 newPosition = vec3(0.0);
newPosition = lookAtMatrix * P;
vec3 newNormals= lookAtMatrixN * N;
// First deform the vertex and normal
// TDDeform always returns values in world space
vec4 worldSpacePos = TDDeform(newPosition);
vec3 uvUnwrapCoord = TDInstanceTexCoord(TDUVUnwrapCoord());
gl_Position = TDWorldToProj(worldSpacePos, uvUnwrapCoord);

I’ve attached a small example of what I want- rotate my camera and have everything stay upright.

To create a rotation matrix you’'ll need 3 vectors point right, up and forward. In your case you want to keep the up vector pointing to vec3(0,1,0). The forward vector would be the normalized vector from the camera position to the worldspace vertex position. Though this forward vector should be perpendicular to the up vector. To make sure there is 90 degrees between the vectors you can use the cross product trick you’re doing in the calcLookAtMatrix().

P is the object-space vertex position. So the position relativily to the SOP origin, it’s not the worldspace position. The world space position is calculated in TDDeform() that’s reading the instance parameters and transforming the P point to workspace.
There are also some other functions like TDInstanceTranslate() to retrieve the current instance worldspace position.

So to conclude it should be something like:

vec3 cameraPos = uTDMats[cameraIndex].camInverse[3].xyz;
vec3 fwd = normalize(TDInstanceTranslate() - cameraPos);
vec3 up = vec3(0,1,0);
vec3 right = normalize(cross(fwd, up));
fwd = normalize(cross(up, right)); // updating fwd to make sure its perpendicular to both up and right
mat3 rotationMatrix = mat3(right, up, fwd);
vec3 newPosition = m * P;
vec3 newNormal = m * N;
vec4 worldSpacePos = TDDeform(newPosition);

Note that there is a predefined uTDMats[cameraIndex].camInverse 4x4 matrix. You can use this to find the camera position (so you dont have to pass it through an uniform yourself)

I think the normals are flipped because of the TDFrontFacing() call in the pixel shader. You can test if the normals are ok by doing something like: oFragColor[0] = vec4(normal,1);return; after ‘normal’ is defined. Then all faces upwards should be green, towards you blue, and to the right red.
Though bit tricky to see since the object is rotating towards the camera

I hope all is well in camera land,
I would like to achieve the exact thing you mentioned above using the regular Geometry instance params. I am having a hard time figuring out what values to get in there. There are 6 fields and a Up vector to select to do so in the Geo’s second instance tab.
Hoping there is someone that has been there before to help with the order of operations, cheers! P