Combining 3D geometry with 2D image or video for AR effect

[art3mis]

Trying to combine 3D geometry with 2D image or video to create a sort of AR effect. Thought I was close using RenderPass but no luck so far. The logic would be something like 'get z depth of geometry and if the value is greater than foo, put it behind the image( ie mask it) In other words, wrap the torus around the banana.

torusAroundBanana.toe (5.6 KB)

[lucasm]

It’s possible to do this for your exact scenario above fairly simply using a glsl top. ( there are some node only ways too ). The basic idea I think is to first create two buffers/tops, buffer A where the torus is on top of the banana, and buffer B where the banana is on top of the torus.

Then, in a third buffer/top, blend between those two inputs using the True/False result of testing the depths of the two buffers.

In the case of 3d renders, the depth buffer is there for you already, but in your case where you have a 2d image, there is no depth - so depth must be assigned some how.

I passed in a constant value for the banana depth that was somewhere in the middle of the torus.

torusAroundBanana.1.toe (6.5 KB)

Below is the glsl code in the toe file:

uniform float bananaDepth;

vec4 Over_Composite(vec4 top, vec4 bot){
	vec4 col = vec4(0);
	col.rgb = (top.rgb*top.a) + (bot.rgb*(1-top.a));
	col.a = (top.a) + (bot.a*(1-top.a));
	return col;
}


out vec4 fragColor;
void main()
{
	vec4 threeDRender = texture(sTD2DInputs[0], vUV.st);
	vec4 twoDRender = texture(sTD2DInputs[2], vUV.st);

	float ThreeDDepth = texture(sTD2DInputs[1], vUV.st).r;
	float TwoDDepth = bananaDepth;
	float depthMask = step( TwoDDepth , ThreeDDepth );

	vec4 color = vec4(0);

	vec4 ThreeD_OnTop = Over_Composite( threeDRender , twoDRender );
	vec4 TwoD_OnTop = Over_Composite( twoDRender , threeDRender );

	color = mix( ThreeD_OnTop , TwoD_OnTop , depthMask );


	fragColor = TDOutputSwizzle(color);
}

[art3mis]

Very cool. Thanks for sharing.

[atsonic]

@lucasm It’s so cool!