Environment Light + PBR with height map = flat texture

I get this flat texture when setting a height map on a PBR from certain perspectives, like projections of a sphere from the environment light. No setting seems to get rid of it other than disabling height map which doesn’t work for what I want to do. Any ideas or workaround?

EDIT:
It also happens with phongMAT and ambient light