Converting a point cloud file to openEXR will vastly improve performance. As an example, a 40 million point xyz file that was running at 11 fps jumped to 25 fps when using the openEXR file.
Intel(R) Core™ i7-10750H CPU @ 2.60GHz 2.59 GHz
Nvidia RTX 3060