You should be focusing on the amount of time it takes it do a frame when doing these comparisons.
The difference between 600fps and 150 fps is 5ms per frame (1.66 → 6.66). The difference between 60 fps and 40 fps is 9 ms per frame, (16.66 ms → 25 ms). So the difference is actually reverse what you think, the 60->40 is a bigger drop than 600->150.