That is my biggest problem.
I don't know how to do this without a very low framerate.
That's why I said it's not an easy task
I'd try to build a system based on so-called loading jobs, you just create a queue of these jobs containing a description of what needs to be loaded. You should be able to split these jobs up into smaller jobs so you can adjust them to your time budget as to avoid any noticeable lag. You could eventually do some kind of priority-based sorting so crucial elements of the scene get loaded first, but this could get nasty if done naively, so really plan this out