I've delved into the profilers with the min plugins listed above and I tracked it to 100% being ATUM.
Here is two screenshots of the profiler.
https://pasteboard.co/JvRTADZ.png
https://pasteboard.co/JvRTJ3I.png
As you can see, it's adding 29 times more execution time for the part of the loading which renders rows when it's enabled. Note that with profiler enabled the entire execution time was 15 secs without multi-inventory, and 47.85 secs with it enabled, this is normal as profilers add a lot of time to execution. This added time however is still going to be faithful to finding the problem. This all tracks well to everything I have experienced. Each row is adding time to a significant degree and it adds up quickly. Please advise on if you will fix this or not as it is a dealbreaker for us, we'll have 270,000 products and I'm sure the scaling will get worse with not only that but when we add more than a few inventories as well.