I think the key issue is there are two things going on, compilation and linking. Source code is compiled into an intermediate form such as bytecode or native assembly. This produces lots of files which are then linked into a final executable.
The difference is important as compilation can be done in parallel, on however many cores you have, and normally scales linearly. So you can speed it up by throwing a faster CPU at it with more cores.
Linking though needs to look at all the compiled source at once, so can't easily be done in parallel. It also does not scale as well, as the bigger your app the more there is to check to resolve symbols for linking. RAM is more important than CPU speed for that.
And with lots of ANEs you are mostly linking. They are pre-compiled, you are not compiling them, just linking them into your app.
Because the scaling is non-linear removing even one ANE might make a big difference, but I guess they are all needed. If any ANEs just support other ANEs then combining their functionality together in a single ANE makes sense, or using larger ANEs which do multiple things over separate smaller ones that do individual tasks.