Thanks a lot for providing me with the Scout files — that's the fastest way for me to analyze this.
And it is just as I expected. When you look at the v2.1 Scout file, you see that the calls on "ByteArray" and "VertexData" are the most expensive ones. If you add up the time it spends in these calls, it becomes clear that this is where the performance is lost.
That said, I've got two ideas that should help fix this problem.
A simple one: if your MovieClips all use 60 fps, try out if that's really necessary. I don't know what exactly you are displaying, but if you reduce the number of frames of the clips (only the movie clips, not the game FPS) to 30 or 20, this might still look great, because you're probably moving the objects around (do you?) and that still happens with 60 fps.
Another one: create a new MovieClip class that works a little differently than the current one. My implementation is basically an Image that changes the texture over time.
Instead, create a Sprite that contains one image for each animation frame, with all but one set to "visible = false". That way, the texture coordinates don't need to be modified all the time (far less ByteArray writing).
The downside: lots of "ADDED" and "REMOVED" events whenever you add such a clip to the stage. There might be a way to get rid of that, too, though — via a display object that forwards the rendering calls to another object, like this:
public function render(painter:Painter):void
I'd have to play around with that myself, to see what would be the perfect base class for that.
In any case — start with the simpler variants and see if that helps. You can then move to the more challenging versions.
Is it possible to have an option like vertexData.useVector=true;
for use when needed instead of the bytearray?
No, that's unfortunately not possible. All the classes that do any rendering expect to get a ByteArray. ByteArrays have other advantages, like being able to store data other than floating points with a vertex, or to use a lower number of bits per attribute. That all helps in many scenarios — so it was important to make that change. It's just a pity that writing and reading is so slow.