Took me some digging to find this comment in the FragmentFilter class, which explains my issue of getting an off scale image result when i draw a filter to a RenderTexture with a custom matrix.
Basically this call:
My work around was to call that in each render call:
_imageWithFilter.texture = null;
_imageWithFilter.texture = originalRenderTexture;
After updating to the latest Starling, that workaround did not work anymore because in the latest version of the BlurFilter this got set to true:
this.maintainResolutionAcrossPasses = true;
By setting maintainResolutionAcrossPasses back to false again, my hack works again.
The downside is, that setting the texture to null and back to the original consumes some performance, since all the vertex setup is done every frame twice.
Is there a better solution to that?