[extension] ColorTransformStyle

Romain

Hello everyone,

I'm working with Starling a lot recently, studying how we could port one of our game from the Flash display list to Starling. We use ColorTransforms extensively and so we need a solid Starling style (and not a filter, as we need to batch a lot of meshes together). I didn't find anything like this, so I've started coding a little something, based on Daniel's ColorOffsetStyle (it was very helpful thanks!)

The work is mostly done, but there are a couple of things to do:

Right now it doesn't support alpha offset, after spending a lot of time trying to do it, I'm still failing to properly replicate it. I'll continue to look into it, but if you have some ideas, let me know!
Optimizations, I need to spend some time to stress test it and see where optimizations can done (I'll probably pack offsets into a single float for example)

You'll find the style here:
https://github.com/rthery/starling-colortransform/blob/master/src/starling/extensions/ColorTransformStyle.as

The repository itself is mostly to have a demo to debug and compare the two ColorTransforms, you can play with it online here:
https://rthery.github.io/starling-colortransform/

I hope it can be useful to some of you 🙂

Daniel

Great work, Romain, very well done! 😃
I'm always happy to see when somebody dives into MeshStyle creation. =)

The problem with the alpha offset will probably have something to do with premultiplied alpha, but I guess you already know about that. When you do the "tex()" operation, you already get RGB values with alpha premultiplied; maybe that's the problem. You might need to un-multiply for your calculations and then re-apply it at the very end. I'm doing that here in the ColorMatrixFilter - perhaps you could mimic that.

I didn't check the AGAL code in detail, though, so it might be something else, too.

Romain

Thank you for your message Daniel, it was indeed due to the PMA. I initially tried to use the same technique you used in the ColorMatrixFilter, but I was missing something.

Just realized today after reading your message that the color attribute has also PMA, so when I was doing the multiplication and the offset at the same time, the colors were wrong. I've reverted the PMA from the color attribute, and now the results are much better!

Although there's one strange problem left, with the max operation to avoid the division by zero when undoing the PMA on the texels color. If I run that operation, fully transparent texels are affected by the alpha offset.
But if I omit this operation, those transparents texels are not affected, and we have the right results most of the time. (it randomly fails to ignore transparent texels and still apply the alpha offset to them sometimes)

So for now, I've decided to skip the max operation and make sure to apply the alpha offset only to non-transparent texels. Cf. https://github.com/rthery/starling-colortransform/commit/2aae8f5a7121dc5cb9e89e040ea7938dbeae44ef#diff-309fa331e3e6a88384f612cd120d39c6L215

I'm not sure if it's a bug with Stage3D or just a problem with my AGAL code (since I'm no expert, quite possibly the second option!)

In any case, the end result works well now 🙂

Daniel

Although there's one strange problem left, with the max operation to avoid the division by zero when undoing the PMA on the texels color. If I run that operation, fully transparent texels are affected by the alpha offset.

Hm ... I don't see an error here, either. But I've run into my share of strange phenomenons in shaders, too — so if it works without that check, I'd also just leave it out. 😉

So, congrats on getting this done, and thanks a lot for sharing the code with us, Romain!

One last thing: do you think you could add this to the list of Starling Extensions? That way, it would be much easier to find for other developers. This wiki page explains the process — it's really just a matter of a few minutes.

Thanks a lot in advance! 😃

Romain

Ah yes, I'll finish a couple of things on it and add it to the wiki page! Thanks again for your hindsight on this and more generally for a great library 😉

Romain

Alright, finally managed to take the time to setup the wiki page, http://wiki.starling-framework.org/extensions/colortransform

I had to unfortunately stop using the Mesh color bytes4 for the multipliers, since even though the docs say they are supposed to go from 0 to 1, in practice you can go up to 128 (and I quickly found out about it when trying the style in our game ahah). So for now I added another float4 for them.

There are still a few things I'd like to do. Things like reproducing the same behaviour when one modifies the alpha of the affected DisplayObject or its ColorTransform alphaMultiplier for example (I believe I can reproduce this by overriding the alpha property of the MeshEffect). I also started to look into improving performances by packing the multipliers and offsets into something else than float4.

JohnBlackburne

The 'alpha' of a Mesh is normally handled by the default fragment shader, along with the colour. As you have replaced that with your own more capable shader you don’t need to handle the mesh alpha - users could instead add it to the matrix, pre-multiplying it if they want to e.g. fade out a whole scene.

As for improving performance your data is going to be uploaded once per mesh, if that (it might be batched across multiple meshes). But fragment code is going to run across every pixel. I would err on the side of doing as much calculation as you can outside the shader, even if this means uploading 2 or 3 float4, and not packing it if it means you have to unpack it in shader code.

Romain

Thank you for your feedback John!

I didn't manage to do what I had hoped for the alpha, so I guess I'll let it rest for a while.

Regarding performances, the scene we have to render is relatively complex with potentially a large amount of color transformed meshes being updated every frame (because of an animation system similar to Daniel's recent release. Since we are CPU-bound, I had hoped that deporting some of the work to the GPU could help (because for most players, their GPU will probably never work fully).
Although we also need to take into consideration Software rendering (for players with unsupported/no GPU), and this style is just a killer when rendering on the CPU, but packing would probably make it worse anyway for them.

JohnBlackburne

You are right that desktop Flash apps are rarely GPU bound, but they can be. Fill rate can be a particular issue if the app has a lot of depth, i.e. multiple layers, and a large window/can go fullscreen. Something to add into your testing perhaps, if it describes your app in any way.

As for software rendering, I would think very few PCs today will not run Stage3D apps on the GPU. The biggest problem is probably users that turn off hardware acceleration in their Flash preferences, who need to be encouraged to turn it on. I think Stage3D prints up a message if you try and run it with HW acceleration disabled.

If you really need to support users who can’t or won’t use HW acceleration, the best option is probably the classic API, which is coded to run only on the CPU. And Starling makes this easy with the way it closely matches the classic API.

Romain

Our game actually already runs the classic DisplayList, but we pushed it to its limits for a while now and it has always limited our development in areas like animations, as they are too CPU intensive. That's why we are looking into Starling.

We looked into supporting both rendering methods, but we didn't find it that simple (although I'd very happy to be proven wrong!). Neither flash.display nor starling.display uses interfaces, both use different Event system, some methods have the same name but not the same signature, etc. So at the end you have to duplicate all your code or use wrappers and tons of 'if'. That would just be too great a cost in term of maintainability for our code base. So we'd prefer to just embrace Starling fully, and try to find ways to keep the Software rendering still playable (even if it means those very few players would end up with a slightly worst graphic experience than currently, it would still largely benefit the vast majority of our players for the years to come).
I could probably start a new thread about this, show what we did so far and discuss it further if you're interested 🙂