When you call, say,
adjustBrightness(0.1) on a filter, that method will internally create a matrix that will make the image 10% brighter, and it will multiply the current matrix with that value. So when you call that same method multiple times, the brightness will rise each time.
So, what you need to do in your case is store your desired values for brightness, contrast and hue in separate variables; then, whenever one of these settings changes,
reset() the colorMatrixFilter and call the three adjustment methods one after another, with the respective values. That should do the trick!