Yes, it's a tricky problem with a general curved surface. Far easier to do if you can decompose it into nicely curved parts, such as sections that can be approximated by circles and straight lines, which have simple vector methods.
In the more general case there's no simple solution. In fact there's no analytic solution, even for seemingly straightforward shapes such as an ellipse colliding with a circle. Instead you have to use numeric approximation methods, to find closest points and then resolve any collisions.
Not the answer you were looking for, but it might give you some pointers how to proceed. It sounds like a potentially quite involved task that would take a lot of someone's time, and might never work exactly as you want. You might be better off with an off the shelf solution such as Box2D which you can experiement with and tweak yourself.