Brainstorming - rigging/skinning

[decltype-auto]

I need to programmatically manipulate armature-rigged humanoid meshes like those on my ava pic¹. I don't see much reason to do this development clandestine and for my need only, because with a little conceptual purity planned right from the start that could well be generally useful others.

So let's think about it a little.

---

As far as I oversee it right now, that would best be incorporated by a policy-based design using a host class within the "scenegraph" subsystem, and policies that incorporate the skinning formula used.

One interesting design question is how to link those skinning policies to the importer plugin that reads data biased by the respective skinning formula. One could think about a dependent plugin that carries along the respective policy implementation code.

And of course, one should think how the rigging is internally stored, one could simply adopt e.g. the OGEX skinning formula as the internal representation and transform all (or just some?) other rigs into this, or let different skinning formulae internally coexist (I'd vote for the latter).

What do you think?

¹ those were created with makehuman.

[Squareys]

Hey @decltype-auto !

Have you seen #101 ? There has already gone a bit of thought into animation support in general, and skeletal animation as a part of that.

One interesting design question is how to link those skinning policies to the importer plugin that reads data biased by the respective skinning formula

I am not entirely sure what you mean. As far as I knew, the skinning data (bone weights and bones) is independent of the formula used for skinning? If so, one could either provide a simple enum for the skinning policy or just have the application choose that itself?

[decltype-auto]

Hi @Squareys!

Let's see if @mosra wants to join this topic back into #101, personally I think the complexity of the topic warrants this branching of the discussion.

---

But btw animation:
When some years ago i wrote a little private tool for using the (yet outdated) first versions of makehuman (that was sth around 0.9 and C++ in contrast to the current >= 1.x versions, which are pure python) with a scene composer OpenGL GUI, I found it most beneficial having made a c/s functionality available for python scripting; the glue between the C++ server side and the python client side I chose was CORBA (omniORB).

One big advantage of that was I could rapidly prototype any higher animation logic in python and could define animation rather arbitrarily before I formalized it and shoved it into the C++ server side.

Another advantage of that approach is that all implementation starts at a formal interface definition (in the IDL) and evolution of the system happens naturally when one spots the need for sth on the client side - thus for animations - and extends the formal interface definition accordingly and then - and only then - implements it on the C++-side.

---

Anyway - before we get into details about how to implement skinning - and morphing(!) - , we should think the what to implement, thus anyway aggregate the requirements to be met.

E.g.:
Skinning as understood in all the e.g. OpenGEX, COLLADA and X3D (especially H-Anim) specs as I get it maps single scalar weights per bone (more precisely per the parental joint of the bone) per vertex (or is isomorphic to that).

That is all nice and fair for low to medium LOD games and anims with focus on the plain limb movement, but not for HQ 3D and arbitrary SFX, because that scalar weighting is manifestly not physiological in terms of skin over active muscle tissue deformation - muscles get ticker and shorter if they contract and thinner and longer if they are stretched, and so is the skin tissue wrapping them.

Let's assume a humanoid rigged in T-Pose standing on the x/y plane at world origin looking along +z.

If e.g. the left arm exerts a pull-like motion around roughly the shoulder joint, the physiological degree of displacement of the vertices of the back and upper front is different than on a +rotation of the left shoulder around the axis pointing off along the humerus.
And even a plain pull is not constantly over its [~-π/2, ~+π/2] range affecting the same vertices by the same weight.

Thus one should e.g. consider whether the representation of skinning is - at least - to be parametrized by the tensor rank of the per-vertex weighting.

[mosra]

Sorry for the late reply -- the backlog from last week was quite huge so I was not able to get to this until now.

I wrote an animation braindump a few months ago, you might want to read it: https://gist.github.com/mosra/0ff8443d623cc0b09efb7a7299a0f692

I found it most beneficial having made a c/s functionality available for python scripting; the glue between the C++ server side and the python client side I chose was CORBA (omniORB).
[...]
One big advantage of that was I could rapidly prototype any higher animation logic in python and could define animation rather arbitrarily before I formalized it and shoved it into the C++ server side.

That braindump is based on experience from a previous project (touch-based animated UI) where we basically coded all our animation directly in C++ code -- and there I tried to make the prototyping as easy as possible. In my opinion any C++11(+) API, if done well, could be as easy to use as Python code. That's what I'm striving for with this engine, anyway :)

As far as I oversee it right now, that would best be incorporated by a policy-based design using a host class within the "scenegraph" subsystem, and policies that incorporate the skinning formula used.

My thought on this is that the animation system should not be tied to the scene graph (or to any other component) at all in order to keep the engine composable and allow the users to e.g. drop the builtin scenegraph implementation in favor of something better tailored for particular use case, if needed.

What I was planning to do in the engine is this (which is also partially explained in the braindump above and #101), again striving for a highly composable API where you can use all of it or just particular parts and implement the rest yourself:

• Ability to import animation data from external formats in various formats
• Ability to store them keyframe by keyframe in some data structure
• Ability to interpolate between keyframes using various builtin algorithms (linear, Bezier, ...) or with a user-supplied algorithm (for example, as you suggested, some volume preserving animation etc.)
• Ability to blend the animations together (using DLB or other...)
• Ability to compress the skinning weights in order to create an efficient in-memory representation for playing back the animations

The user is then responsible for putting all the things together in an app-specific way, integrating it into the scenegraph or whatever else, where the builtin scenegraph implementation again provides just a basic general support for driving animations (such as the SceneGraph::Animable or Timeline classes). Providing an all-in-one scenegraph "Skinnable" object that would do everything behind the scenes would limit user choice and making it general enough in my opinion is outside of scope of this library. Also, personally, when implementing skinning, I would go for a GPU solution, just passing all the input data via buffers/textures and then do all the skinning calculations in the shader -- and then one has to take really app-specific decisions which are hard to generalize for the engine.

I would maybe start with just putting together some sample code that manually extracts the animation data from OpenGEX (see the docs for details on how to access the underlying OpenDDL structures), puts them into some structure and then does the interpolation by using the algorithms from Math library directly. That way one can iterate way quicker than when trying to design an API in the engine right on the first try. This worked very well with the shadow mapping example, which I'm now cleaning up and extracting reusable parts of it (such as frustum calculations) back to the engine.

[decltype-auto]

Hi @mosra !

Ty for the detailed thoughts, I'll have to ponder about that a little; will get back 2 u next week.

But one quick shot right now:

My thought on this is that the animation system should not be tied to the scene graph (or to any other component) at all in order to keep the engine composable and allow the users to e.g. drop the builtin scenegraph implementation in favor of something better tailored for particular use case, if needed.

What about a general purpose (maybe abstract) UpdateVisitor inside the scenegraph sub-system and a specialized AnimationVisitor outside?

[mosra]

Initial animation support has been merged in #191 and I have now a pretty clear idea about how to proceed with skinning -- basically matching the design in glTF, having the simplest possible implementation first and then adding features later. Closing this then, follow #101 for further updates.