2D Puppeteering Software?

DK · Post by DK » Thu Sep 22, 2011 3:46 am

Does anyone know of any 2D digital puppet software that might suit an AS character? Any ideas on how this could be done in a basic way in AS?

Cheers
D.K

GCharb · Post by **GCharb** » Thu Sep 22, 2011 3:53 am

Could you elaborate on what you call a digital puppet software?

DK · Post by DK » Thu Sep 22, 2011 3:56 am

Hi GCharb.
Digital puppetry is where you take a character and make him move in realtime on a computer screen just like a real puppet. The voice is usually done via a headset mic by the operator. You can almost do this with AS as it is by grabbing bones and dragging them around.

Maybe using preset actions? You would have to have some sort of live audio feed so that the mouth is triggered when you speak though.

Cheers
D.K

Genete · Post by **Genete** » Thu Sep 22, 2011 12:32 pm

Hi DK,
this has been covered before in this forum but I don't remember where.
Just do this:
-Take your real puppets and add them some highlight dots (a brillant button would be enough)
-Grab them with a video camera.
-Take the video and inside AS use the Video track feature and you can move the AS puppet like the real one was moved.

-G

DK · Post by DK » Thu Sep 22, 2011 12:57 pm

Hi Genete.
Thanks for the response but I think you misunderstand.....
These puppetry tools would be used for a live performance of an AAS character....ie...the AS character or digital puppet would be manipulated in real time to a live audience. maybe a live feed audio that would move the mouth and sme hot keys that would induce movments in the character to amuse a live audience.

Possibly something like this could also be recorded for later playback or rendering to video?

Cheers
DK

Post by **Víctor Paredes** » Thu Sep 22, 2011 1:11 pm

I remember some videos from people hacking kinect for that task.
http://www.youtube.com/watch?v=tAGnSrdOfyA
Probably you can adapt an AS rig to work with it, but it should involve a lot of coding.

DK · Post by DK » Thu Sep 22, 2011 1:21 pm

Hi Selgin.
Thats exactly what I meant. Thanks for the post. I will investigate

Cheers
D.K

Post by **Víctor Paredes** » Thu Sep 22, 2011 1:23 pm

I just found this, I think it's closer to your idea
http://www.youtube.com/watch?v=OA_4rAWwYdg

GCharb · Post by **GCharb** » Thu Sep 22, 2011 2:17 pm

Oh, real-time capture, that would be fun!

You need something like motion builder and a script to import in AS I would say.

Rudiger · Post by **Rudiger** » Fri Sep 23, 2011 12:22 pm

It's not quite 2D puppetry, but you could certainly adapt my MorphDials script to the task http://www.youtube.com/watch?v=-oJcqAxbjoE, especially if they add relative morphing to the scripting interface to make it real time.

Kinect + MorphDials would be very interesting to play around with. I'm sure you could do something with dlls to get the data from the kinect driver in real time. You would also have to somehow generate events on the mouse device to get the frame to update as well.

DK · Post by DK » Fri Sep 23, 2011 12:56 pm

Reason I'm exploring this is that I used to dabble in puppetry commercially and it would be extremely useful to be able to create a puppet from vectors using AS as a realtime performance tool.

I wonder if you could assign hot keys to the Morph dials or actions so that you could operate the 2D puppet in real time from your keyboard.
You could then possibly write a script that would allow a live audio feed to operate auto lipsinc on your character you would almost be there without having to use any third party software.

Any thoughts? Could such scripts be written?

Cheers
D.K

SvenFoster · Post by **SvenFoster** » Fri Sep 23, 2011 1:44 pm

need one of these http://www.engadget.com/2011/07/24/soft ... -intimida/

alternatively you could take a look at one of the decent game engines out there..... or maybe even blenders game engine? its has decent python scripting (maybe others two..) you can draw boxes to write scripts too... It supports blenders actions and even a form of the NLA mixing (mixing actions *properly* smith micro... ) easily be able to do a 2d ortho projection and skeleton.

you can use keyboard, mouse or on screen HUD controls to control or if you have a multi touch device that would be a sweet way to manipulate it

aka iphone/ipad andriod device? have to write that yourself

just a thought.

Rudiger · Post by **Rudiger** » Fri Sep 23, 2011 2:22 pm

DK wrote:Reason I'm exploring this is that I used to dabble in puppetry commercially and it would be extremely useful to be able to create a puppet from vectors using AS as a realtime performance tool.

I wonder if you could assign hot keys to the Morph dials or actions so that you could operate the 2D puppet in real time from your keyboard.
You could then possibly write a script that would allow a live audio feed to operate auto lipsinc on your character you would almost be there without having to use any third party software.

Any thoughts? Could such scripts be written?

Cheers
D.K

I have used keyboard shortcuts to control MorphDials before, so that's definitely possible. However, I'm pretty sure only a single event is triggered when you hold a key down, so you probably wouldn't be able to hold down keys to control things gradually.

Using the voice to control the mouths would be harder, but you could probably do it using a DLL like I mentioned for the Kinect. I'm sure there are open source libraries around for voice to phoneme conversion.

I guess if you were using a DLL to control the mouths, you may as well use a DLL for the keyboard as well, then you could support holding keys down or pushing multiple keys at once.

hayasidist · Post by **hayasidist** » Fri Sep 23, 2011 8:19 pm

big issue as I see it is that a keyboard is essentially a serial device - unlike a manipulator's fingers and hands working the control bars and strings on a marionette. if you're pressing and holding many keys then all you're doing is pressing keys - not sending them all to the system... you can't send two or more "rolling over" keystrokes at once. (if you're on windows, take a look at the Control Panel / keyboard for time lag before repeat and then repeat rate...) it's the "last pressed" key that gets repeated.

that said, you could probably get some way towards parallel by (say) an AS script that would recognise keys as "toggles" -- hitting a key to start motion on a contol item (or transition to a different morph) then hitting it again to stop (if it hadn't completed). You could define the default pace of change in the same way as you can choose interpolation (smooth, linear, step...) and/or use key modifiers (ctrl - shift- alt) in some combination to change from the deafult (faster, slower, step, stop,...). and /or have a "commit" key that signals multiple actions to be undertaken concurrently.

so for example L raises the left arm and l lowers it; similarly R and r for the right arm; <escape> stops the current action; open-bracket says mulitple actions coming; LR says get ready to raise raise both arms; close-bracket then says do it... or whatever (will need to ensure no conflict with AS pre-defined key assignments) but / and, as you've implied, rather than keyboard you could get into something like a ShuttlePro which will improve the ergonomics...

the real trick will be to link those keystrokes to an arbitrary rig -- in other words there are likely to be design conventions needed for the AS marionette.

But conceptually it's really not hard (implementing it might take a while tho'

) ... papagayo requires mouth shapes named to a standard - there is no reason why that concept can't be applied to other body parts or - instead of switched vector layers - to doing well-defined translate /scale / rotate operations on bones.

and so to lip-sync. any voice to text system has a latency - the system needs to figure out where one word ends and the next one starts. But if the input audio is then delayed by the same latency before output ...

Now bolt all that together and provide the manipulator with a control environment that provides timely visual (and audio?? and kinaesthetic???) feedback and that's it!

oh - apart from actually implementing this - which is a bit like Walt Disney saying "I've got this great idea for an animated film about an orphaned deer - just do it will you!"

DK · Post by DK » Sat Sep 24, 2011 12:45 am

Wow!!! Nice post on this subject hayasidist. There is an open source puppetteering software avalable I found on google.

ANIMATA
http://animata.kibu.hu/

It's pretty darn basic though. I would think AS had a LOT more going for it.

Cheers
D.K