Facial tracking and auto lip syncing in MOHO would change everything!

[Poptoogi]

Hello all! It has been a minute since I've been on the forum but I'm sooooo excited about Mike and Victor owning the software now and I can not wait for the program to get the love and attention it deserves. For anyone reading this my name on YouTube is Poptoogi and I've been using MOHO for almost 10 years as a hobbyist but I've also made dozens and dozens of tutorials for it over the years and I'm still learning new things about the program all the time!

One thing I've always wanted to do was to make an animated avatar that I could put on top of my videos for more engagement and to just make the videos more interesting overall. The problem is I tend to ramble and make REALLY long videos sometimes and lip-syncing and animating a character would take up so much time that I would never be able to post anything.

This is why I started messing with Adobe Character Animator. It allowed me to recreate my MOHO character and use triggers (same as actions in MOHO) to animate my character and then use motion tracking and auto lip-syncing for the entire audio track. This was my first attempt so it's not great but the possibilities are amazing.



HOWEVER, this program is extremely weak in terms of rigging and producing fine-tuned animations AND all artwork has to be made in another program.

MOHO has motion tracking capabilities for years and is hardly ever used to my knowledge and it seems like it could be utilized and tweaked to be used in such an incredible way!! Just look what this user has already created in MOHO using some motion capture!


Plus, if MOHO had its own internal lip-syncing feature it would save animators hundreds of hours of work!

Look at the rise of VTubers and the popularity of motion capture animated filters on mobile platforms. Think about the popularity of a program in which you can create your own unique (non-anime) avatars that you can then use to create web-series, videos, and GIFS!

I know people are going to ask for very specific updates that are very technical but I'm telling you, if you want MOHO to complete with Harmony and Adobe you need to have something revolutionary and something new users would say "O.K. that's amazing, I need to have it!" to really boost the software. Adobe has pieces of greatness but if things like this are implemented into future versions of MOHO you will come out on top. I would bet money on it.

What do you guys think?

[SimplSam]

This seems to be a recurrent theme.

Similar threads on:
- Motion tracking was raised recently by sang820 - http://www.lostmarble.com/forum/viewtop ... =3&t=33780.
- Improved automated Papagayo Lip sync variant presented by alanthebox - http://www.lostmarble.com/forum/viewtop ... 15&t=33620

Not sure what plans LM have, but there is certainly ongoing and growing demand for improved automation tools in Moho in these areas. These can enhance productivity and help to take the sting out of some of the more laborious tasks of Moho character animation - particularly when there are other apps in the marketplace that have demonstrated both the comparative ease-of-use and the effectiveness of such tools.

[cgrotke]

if you want MOHO to complete with Harmony and Adobe you need to have something revolutionary and something new users would say "O.K. that's amazing, I need to have it!" to really boost the software.

One company I've recently been impressed by is Pixelmator. They sorta snuck in with a basic Mac image editing app, then came out with a pro version to compete with Photoshop.

A few things I like:

- they really take advantage of the Mac - it is built for it and updates come via automatic software updates
- they have regular, meaningful, free updates. Not just bug fixes but also additions of new and useful features.
- they have the "something revolutionary" angle - one example is "Super resolution" which does just what it says - improves the resolution of images. Makes tiny old photos useable.
- like Moho, Pixelmator has a lot of GPU math going on behind the scenes. : )

Plus, it's cheap and doesn't require monthly tributes to Adobe overlords. : ) I can recommend it to others!

....
Facial tracking could be fun, but I don't have immediate needs for it. General tracking improvements to be able to better combine animation and live action wouldn't be bad.

Any improvements to lip sync and audio would be heartily welcomed! (The existing built-in lip sync help has already saved me a ton of time...)

[DK]

From what I have gathered looking into this face tracking, live puppetry stuff not only do you need a program with the right tools to do the animation but also a way of streaming the results?

D.K

[Poptoogi]

That's true DK. I forgot that MOHO would also have to be integrated to be used with some type of broadcasting software like OBS or Streamlabs which might be a huge undertaking for programmers. It May have to be a separate plugin where you can import your artwork and use it to stream but yeah, that might be a big project, at least for live streaming. Still would be great for speeding up a workflow though.

cgrotke I agree, I think that improvements on auto lip-syncing would be the biggest and best thing MOHO could to for the program. Motion capture would be nice but not necessary. Plus, having new meaningful features is always a great way for a program to stay relevant. I'm not a super fan of Adobe since they went to the subscription-based model but I've used Adobe products for 20 years now and for most of that time it was for free because it was used at my job so I'm very accustomed to their programs and there are things I love about them. The overlords are definitely trying to squeeze end users out of their money but I respect the programmers and creatives that make those programs.

BUT....If I had a choice, I would rather use MOHO for all the things I want to do animation-wise. But right now, since Adobe is offering some great functions for animating certain things I'm o.k. with Frankenstiening stuff together to see what results I can come up with. Like I said though, I'm always on team MOHO and I'm hoping Victor and Mike will make it shine like it's supposed to!!

[DK]

I still think this is a brilliant pathway for Moho to take Poptoogi. Not sure what it would take to crate a plugin for OBS but i really think it is worth taking a Moho step in this direction. I've always wanted to create a live animated character to do reviews on YT.

Cheers
D.K

[SimplSam]

I would certainly like to see a MoCap Record-mode in Moho. Where basic character motions, gestures, facial expressions & lip syncing could be captured for fine tuning, retargeting and replay.

I recently watched (and enjoyed) your Face Puppet Mouths video and it made the final animation of the virtual Actor look so straight-forward and effective.

https://youtu.be/K5hRDQoNCMQ?t=3097 (Jump to final animation @ 51:37 - with your mocap, character edit and final actor in view)

[DK]

Ye...I must admit the character has a very natural looking action which gives a more relaxed feel to the animation overall. Some hand done stuff can look very raw and forced.
Cheers
D.K

[sodiac750]

Facial tracking and auto lipsync like in adobe character animator, but of course better cuz its moho

[chucky]

I want exactly the same , have for years.
I have posted so many times about this so I won't digress.
Right now I would settle for the bone wiggle to work live while the timeline plays.

As for the OBS , no plugin needed, just capture the window and mask the area in OBS.
I did just found two workarounds for a single frame 'flappy mouth' system in OBS... surprisingly effective.

I'll spill the beans when I get time, but in the meantime I could make it a lot better If I can get my gif from Moho not to loop.
Anyone know how to do that?

[jahnocli]

Here's some I just found:
Wondershare

[chucky]

Ha yes jahnocli thanks! I'd already done that just this morning, it works great now the lips sync even though only allows for one mouth shape syncs and animates really nicely.
I should have posted but I had to go out for a bit.
I still think Moho should have the option to render no loop without having to load it to an online service, re-render and download from there.

Anyway I should post the results, even though I half want to keep the secret to myself but I have to at least share with Jeff ( Poptoogi), Eh?

[Phazor]

What would also work in order to implement Vtuber live motion tracking is to simply make a Moho Pro add-on for Blender which converts the Moho Rig into a Grease Pencil Rig so that a person can use Blender's live motion tracking for Vtubing.

You can use Moho's ease-of-use interface to design the character and actions, but then use Blender to do the live tracking, converting actions to shape keys. We can use the Best of both worlds that way. You would have access to all of blenders lighting capability to make the scenes more realistic without using much CPU. The only thing that remains is to make blender touch screen the same way Moho is so there isn't much change in workflow.

In fact, if we want to update the blender file with the Moho rig, it'll be great to just reopen Moho Pro and make corrections and adjustments and then update it in blender in order to avoid blenders complicated workflow.

Literally the only thing you would have to do in blender is the camera setup and changing the pose of the character.

Imagine the possibility
https://youtu.be/GX-3F0BG3t4