Script for lip sync I'm working on

Moho allows users to write new tools and plugins. Discuss scripting ideas and problems here.

Moderators: Fahim, Distinct Sun, Víctor Paredes, erey, Belgarath, slowtiger

Script for lip sync I'm working on

Postby strider2000 » Fri Nov 20, 2015 6:16 am

I thought I'd show the lip sync script I'm working on. The goal is for something that can be applied to just a porting of audio very quickly, and can be used against just about any mouth rig. Unfortunately the screen capture didn't capture all of the frames and so the lip sync looks bad when I'm showing Anime Studio. It didn't look like that at all in real life. In any case I decided to post this anyway just to show the idea. Sorry about the really bad screen capture :oops:

My new website (just getting started) | M1 Characters for Moho
User avatar
strider2000
 
Posts: 424
Joined: Sat Mar 07, 2015 5:14 pm

Re: Script for lip sync I'm working on

Postby lehtiniemi » Fri Nov 20, 2015 2:07 pm

Wow, looks really helpful! I'm most definitely interested in this. Smart with the phonetic spelling, works for any language much easier that way.
lehtiniemi
 
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Postby Breinmeester » Sat Nov 21, 2015 6:37 pm

I like the idea of this script. Does it analyse the sound file to determine the timing of the phonemes? Or is it evenly spaced out over the given number of frames?
Breinmeester
 
Posts: 294
Joined: Thu May 13, 2010 2:01 pm

Re: Script for lip sync I'm working on

Postby strider2000 » Sun Nov 22, 2015 1:39 am

Breinmeester wrote:Does it analyse the sound file to determine the timing of the phonemes?

I had written a script that located the maxima and minima of the audio. It even had tolerance controls, to help identify local maxima and minima. It worked ok, but for a number of reasons correlating to specific phonemes is more complex.

For this approach I took a different tack and focused on getting the appropriate phonemes in order. It does space space out the phonemes based on the number of frames with the following exceptions
  • it knows the difference between characters that can have extended sound and ones that do not
  • stops (ie consonants that are not like s, z, v, f etc) are assigned only one frame
  • you can repeat letters (typically you'd do vowels and character like s and z) to extend them
  • you can use "-" for the rest mouth position
So "noooo waaaaay" would extend the "o" and "a" sounds (I'd actually spell it "nOOOOOO wAAAAA" for long "o" and "a" and no real "y" sound and use the phonetic approach)

I originally put it a control to set the length for vowels. The idea was that you only needed to put in the starting frame. It turned out that start and end frame was easier and more accurate for a wide range of audio.

Maybe that's more than you wanted to know, but I hope it answered your question.
My new website (just getting started) | M1 Characters for Moho
User avatar
strider2000
 
Posts: 424
Joined: Sat Mar 07, 2015 5:14 pm

Re: Script for lip sync I'm working on

Postby Breinmeester » Mon Nov 30, 2015 1:24 pm

I'd love to test this out. Keep me updated on your progress please! 8)
Breinmeester
 
Posts: 294
Joined: Thu May 13, 2010 2:01 pm

Re: Script for lip sync I'm working on

Postby funksmaname » Mon Nov 30, 2015 9:41 pm

This looks like a really useful tool!! :)
User avatar
funksmaname
 
Posts: 3107
Joined: Tue May 29, 2007 11:31 am
Location: Nelson, New Zealand

Re: Script for lip sync I'm working on

Postby strider2000 » Tue Dec 01, 2015 4:48 am

I think that the script is in good enough shape to make available. I've put it here
http://www.mediafire.com/download/pp3u0cr44jan3sa/msLipSync.zip
The zip file has a readme file that hopefully explains things well. As an FYI, it should be installed in the custom content folder, because it will look for the lipSync.txt file under there (that's what allows you to use bone controlled lip Sync for any bone rigging of the mouth). I've tried to make it robust. Hopefully no one will have problems. I did provide a debug check box to help if you do. If you have problems let me know, but be aware I just do this when I have a few cycles :( Hope it's helpful for people.
My new website (just getting started) | M1 Characters for Moho
User avatar
strider2000
 
Posts: 424
Joined: Sat Mar 07, 2015 5:14 pm

Re: Script for lip sync I'm working on

Postby funksmaname » Tue Dec 01, 2015 9:55 pm

Thanks Strider :) I look forward to trying it out!
WOW - Thanks for the excellent documentation as well!!!!! :!: :!: :!:
User avatar
funksmaname
 
Posts: 3107
Joined: Tue May 29, 2007 11:31 am
Location: Nelson, New Zealand

Re: Script for lip sync I'm working on

Postby DK » Tue Dec 01, 2015 11:48 pm

Yes thanks Strider! What a wonderful and much needed addition to AS!

Cheers
D.K
User avatar
DK
 
Posts: 2357
Joined: Mon Aug 09, 2004 6:06 am
Location: Australia

Re: Script for lip sync I'm working on

Postby strider2000 » Wed Dec 02, 2015 4:05 am

Hope it works for y'all :) (I live in North Carolina) My hope is that no one has any problems, but if you do let me know. I can't promise a quick response (especially with Christmas coming), but will do what I can.
My new website (just getting started) | M1 Characters for Moho
User avatar
strider2000
 
Posts: 424
Joined: Sat Mar 07, 2015 5:14 pm

Re: Script for lip sync I'm working on

Postby lehtiniemi » Thu Jan 07, 2016 8:19 pm

Thank you so much for this work! It's working really well and making things soooo much faster. The only thing that I came up with when trying it a bit was that is it possible it would read the first frame number from the playhead position and prefill it into the field? Then it could already focus the end frame number field and select the number in this box so that if you start typing, it would replace it. This would make things a lot faster.

Great work!!! Really!!!!!!! Thank you!
lehtiniemi
 
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Postby lehtiniemi » Fri Jan 08, 2016 10:48 am

Here are some issues I've found so far (besides the amazing speed in workflow that this script provides!!! :))
-I have a switch layer where each switchable mouth is a folder (containing an image and a bone structure). When I initialize the script when one of these folders is chosen (instead of the main switch layer), the script somehow "crashes" (it doesn't popup and the values it contains are reset the next time I start it). This is nothing serious, but maybe the value reset could be avoided if I accidentally launch it when an incorrect layer is accidentally selected?

-For some reason, each time I open the script, the Lua console pops up, telling me the lipSync.txt is not found in the directory /Users/juhana/Documents/Anime Studio Pro/scripts/utility. However, if I check this directory, it is there. What could cause this? I don't know if this has nothing to do with it, but initially I installed the script using ASPRO:s built-in "install script" and then using your instructions. Can there be some duplicate installation existing that causes this?

Thanks again. Please let me know if you accept donations via paypal or something. :)
lehtiniemi
 
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Postby strider2000 » Sat Jan 09, 2016 4:56 pm

Hi lehtiniemi,
I'm glad the script is helpful. I don't know why ASP is poping up the Lua console. It probably is a good idea to check to see if lipSync.txt is in the build-in install directory as well as the user directory. Maybe that is the problem.

Your suggestions are good ones and I'll look into the issues. Right now I've got a busy schedule so it may take a little before I can get some updates. I hadn't considered paypal before, but just recently started doing some scripts for fee so I'm looking into that now.

I'd love to just allow you to highlight the timeline, then right click or something and have the frames pre-populated, but I don't know yet how to do that in the timeline (if it can be done at all). Mouse movements in the workspace area are easier. I may try to map those to timeline movements, but ... I just don't have as many hours in the day as I'd like :o
My new website (just getting started) | M1 Characters for Moho
User avatar
strider2000
 
Posts: 424
Joined: Sat Mar 07, 2015 5:14 pm

Re: Script for lip sync I'm working on

Postby lehtiniemi » Sat Jan 09, 2016 5:19 pm

I found the reason. In the msLipSync.lua there's the line:
msLipSync.myPhonemes.BuildPhonemeMap(moho:UserAppDir().."\\scripts\\utility\\lipSync.txt")

This must be changed to the following on OSX:
msLipSync.myPhonemes.BuildPhonemeMap(moho:UserAppDir().."/scripts/utility/lipSync.txt")

Otherwise the resulting line will contain both \ and / in the path which causes the lipSync.txt not be found, like this:
juhana/Documents/Anime Studio Pro\scripts\utility etc
lehtiniemi
 
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Postby synthsin75 » Sat Jan 09, 2016 6:38 pm

lehtiniemi wrote:This must be changed to the following on OSX:
msLipSync.myPhonemes.BuildPhonemeMap(moho:UserAppDir().."/scripts/utility/lipSync.txt")


That syntax should work for both Mac and Windows.
- Wes
ImageImageImage
User avatar
synthsin75
 
Posts: 6678
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma

Next

Return to Scripting

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron