Script for lip sync I'm working on

Moho allows users to write new tools and plugins. Discuss scripting ideas and problems here.

Moderators: Víctor Paredes, Belgarath, slowtiger

User avatar
strider2000
Posts: 506
Joined: Sat Mar 07, 2015 5:14 pm
Contact:

Script for lip sync I'm working on

Post by strider2000 »

I thought I'd show the lip sync script I'm working on. The goal is for something that can be applied to just a porting of audio very quickly, and can be used against just about any mouth rig. Unfortunately the screen capture didn't capture all of the frames and so the lip sync looks bad when I'm showing Anime Studio. It didn't look like that at all in real life. In any case I decided to post this anyway just to show the idea. Sorry about the really bad screen capture :oops:

lehtiniemi
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Post by lehtiniemi »

Wow, looks really helpful! I'm most definitely interested in this. Smart with the phonetic spelling, works for any language much easier that way.
Breinmeester
Posts: 303
Joined: Thu May 13, 2010 2:01 pm

Re: Script for lip sync I'm working on

Post by Breinmeester »

I like the idea of this script. Does it analyse the sound file to determine the timing of the phonemes? Or is it evenly spaced out over the given number of frames?
User avatar
strider2000
Posts: 506
Joined: Sat Mar 07, 2015 5:14 pm
Contact:

Re: Script for lip sync I'm working on

Post by strider2000 »

Breinmeester wrote:Does it analyse the sound file to determine the timing of the phonemes?
I had written a script that located the maxima and minima of the audio. It even had tolerance controls, to help identify local maxima and minima. It worked ok, but for a number of reasons correlating to specific phonemes is more complex.

For this approach I took a different tack and focused on getting the appropriate phonemes in order. It does space space out the phonemes based on the number of frames with the following exceptions
  • it knows the difference between characters that can have extended sound and ones that do not
  • stops (ie consonants that are not like s, z, v, f etc) are assigned only one frame
  • you can repeat letters (typically you'd do vowels and character like s and z) to extend them
  • you can use "-" for the rest mouth position
So "noooo waaaaay" would extend the "o" and "a" sounds (I'd actually spell it "nOOOOOO wAAAAA" for long "o" and "a" and no real "y" sound and use the phonetic approach)

I originally put it a control to set the length for vowels. The idea was that you only needed to put in the starting frame. It turned out that start and end frame was easier and more accurate for a wide range of audio.

Maybe that's more than you wanted to know, but I hope it answered your question.
Breinmeester
Posts: 303
Joined: Thu May 13, 2010 2:01 pm

Re: Script for lip sync I'm working on

Post by Breinmeester »

I'd love to test this out. Keep me updated on your progress please! 8)
User avatar
funksmaname
Posts: 3174
Joined: Tue May 29, 2007 11:31 am
Location: New Zealand

Re: Script for lip sync I'm working on

Post by funksmaname »

This looks like a really useful tool!! :)
User avatar
strider2000
Posts: 506
Joined: Sat Mar 07, 2015 5:14 pm
Contact:

Re: Script for lip sync I'm working on

Post by strider2000 »

I think that the script is in good enough shape to make available. I've put it here
http://www.mediafire.com/download/pp3u0 ... ipSync.zip
The zip file has a readme file that hopefully explains things well. As an FYI, it should be installed in the custom content folder, because it will look for the lipSync.txt file under there (that's what allows you to use bone controlled lip Sync for any bone rigging of the mouth). I've tried to make it robust. Hopefully no one will have problems. I did provide a debug check box to help if you do. If you have problems let me know, but be aware I just do this when I have a few cycles :( Hope it's helpful for people.
User avatar
funksmaname
Posts: 3174
Joined: Tue May 29, 2007 11:31 am
Location: New Zealand

Re: Script for lip sync I'm working on

Post by funksmaname »

Thanks Strider :) I look forward to trying it out!
WOW - Thanks for the excellent documentation as well!!!!! :!: :!: :!:
User avatar
DK
Posts: 2849
Joined: Mon Aug 09, 2004 6:06 am
Location: Australia

Re: Script for lip sync I'm working on

Post by DK »

Yes thanks Strider! What a wonderful and much needed addition to AS!

Cheers
D.K
User avatar
strider2000
Posts: 506
Joined: Sat Mar 07, 2015 5:14 pm
Contact:

Re: Script for lip sync I'm working on

Post by strider2000 »

Hope it works for y'all :) (I live in North Carolina) My hope is that no one has any problems, but if you do let me know. I can't promise a quick response (especially with Christmas coming), but will do what I can.
lehtiniemi
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Post by lehtiniemi »

Thank you so much for this work! It's working really well and making things soooo much faster. The only thing that I came up with when trying it a bit was that is it possible it would read the first frame number from the playhead position and prefill it into the field? Then it could already focus the end frame number field and select the number in this box so that if you start typing, it would replace it. This would make things a lot faster.

Great work!!! Really!!!!!!! Thank you!
lehtiniemi
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Post by lehtiniemi »

Here are some issues I've found so far (besides the amazing speed in workflow that this script provides!!! :))
-I have a switch layer where each switchable mouth is a folder (containing an image and a bone structure). When I initialize the script when one of these folders is chosen (instead of the main switch layer), the script somehow "crashes" (it doesn't popup and the values it contains are reset the next time I start it). This is nothing serious, but maybe the value reset could be avoided if I accidentally launch it when an incorrect layer is accidentally selected?

-For some reason, each time I open the script, the Lua console pops up, telling me the lipSync.txt is not found in the directory /Users/juhana/Documents/Anime Studio Pro/scripts/utility. However, if I check this directory, it is there. What could cause this? I don't know if this has nothing to do with it, but initially I installed the script using ASPRO:s built-in "install script" and then using your instructions. Can there be some duplicate installation existing that causes this?

Thanks again. Please let me know if you accept donations via paypal or something. :)
User avatar
strider2000
Posts: 506
Joined: Sat Mar 07, 2015 5:14 pm
Contact:

Re: Script for lip sync I'm working on

Post by strider2000 »

Hi lehtiniemi,
I'm glad the script is helpful. I don't know why ASP is poping up the Lua console. It probably is a good idea to check to see if lipSync.txt is in the build-in install directory as well as the user directory. Maybe that is the problem.

Your suggestions are good ones and I'll look into the issues. Right now I've got a busy schedule so it may take a little before I can get some updates. I hadn't considered paypal before, but just recently started doing some scripts for fee so I'm looking into that now.

I'd love to just allow you to highlight the timeline, then right click or something and have the frames pre-populated, but I don't know yet how to do that in the timeline (if it can be done at all). Mouse movements in the workspace area are easier. I may try to map those to timeline movements, but ... I just don't have as many hours in the day as I'd like :o
lehtiniemi
Posts: 107
Joined: Mon Jan 14, 2013 3:18 pm

Re: Script for lip sync I'm working on

Post by lehtiniemi »

I found the reason. In the msLipSync.lua there's the line:
msLipSync.myPhonemes.BuildPhonemeMap(moho:UserAppDir().."\\scripts\\utility\\lipSync.txt")

This must be changed to the following on OSX:
msLipSync.myPhonemes.BuildPhonemeMap(moho:UserAppDir().."/scripts/utility/lipSync.txt")

Otherwise the resulting line will contain both \ and / in the path which causes the lipSync.txt not be found, like this:
juhana/Documents/Anime Studio Pro\scripts\utility etc
User avatar
synthsin75
Posts: 9935
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma
Contact:

Re: Script for lip sync I'm working on

Post by synthsin75 »

lehtiniemi wrote:This must be changed to the following on OSX:
msLipSync.myPhonemes.BuildPhonemeMap(moho:UserAppDir().."/scripts/utility/lipSync.txt")
That syntax should work for both Mac and Windows.
Post Reply