Masked shape outline upon rendering

Wondering how to accomplish a certain animation task? Ask here.

Moderators: Víctor Paredes, Belgarath, slowtiger

User avatar
cableon
Posts: 155
Joined: Thu Jan 17, 2008 6:44 pm
Contact:

Masked shape outline upon rendering

Post by cableon »

Hi everyone,

Just a little issue that has been triggering me for a while, regarding mask rendering : would anyone know the reason why, once rendered, masking shapes set in "multiply" mode tend to show those small, undesirable outlines around the layers they're masking ?
And how to get rid of those outlines ?

Image

Such tiny, darker contours as the one around the arm above indeed are particularly noticeable in the case of group-masked folders (i.e. folders comprised of a "multiplied" mask + a "normal" vector layer below, added to that mask) set above one another -- regardless of the fact that each folder is in "normal" mode (the inner mask only is multiplied) and that, of course, neither the masking or masked shape are stroked.

Here is a little screenshot of the kind of folder I'm talking about, in case that wasn't clear :

Image

One more crucial detail : those outlines also actually seem to occur upon rendering whenever the mask itself exceeds the surface of the masked shape below. Here is what the arm looks like without masking for instance :

Image

Shouldn't however mask shapes be strictly constrained to the shapes they're masking ? Why do those contours show up in case of superposed group masks ?

"Expanding the mask by a pixel" and/or "excluding strokes" of the masked layer below does not make those outlines disappear -- nor does exporting sequences with "antialias edges" and/or "extra-smoothed images" left unchecked.

Hence my request : would anyone know a way to get rid of those outlines please ? What basic detail am I neglecting ?

And, subsidiarily : would it be realistic to expect a future option that prevents a folder from being affected by any other layers'/folders' blending mode ? :shock:

Cheers!
User avatar
Greenlaw
Posts: 9192
Joined: Mon Jun 19, 2006 5:45 pm
Location: Los Angeles
Contact:

Re: Masked shape outline upon rendering

Post by Greenlaw »

Are you using a transparent color or solid color for the shadow? If you're using a transparent color, try this:

Set the color style to be solid (Alpha 255). Now use Layer Transparency instead to make the shadow transparent. You can (and probably should since it's a shadow) leave the Layer Blend mode to Multiply.

Coincidentally, I ran into an almost identical situation this morning and the trick above solved it for me. Hope this works for you too.
User avatar
Greenlaw
Posts: 9192
Joined: Mon Jun 19, 2006 5:45 pm
Location: Los Angeles
Contact:

Re: Masked shape outline upon rendering

Post by Greenlaw »

Now that I'm thinking about it, my previous suggestion might not help you. I think what's happening is that the shadow color in the AA for the arm is multiplying against the shadow color of the body, making it even darker in the AA region.

To make this work, you might have to use a solid for the actual shadow color, not as a transparent layer and not use Multiply for the Blend mode. This should give you the same look but without the problem in the AA region.

Alternatively, you could try rendering the arm separately using Layer Comps and then compositing it in AE or another compositing program. But personally, I think it's better/easier to just make the shadow a solid.
User avatar
cableon
Posts: 155
Joined: Thu Jan 17, 2008 6:44 pm
Contact:

Re: Masked shape outline upon rendering

Post by cableon »

Thanks for this prompt reply, Greenlaw.

I do usually rely on solid colors for shading indeed, with layer opacity set to 75% (you are very welcome ;) ). I tried using darker shading colors with lower layer opacity after reading your first post, with but seldom difference as for those outlines.

Not sure to understand what "AA" stands for, but you are exactly right though :
the shadow color in the AA for the arm is multiplying against the shadow color of the body
That is exactly my question : why should two masking layers, covering separate, "regular" layers within two distinct folders, cause such an interference when overlapping each other in multiply mode ?

Even considering that a multiplied layer affects all layers underneath, aren't mask layers supposed to cover their masking surface only ? I don't get it.

As for solid color shading in normal blending mode, that is precisely what I want to avoid : the huge advantage of "group-mask shading" in multiply mode is to let you use a single shading tone for a whole character or object for instance, one that correctly and overally fits the ambient light, without having to set a specific, darker shading color for each and every part. I am well aware that that last option would of course prevent interferences between overlapping shades, and solve the present issue I guess -- but what a restraint, and what a tedious fix. Plus, what would be the point of combining two such awesome options as mask layers and the multiply mode ?

My guess is that I must be missing a very basic principle, and that there must be a simple way to solve this issue.

Thanks a lot once again for your help, though!! #thumbsup
User avatar
synthsin75
Posts: 9935
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma
Contact:

Re: Masked shape outline upon rendering

Post by synthsin75 »

"AA" is anti-aliasing. It may be that, or it may be a result of how Moho composites layers when rendering. Have you tried the "Expand mask by one pixel" option? If so, I'd probably need to see a paired down example of the problem to further troubleshoot.
User avatar
cableon
Posts: 155
Joined: Thu Jan 17, 2008 6:44 pm
Contact:

Re: Masked shape outline upon rendering

Post by cableon »

Update : unchecking the "Antialiasing" option in the project settings window does make those outlines disappear, however notching everything. :/
Have you tried the "Expand mask by one pixel" option?
Yes I have, as well as the "exclude strokes" one, in vain.
Unchecking such rendering options as "antialias edges" and/or "extra-smoothed images" seems ineffective, too...

Let's just hope that Moho experts & designers, and/or sensible users, will enlight us with a tip of their own. ;)

(Should've guessed for AA!)
User avatar
synthsin75
Posts: 9935
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma
Contact:

Re: Masked shape outline upon rendering

Post by synthsin75 »

I must be missing something in your setup. I can't reproduce the problem here. Groups like you show, with a normal blend bottom layer adding a multiply top layer (with layer transparency) to the mask, do not have that edge.
User avatar
Greenlaw
Posts: 9192
Joined: Mon Jun 19, 2006 5:45 pm
Location: Los Angeles
Contact:

Re: Masked shape outline upon rendering

Post by Greenlaw »

Same here. I just tried to recreate this error based on the above description, but I'm not getting those edges with transparency and multiply. There's gotta be something else going on here.

If you could post a scene with the error, that would be helpful. No need to use the actual art, just some shapes that show the edge problem. I'm really curious about this now.
User avatar
Greenlaw
Posts: 9192
Joined: Mon Jun 19, 2006 5:45 pm
Location: Los Angeles
Contact:

Re: Masked shape outline upon rendering

Post by Greenlaw »

BTW, does the error show up in a real render too? By that, I mean when rendering with Moho Exporter or Export Animation as opposed to Preview (ctrl-R). I know Preview (ctrl-R) doesn't render exactly the same as the two final animation render commands. I sometimes notice rough AA and other render issues with Preview (ctri-R) that don't see in renders created with the animation exporters.
User avatar
cableon
Posts: 155
Joined: Thu Jan 17, 2008 6:44 pm
Contact:

Re: Masked shape outline upon rendering

Post by cableon »

So, a good night sleep and a reboot later, I finally had the idea of deleting and refilling the arm's shape, just to make sure it was indeed exempt from the slightest stroke.
Surprise : the outlines now seem to be gone, or at least imperceptible. :shock: :oops:

Image

I couldn't remember adding any stroke, however tiny, to the masked arm's shape, and therefore fetched the older, backup version of the same character again to be sure : here.

Image

Aaaaand it doesn't indicate any stroke at all indeed. Can you now hear the X-Files theme, too ? :)

A subsequent hypothesis could be that the arm shape might have been firstly filled (Cmd + U) with the "both" option checked (not only the "fill" one) in the Status bar, and that the stroke option was then unchecked in the Style window -- hence perhaps a ghost outline remaining once rendered. I cannot remember that happening though, but it happens.

BTW, does the error show up in a real render too? By that, I mean when rendering with Moho Exporter or Export Animation as opposed to Preview (ctrl-R). I know Preview (ctrl-R) doesn't render exactly the same as the two final animation render commands. I sometimes notice rough AA and other render issues with Preview (ctri-R) that don't see in renders created with the animation exporters.
I do indeed use the Exporter (Cmd + B) to render sequences, and the Preview (Cmd + R) to get clean overviews -- with daily clean-ups of the overflowing Render Cache folder.
Layout Preview is slightly rougher than the final Export for sure, and may display a couple interferences, that usually disappear once rendered via the Exporter indeed.
The above mentioned outlines however do remain in the final sequence export, although so imperceptible now that an uninformed eye probably won't notice them.

Image

This issue has nonetheless been recurring now and then for a long time now, and Greenlaw's example somehow comforts me regarding a possible mistake or oversight on my part only : it has furthermore been happening both with vector and bitmap layers, as in the stupid .gif below (cf. the big bearded dude's right arm) :

Image

One might presume that perhaps the bearded dude's shaded right arm (a transparent .png, topped with a vector mask in multiply mode) isn't cleanly cut-out, and suffers from slightly blurred edges caused by file compression -- hence the darker contours briefly showing when the arm is stretched. There again though, backup .png files do seem to show as clean and sharply-cut edges as can be (the old Anime Studio Pro archive is available there).

I will, in any case, keep to the "erase/redo shape fill" method from now on, and keep you updated if those contours reappear.

Thanks once again for your patience and kind interest! :)
User avatar
synthsin75
Posts: 9935
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma
Contact:

Re: Masked shape outline upon rendering

Post by synthsin75 »

For the first problem, the closer to black the multiplied shading color the less you'll see the edge. Here I changed the arm and body shade colors to pure black, with 50% layer opacity.
Image

For the second problem, it's a typical problem with hidden edges (hidden or zero width). For this, I'd just make the shirt fill and stroke different shapes, with the stroke not covering the joint. If you sometimes need that stroke all the way around, just make a second stroke shape and animated their opacity to switch between them.
Image
User avatar
Greenlaw
Posts: 9192
Joined: Mon Jun 19, 2006 5:45 pm
Location: Los Angeles
Contact:

Re: Masked shape outline upon rendering

Post by Greenlaw »

Thanks for looking into that Wes. Your discovery explains why I wasn't seeing that edge in my test this morning, and maybe why I saw it at work earlier this week.

I still don't know what causes that occasional line or gap I find in internal edges but now I think that's a completely different issue.
Last edited by Greenlaw on Sat Apr 08, 2017 11:43 pm, edited 1 time in total.
User avatar
synthsin75
Posts: 9935
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma
Contact:

Re: Masked shape outline upon rendering

Post by synthsin75 »

Greenlaw wrote:Thanks for looking into that Wes. Your discovery explains why I wasn't seeing that edge in my test this morning, and maybe why I saw it at work earlier this week.
Yeah, it initially confused me too, because my test used black to shade.
User avatar
cableon
Posts: 155
Joined: Thu Jan 17, 2008 6:44 pm
Contact:

Re: Masked shape outline upon rendering

Post by cableon »

Thanks a lot Synthsin!!
synthsin75 wrote:For the first problem, the closer to black the multiplied shading color the less you'll see the edge. Here I changed the arm and body shade colors to pure black, with 50% layer opacity.
Awesome!! I think Greenlaw had suggested this option far above, but it didn't seem to make much of a difference once tried here -- probably because of a still too light shading tone. It does work now though -- THANK YOU. :D
For the second problem, it's a typical problem with hidden edges (hidden or zero width). For this, I'd just make the shirt fill and stroke different shapes, with the stroke not covering the joint. If you sometimes need that stroke all the way around, just make a second stroke shape and animated their opacity to switch between them.
Ouch, how many layers will that make ? :shock: Thanks a lot for the tips!!
User avatar
synthsin75
Posts: 9935
Joined: Mon Jan 14, 2008 11:20 pm
Location: Oklahoma
Contact:

Re: Masked shape outline upon rendering

Post by synthsin75 »

cableon wrote:Thanks a lot Synthsin!!
synthsin75 wrote:For the first problem, the closer to black the multiplied shading color the less you'll see the edge. Here I changed the arm and body shade colors to pure black, with 50% layer opacity.
Awesome!! I think Greenlaw had suggested this option far above, but it didn't seem to make much of a difference once tried here -- probably because of a still too light shading tone. It does work now though -- THANK YOU. :D
Glad I could help.
For the second problem, it's a typical problem with hidden edges (hidden or zero width). For this, I'd just make the shirt fill and stroke different shapes, with the stroke not covering the joint. If you sometimes need that stroke all the way around, just make a second stroke shape and animated their opacity to switch between them.
Ouch, how many layers will that make ? :shock: Thanks a lot for the tips!!
Shouldn't be more layers, just more shapes. If selecting the stacked shapes gets to be troublesome, just create a small circle or something outside of the camera view as a part of each shape. That way you can select the shapes by clicking these.
Post Reply