Page 1 of 2

Masking explained... I hope!

PostPosted: Mon Oct 15, 2007 3:05 pm
by heyvern
First things first:

Masking looks incorrect in the preview when you use it with nested layers. It must be rendered to see it properly. This is what made me think that it was confusing at first. If you don't trust the preview and only look at a render than masking should make more sense.


A mask is just like an alpha channel in Photoshop. Adding isn't the same as... "adding" as you might think of it. A mask is a black or white "alpha channel".

This has nothing to do with the color of the shapes used in the mask layer. Black and white is just a term used that is related to alpha channels. When a layer has a filled shape it is the "transparency" that determines its masking or whether that layer is "black" or "white" in the masking alpha channel.

The really confusing bit is Reveal all and Hide all. These are opposites. The mask "starts out" in the opposite way.

Think of it this way:

Black hides
white reveals

Adding to a mask creates white, subtracting from a mask creates black It isn't really like taking away or adding to a mask at all... it is based on black or white pixels. Subtracting only "adds" black. Adding only "adds" white.

This has its origins from the mathematical concept of "adding" and "subtracting" the grayscale or RGB values of pixels. Each pixel has a number value from 0 to 255. Adding those numbers makes a pixel "brighter" subtracting wil make it "darker".

In the version of photoshop I started with that was the only way to combine masks by using Add or Subtract or Multiply from one alpha channel to another.

The hide and reveal with white and black alpha channel concept relates to the following group layer masking properties:

Reveal all
is a totally "white" mask revealing everything. It is as if there is no mask. Adding a layer to it is adding "white" to "white" which means nothing happens... unless you use "clear the mask and add to it". This "erases" the mask and "starts over". If you "clear" the mask you are making it all black. If the group mask property is set to "reveal all" and you clear it you basically have turned it into "Hide all".

Subtracting from Reveal all creates a "blank" spot. A black hole through the mask. remember black hides white shows or reveals and Reveal All is all white, unless of course you have "cleared it" now it is reversed. if not then the mask is all white so subtracting creates a black or hidden spot.

I can see how this could get confusing... ;)

Hide All
Now the mask is "turned on" or completely black and hides everything. Adding to the mask will create a "white hole" through the black. remember when you "add" you are creating a white spot that makes things visible. If you use subtract this is just "adding black" to the black mask... nothing happens.

clear the mask and add this layer to it works differently now. Clearing the mask makes it all black and then adds that layer... but.. how it looks depends on layer order. If this layer is above another mask layer it will override that layers masking (see below about layer order).

Layer Order!!!
The ORDER of the masking layer is also a factor. Layers higher in the stacking order take precedence over layers below it.

This is critical for making complex masks. It allows you to stack elements under mask layers with different mask layers... <sigh> a lot of this requires experimentation to see how it all works. Eventually it starts to make sense.

Making a mask layer invisible will also turn off its effect on the mask... obviously.


This is the idea you start with. It will get very complicated when you start to add multiple group layers with different masking to this setup. You can create a complex masking situation in a group layer with multiple layers all doing different things, then set the masking of the GROUP layer itself to mask layers in it's parent layer which has DIFFERENT masking set.

You can create amazing stuff if you get the hang of it.

In the sample below the character on the right pours a can of red paint over the character on the left. This done with several group layers and duplicated layers to create the perfect mask to produce this effect. This would be an example of a "complex" mask because the character is duplicated in "red" and a mask is used to reveal the red character... but also the strokes had to be duplicated on another layer since they are ALSO part of the mask... etc etc etc... <sigh> very complicated but it works and I know WHY it works.


Additional info moved here from later post

Sample file used below:
Masking in AS "in general" is based on the transparency of the layer or shapes on a layer NOT ON COLOR VALUE!

What this means is, if you have a soft edge fill... then you get a "fuzzy" edged mask based on the "transparency" or "falloff" of the soft edge of that fill.

BUT if you use a black to white gradient... that has no effect on masking. The shape is filled but it is filled with black to white. I think this is incorrect. I think it should be an option in masking to use the color values. I think you should be able to create a black to white gradient and use it as a gradient mask but for now this will not work.

Now on to that other gradient example in your file which I think should ALSO work but doesn't... the one with a 100% transparent color for one of the colors in the gradient... This ABSOLUTELY SHOULD WORK but it does not.

I think in both of these situations masking is based on the "filled-ness" of the shape. If the shape has a fill PERIOD it is 100% filled and any transparency other than soft edge effect is ignored. So even though you have a transparent "end" on the gradient... the shape is still "filled" so no gradient mask. I did EXTENSIVE testing of this and absolutely no combination of layers or effects even with separate styles will create the proper gradient mask.

I believe this is INCORRECT behavior but unfortunately this is how it works.

There is however a "trick" you can use to make this work:

Simple gradient mask:

Simple version:
Soft edge does work as a mask. The trick then is to use a soft edged fill to create a "pseudo" gradient mask on a separate layer. You create a soft edge fill that is very large with a large edge radius for the soft edge and use the "edge" as the gradient mask. You would then have 2 mask layers.

Complex Gradient mask:

Complex version with 3 mask layers:
There are problems with the simple version since the large shape will mask EVERYTHING not just a specific bound area like a box with a gradient. So to do anything more complex will require breaking this up into several layers and using a combination of masking settings.

I will try to explain how the complex version in the sample file works:

The main group is set to hide all. The "mask" is now "black" hiding everything.

Layer 1 ---> the color shape layer to be masked
Mask 1 ----> Add to mask but keep invisible
(Adding is "white" so now the shapes are revealed in the layer above it but the mask layer is invisible. We have added white areas to the original black mask)

Mask 2 ----> Subtract from mask
(this layer is soft edge shape to create the gradient. Subtracting removes from the the whole mask. The mask started out black, then we added white areas, now we add a black area which is the soft edge fill.)

Mask 3 ----> Add to mask but keep invisible
This layer "masks" the soft edge layer. The last mask confines the soft edge layer shape to a specific shape but has no effect on the first mask layer above it because it is ADDING. The first mask layer is also adding... adding is creating "white" so adding white to white = white... so no change in the masking of the first mask layer. Mask 2 however is SUBTRACTED which is black. So now the third mask layer is ADDING WHITE only to the second mask. All of this is working based on layer order.)

That last bit for the 3rd mask layer seems... strange. You would think that "adding" to the 2nd mask layer would eliminate it... what is really happening is the 3rd mask is MASKING the 2nd mask. The end result is the actual mask that is masking Layer 1 at the top. As I said it is confusing. Once you "understand it" it gets easier. Took me ages to get the hang of it.

You can get more sophisticated with this by putting these "complex" masks inside ANOTHER group layer to further modify the mask. Remember a group layer can have its own set of masking with in it that is completely separate from the parent group.

You can even have groups within groups within groups. Remember though.. if you do THAT you can't rely on the preview you must render to see the results.

Remember that none of this has anything to do with color values! It has to do with the "filled-ness" (made up term) of shapes in the layer.

Using black and white to refer to the values of a mask is just to put it into alpha channel terms.


PostPosted: Mon Oct 22, 2007 12:07 am
by jorgy

Been a long time, huh? The last I was on here was when the 2007 Foota contest was starting. Time flies!?

Anyway, I've been trying to get some masks working with an alpha.

In my example, located here:, I have a masking group with blue that's supposed to show through, and then two bars on the right hand side where I've done gradients - going from white to alpha and white to black. However, it doesn't have any effect and the two bars appear the same as the blob!

Let me know if that doesn't make any sense and I need to clarify.



p.s. Is it also possible to create a png image with an alpha, and use that as a mask?

PostPosted: Mon Oct 22, 2007 4:49 am
by heyvern
EDIT: The information in this post was moved to the top


PostPosted: Mon Oct 22, 2007 5:02 am
by heyvern
p.s. Is it also possible to create a png image with an alpha, and use that as a mask?

Yes! Absolutely!

This is another alternative to using multiple layers for a gradient mask.

Remember that the alpha or transparency of a PNG file effects the transparency of that layer directly. Using it as a mask will use the transparency of the PNG to "hide" or "reveal" the masked layers above it.

Once again I believe that the color values themselves should be an OPTION at least when doing masking even with an image. In most applications with masking black and white values are used to determine masking.

Until that happens we do what we can to work around it.


PostPosted: Mon Oct 22, 2007 11:43 am
by cribble
Someone needs to sticky this. This post is fantastic, considering i've been using AS/Moho for years and still can't figure out the mask. This helped alot. And i liked the example animation.

PostPosted: Fri Oct 26, 2007 12:23 am
by jorgy

Thanks so much! Your explanations have helped so much!


PostPosted: Mon Nov 19, 2007 1:34 am
by Nucleus
Wow! Vern, you dropped that down to a science. Kudos.
Ernest would be proud :)

PostPosted: Thu Nov 22, 2007 4:19 am
by speedking4
I am filled-ness with gratitude for explaining and showing examples . Another thing that I use that helps is i import that 1 picture with the red ball and blue ball which lists what each layer should be masked.
I dont know how to upload it but its on this forum somewhere.


PostPosted: Sun Feb 10, 2008 9:12 pm
by nanticokerailroad
I haven't messed with masking yet but to add my 2 cents; according to the new manual the most important thing to remember is that all this masking will only work if all the layers are contained within a GROUP layer. If you don't have AS PRO you have to simulate this by putting them in a BONE layer for it to work properly. Not sure if this is true or not. According to the book. Mike

PostPosted: Sun Feb 10, 2008 11:25 pm
by heyvern
Group layers and bone layers are identical. They are both "Group Layers"... bone layers have the benefit of having... bones.

A switch layer is also a "Group Layer" because it contains other layers (but it doesn't have group masking options of course). If the new manual doesn't make that clear it's a mistake. Using a bone layer to "simulate" a group layer is kind of... silly if you ask me. It isn't "simulating" anything. It is doing exactly what it does.

I have pro and I almost NEVER use group layers. In the past I have used group layers "by mistake" and later had to transfer all those layers into a new bone layer, or use a text editor to change the group layer to a bone layer. Both are a pain.

Quite frankly anyone with the standard version is not missing anything by not having group layers. Group layers are redundant. They aren't even needed if you ask me. They could just make the bone layer a "universal group" layer and eliminate the group layer completely.


group layers

PostPosted: Tue Feb 12, 2008 12:21 am
by nanticokerailroad
HeyVern, One big advantage the book gives about the group layer really has to do with moving an animated sequence of events along with its keys to a different location within the camera view/screen. I tried this by just animating some circles I created on different vector layers within a group layer. I could see where this might come in handy. Here's another scenario that I thought of to really show the power of the group layer; Let's say you have an animation of a group of birds, say ducks; each on their own vector layer flying a pattern in the sky you really think is cool and eventually they land on a pond. Ok, you find out later you need to move the pond to really make the shot work. Now what do you do about the ducks? You have to re-animate all the ducks to land at the new location, but not if they are in a group layer. Hi-light the group layer and use the translate layer tool to move the entire animation sequence to a new starting location at frame 0 so the ducks land at the new location of the pond. I know this example is a little extreme but it works. Or just moving a dirt path in the background that have characters walking on it. If they're in a group layer you just move the entire group and they're back on the path. You don't have to re-animate. I played with it and think it can come in handy. I'm sure you'll come up with an argument! JUST KIDDING!!!!!. Mike

PostPosted: Tue Feb 12, 2008 1:20 am
by heyvern
I think you may have misunderstood me.

You can do all of those things with bone layers OR group layers. A bone layer is EXACTLY the same as a group layer.
If you don't need bones just don't create any.

I do those things you describe all the time. I just use bone layers instead of group layers. A group layer can't do anything. A bone layer can do what a group layer does PLUS it has bones.

For me group layers are redundant. They serve no purpose only because bone layers are exactly the same and have more options for controlling the layers inside. I stopped using group layers the last time I had to move 20 layers from a group layer to a bone layer so I could have more control over the layers.

There is no reason to ever use a group layer. Use a bone layer instead.


group layer

PostPosted: Tue Feb 12, 2008 2:20 am
by nanticokerailroad
Your right, i tried it with bone layers! That's why your the guru. (Master Yoda Vern) I guess it just boils down to choice on how you want to group layers.

Thanks for all the clarification on the masks too!!!

PostPosted: Mon Mar 10, 2008 11:49 pm
by esotropiart
Cool, thanks for the info on masking. When I get that far, I'll refer back to this. Hopefully what you are talking about works in regular AS, not Pro, since that's what I have. If not, oh well. I think from scanning your post you said AS users aren't missing out on anything in this regard.

I love the animation. The emotions of the innocent bystander are greatly expressed in his eyes and motion of the paper. It reminds me of the style of Herman comics.


PostPosted: Tue Mar 11, 2008 12:48 am
by heyvern
Everything regarding this masking information is the same for the standard version. The big difference is that you don't have group layers in standard but as I said, I don't use them for much anyway.

I think group layers are "psychological" anyway. They give a mental image of putting things in nice groups that look like a folder. A bone layer has that funky bone icon which makes you think it should do something or you are "wasting" its function.

At least that is how I think of it. I'm often tempted to use a group layer just because it looks "nicer" in the palette. ;)