Masking is a little-understood concept that is important to composers and mix engineers. Essentially, masking is what happens when one sound makes it difficult to hear another sound. An obvious example of this is two instruments playing the same note, with one instrument sounding much louder than the other.
This can happen with notes or chords, where the voicing of one instrument covers up another, softer instrument. It can also happen with frequencies, where an element of one sound covers up an element of another sound. As with the example above, this happens when two instruments are playing the same note or frequency range and one is much louder than the other.
It can also happen when the notes or frequencies are not exactly the same, but nearby. The effect is particularly strong when both instruments are playing the same or similar parts, and the sounds blend very well. A common example is of distorted guitars and distorted bass. On its own, the distorted bass might have a heavy growl caused by a lot of energy in the lower mids and a crunchy fuzz on top. Once the guitars are brought in, however, the bass is reduced to a low-frequency rumble beneath the guitars. Even though the main energy of the guitars might be in the upper mids, it masks the upper harmonics in the distorted bass.
Another example is vocal harmonies. A song might have a section where the main melody is sung in parallel harmony – perhaps a third or fourth apart. If both voices are similar (sung by the same singer, in the same style, with similar processing), our ear will hear the upper harmony as being much more prominent than the lower harmony. The effect is sometimes quite striking – the lower harmony simply blends into the upper harmony.
These are both cases of the higher sound masking the lower sound.
Sometimes masking is useful, as it allows a sound to be thickened or deepened by adding other sounds to it. Other times it is undesirable as it makes it difficult for the listener to distinguish between the different sounds.
In the bass/guitar example, greater separation could be achieved by filtering or EQ so that each instrument contributes a unique sonic component to the mix. Alternatively, each instrument could be given a different depth. For example, the bass could be up front and the guitar further back in the mix.
In the vocal example, greater separation could be achieved by instructing the singer to perform each part differently – such as whispering one part, or perhaps singing one part forcefully. Better yet, have a different singer perform one of the parts.