The magenta content of the upper layer replaces the magenta content of the lower layer composite. This is the "M" in CMY and CMYK.
The display of the layer view presents only magenta information as a cue that the layer is in this layer mode. All of the red, green, blue, cyan, magenta, yellow, black, mask and inverted mask modes do this.
Where the higher layer is partially transparent, the result is mixed with the lower layer composite in a manner proportional to the amount of transparency.