Raycast2D not working with LayerMask

Question

I am attempting to use a Raycast2D to detect if an attack has hit an enemy. I created a LayerMask for the "Enemy" layer and I have an object with a collider on the "Enemy" layer. I have no problem hitting the collider without using a LayerMask, however when I do, I get nothing back. What am I missing here?

LayerMask EnemyLayer;

void Start ()
{
    EnemyLayer = LayerMask.NameToLayer ("Enemy");
}

public void BasicAttack()
{
    PlayerVitality.Stamina = 0;

    Vector2 attackDirection = CalculateAttackDirection ();
    float attackDamage = CalculateDamage (BasicAttackDamage);
    Debug.Log (EnemyLayer.value);
    RaycastHit2D hit = Physics2D.Raycast(new Vector2(transform.position.x + 2, transform.position.y), attackDirection, BasicAttackDistance, EnemyLayer.value);

    if (hit.collider != null)
    {
        Debug.Log("You hit: " + hit.collider.gameObject.name);
    }
}

Answer 1

The casting between int and LayerMask can be a bit confusing.

When you call LayerMask.NameToLayer("Enemy"), you get the layer index. Let's assume that "Enemy" is layer 7, so that call would return 7. So far, so good, but a layer index is not the same as the bitwise layer mask that is used by the raycast function.

In order to mask layer seven, you set the seventh bit: 1 << 7, which is 128. If you're not familiar with bitwise operators such as | bitwise OR, & bitwise AND, or << left shift, you can look up some other tutorials to understand bitmasks.

So, you're passing in 7 (bits 0, 1 and 2) where you need to pass 128 (bit 7)

Long story short, you need to turn your index into a bitmask.

Either of these will work:

//get layer index, use it to make a bitmask
EnemyLayer = 1 << LayerMask.NameToLayer("Enemy");

//ask unity for a bitmask directly
EnemyLayer = LayerMask.GetMask("Enemy");