Unity3D Physics2D.OverlapAreaNonAlloc strange (bug?) behaviours with layers

Question

I have encountered a strange bug with Physics2D.OverlapAreaNonAlloc and created a test project to confirm it.

Scene : two cube on top of each other, with rigidbody2d and box collider, bottom one kinematic. The top one is the "Player".

I wanted to use Physics2D.OverlapAreaNonAlloc to do some enemy detection so I used a code that looks like this, that I attached to the player :

using UnityEngine;
using System.Collections;

public class Test : MonoBehaviour {

    Vector2 pointA;
    Vector2 pointB;
    Collider2D[] enemies = new Collider2D[2];

    void Update () {

        //Enemy detection
        pointA = (Vector2)gameObject.transform.position - new Vector2(0.1f, 0f);
        pointB = (Vector2)gameObject.transform.position + new Vector2(0.1f, 0.1f);

        if (Physics2D.OverlapAreaNonAlloc(pointA, pointB, enemies, 9) > 0)
        {
            Debug.Log("first enemy :" + enemies[0].name);
        }
    }
}

9 is the layer number of the enemies layer. There is no enemies in the scene so the previous code should never display something in my console. Sadly...

When I run the code, here is what happens : - if the "Player" layer is default, it is somehow detected by my Physics2D.OverlapAreaNonAlloc - If the "Player" layer is another layer, it is not detected.

I don't understand this behaviour, is "default" supposed to be a "super" layer that contains all layers ?

Answer 1

Use LayerMask.GetMask() in the call to Physics2D.OverlapAreaNonAlloc().

// "Enemy" represents the layer assigned to your enemies
Physics2D.OverlapAreaNonAlloc(pointA, pointB, enemies, LayerMask.GetMask("Enemy"))