raycast not showing hit

Question

I have a raycast, however it goes through the enemies as it is suppose to but it hits nothing else. If I remove the mask it hits the enemies layer. If I remove the layer and use raycastall it hits only the enemies.

If I use raycast it goes clear through a wall and hits the player but does not show as a hit, in fact i get the error

    NullReferenceException: Object reference not set to an instance of an object
    EnemyAI.OnTriggerEnter2D (UnityEngine.Collider2D other) (at Assets/Scripts/EnemyAI.cs:32)

Line 32 is Debug.Log (hit.transform.gameObject);. If I remove it, nothing happens at all. No error, and no hit.

Heres is the code

void OnTriggerEnter2D(Collider2D other){
    if (other.gameObject.tag == "Player") {
        myStats.inRange = true;

        Vector2 direction = other.transform.position - transform.position;
        hit = Physics2D.Raycast(transform.position, direction, myCircle.radius + 1, LayerMask.GetMask("enemies"));
        Debug.Log("Radius size is " + (int)myCircle.radius);
        Debug.Log("donthit value " + LayerMask.GetMask("Enemies"));
            Debug.Log("direction " + (myStats.player.transform.position - transform.position));
        Debug.DrawRay(transform.position, other.transform.position - transform.position, Color.white);
        Debug.DrawLine(transform.position, myStats.player.transform.position, Color.white);
        //Destroy(hit.transform.gameObject);
        Debug.Log (hit.transform.gameObject);

        if(hit != null && hit.transform.gameObject != null){
            if (hit.transform.gameObject.tag == "INDESTRUCTIBLE") {          
                Debug.Log("WALL");
                //  Destroy the Tag "Enemy" here
            }
            if (hit.transform.gameObject.tag == "Player") {          
                Debug.Log("player");
                //  Destroy the Tag "Enemy" here
            }
         
            Debug.Log("Tag name is " + hit.collider.tag);
        }

        Debug.DrawRay(transform.position, myStats.player.transform.position - transform.position, Color.white);
    }   
}

Answer 1

So it seems there are two things at work here. First, according to the documentation, Raycast 2D will also detect collider(s) at the start of the ray. If you don't use raycastAll then the source enemy will stop your ray before it goes out into the world. To prevent this you could use a layer mask. Just to be sure, a layer mask signifies the layer that you want to hit, not the layers you want to ignore. So to make sure the enemy layer is the only layer you ignore you can use this:

var layerMask = Physics2D.DefaultRaycastLayers & ~LayerMask.GetMask("Enemies");

And then use this mask in your raycast.

Second, there must be a reason you're not hitting anything else. If you applied the layermask in the wrong way then you filter out any hits against the wall or player. It looks like you both have an enemy tag and an enemy layer so make sure to get this straight. But if you used raycastAll without mask and still didn't hit anything then make sure your objects meet all the requirements. They must have 2D colliders of course. If they have colliders, make sure they either aren't triggers or that "Raycasts hit Triggers" is enabled in Edit -> project settings -> physics2d.

A final thing to check is whether or not your ray goes far enough. I noticed your debug draw does not reflect your raycast perfectly. Use this to draw the actual ray:

Debug.DrawLine(transform.position, transform.position + direction * (myCircle.radius + 1) / direction.magnitude, Color.white);