2D Raycast does not correctly detect overlapping colliders and ignores Sorting Layer

I want to be able to interact with my gameobjects while organising my sprites with Sorting Layer. Right now, Sorting Layer do not affect my Raycast/collider situation and only change the visual appearance of my order.

In my Test scene I have two sprites: a Background and one Object. Both have a 2D Box Collider component. My Raycast script is suppose to detect on which Object my mouse is hovering on; either the Background (BG) or the Object, which lays on top of the BG.

As long as the BG collider is enabled in the inspector, my script will ONLY recognize the BG, even if the other object lays on top of my BG (visually and according to the Sorting Layer) (both Z-positions = 0, Sorting Layer Default (BG: 0, Object 1)). The collider have the correct size and if I disable or remove the Collider from the BG, my script will detect the other Object just fine. The only way to make sure that my Object gets detected, while my BG collider is activated is by changing the Objects Z position to a position between my BG and my Camera (between smaller 0 and greater -10)

I tried to:

this is a fresh 2D project, so my cameras Projection is set to Orthograph, my BG and Object use Sprite Renderer and is use 2D collider
change the "Order in Layer", back and forth, including negative numbers.
add a Foreground (and Background) "Sorting Layer" (both options only had a visual effect and did not affect the Raycast output)
change the positions in the hierarchy.
removing and re-adding the colliders. (!) I observed that the 2D Box collider I added last, will always be behind every other colliders. This is something I can work with for now. But will become a problem later in the future, when the project gets bigger.

I want to be able to interact with items and doors etc. (clicks and reactions to mouse hovering) in my 2D click adventure. If 2D Raycast is not the way to go to archive this, I would be glad to know about the proper technique to use in this situation.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;


public class RayCast : MonoBehaviour
{
    [SerializeField] private Vector2 mousePosWorld2D;
    RaycastHit2D hit;

    void Update()
    {
        mousePosWorld2D = (Vector2)Camera.main.ScreenToWorldPoint(Input.mousePosition);

        hit = Physics2D.Raycast(mousePosWorld2D, Vector2.zero);
        if (hit.collider)
        {
            Debug.Log("Found collider: " + hit.collider.name);
        }
        else
        {
            Debug.Log("No collider found.");
        }
    }
}

I expected the "Sorting Layer" or "Order in Layer" to have an impact on my Raycast, while changing the Z position does nothing. But it is the other way around. Sorting Layer only have a visual effect and moving my Object closer to the camera (Z position) helps to detect the right collider.

How do I solve the problem?

Solution

What gets hit first from a Ray has only to do with how close the collider is to the camera, and nothing with the order in the Hierarchy or the SortinLayers.

To find the "highest" RaycastTarget based on the SortingLayer, I created the Script below.

We fire a Raycast and iterate over all the Colliders that we hit, by using a foreach-loop. Then the function will return the GameObject with the highest SortingLayer. Note: If there is more than one GameObject with the highest SortingLayer, the function will return the last one it found.

using UnityEngine;

public class Raycast : MonoBehaviour
{
    // The name of the relevant Layer
    [SerializeField]  string layerToCheck = "Default";

    void Update()
    {
        Vector2 mousePosWorld2D = (Vector2)Camera.main.ScreenToWorldPoint(Input.mousePosition);

        GameObject result = GetHighestRaycastTarget(mousePosWorld2D);


        // Do Stuff example:
        Debug.Log($"Highest Layer: {result}");

        if (Input.GetMouseButtonDown(0)
            && result != null)
        {
            Destroy(result);
        }
    }


    // Get highest RaycastTarget based on the Sortinglayer
    // Note: If multiple Objects have the same SortingLayer (e.g. 42) and this is also the highest SortingLayer, then the Function will return the last one it found
    private GameObject GetHighestRaycastTarget(Vector2 mousePos)
    {
        GameObject topLayer = null;
        RaycastHit2D[] hit = Physics2D.RaycastAll(mousePos, Vector2.zero);

        foreach (RaycastHit2D ray in hit)
        {
            SpriteRenderer spriteRenderer = ray.transform.GetComponent<SpriteRenderer>();

            // Check if RaycastTarget has a SpriteRenderer and
            // Check if the found SpriteRenderer uses the relevant SortingLayer
            if (spriteRenderer != null
                && spriteRenderer.sortingLayerName == layerToCheck)
            {
                if (topLayer == null)
                {
                    topLayer = spriteRenderer.transform.gameObject;
                }

                if (spriteRenderer.sortingOrder >= topLayer.GetComponent<SpriteRenderer>().sortingOrder)
                {
                    topLayer = ray.transform.gameObject;
                }
            }

        }

        return topLayer;
    }
}