unity3d - Problem with Unity's Pyshics.RaycastAll layer parameter

Question

This raycast always returns an empty array if I use the layer parameter, but if I don't use it everything works properly. I wanted to make a separeted layer for my enemies(trigger colliders) in order to optimize the process.

private RaycastHit[] GetRaycastHits(int layer = 0)
{
    return Physics.RaycastAll(partToRotate.position, partToRotate.forward, ShootRangeDistance, layer);
}

Its very weird that even if a pass LayerMask.NameToLayer("Default") it doesn't work. I made sure that my enemies are on default layer for now and I already tried to pass the parameter that allow the cast to hit triggers.

Answer 1

You are using the Layer mask wrong!

See Layers

The Physics.Raycast function takes a bitmask, where each bit determines if a layer will be ignored or not. If all bits in the layerMask are on, we will collide against all colliders. If the layerMask = 0, we will never find any collisions with the ray.

For further understanding of bitmasks

A bit mask basically works like the following:

Each (in this case) layer is represented by one enabled (1) or disabled (0) "bit".

Remember that one int in the binary backend is basically just 4 bytes => 32 bits => 32 available layers in Unity.

Let's say for now there are only 8 layers as an example.

So the bitmask looks like e.g.

0000 0001

Now I'll add the indices on top of it so you understand what I'm talking about

//indices 7654 3210
          0000 0001

this means the layer at index 0 is enabled, anything else is disabled. This would have the int value 1 (= 2^0)

A second example

// indices 7654 3210
           0000 1010

this would mean the layers at index 1 and 3 are enabled, the int value would be 10. **Why? Because it equals

0*2^7 + 0*2^6 + 0*2^5 + 0*2^4 + 1*2^3 + 0*2^2 + 1*2^1 + 0*2^0

= 0 + 0 + 0 + 0 + 8 + 0 + 2 + 0

= 10

In order to write it easier you would usually use bitshift operations like e.g.

var layers = 1<<3 | 1<<1;
// In bits   1000   10

which would result in the last mask example.

---

So the Default layer actually would be

int defaultLayer = 1<<0;

which does not equal 0 but rather 1!

---

In order to not have to do the Layer calculations or hardcode it via string you should rather use a LayerMask field like

public LayerMask raycastLayers;

in your component, configure it via the Inspector and later pass it in like

// This is possible because LayerMask has an implicit conversion from int to LayerMask
//                                                       |
//                                                       v
private RaycastHit[] GetRaycastHits(LayerMask hitLayers = 1<<0)
{
    return Physics.RaycastAll(partToRotate.position, partToRotate.forward, ShootRangeDistance, hitLayers.value);
}

and

var hits = GetRaycastHits(raycastLayers);