Physics fixture/shape :setMask bits are inverted in love compared to box2d

[snowkittykira]

Currently a fixture/shape needs to have in its mask every category that it doesn't want to collide with, which is the opposite of how it works in box2d

fixture:getMask says "Returns which categories this fixture should NOT collide with. "
https://love2d.org/wiki/Fixture:getMask

the box2d documentation says "The categoryBits flag can be thought of as the fixture saying 'I am a ...', and the maskBits is like saying 'I will collide with a ...'."
https://www.iforce2d.net/b2dtut/collision-filtering

This is awkward for a few reasons:

• an object that only wants to collide with one category needs to specify the names of all the game's other categories in its mask
• it means people reading the box2d documentation will get confused, for example even on the love2d wiki the following diagram repeats the box2d documentation, rather than the love documentation https://love2d.org/wiki/love.physics
• it's rather hard to reason about, because to avoid collision a shape needs to mask every category the other one has, whereas in box2d the logic is simply "if any category matches the mask it matches", with the object colliding if it matches in both directions.

the inversion that love2d does can be seen here and on the corresponding functions in Fixture.cpp:

love/src/modules/physics/box2d/Shape.cpp

Line 281 in d576692

f.maskBits = ~(uint16)getBits(L);

I'm suggesting that in love 12 the API be simplified to do the same thing as the corresponding box2d functionality. If appropriate the functions could be renamed to "setCollisionMask" from "setMask" for backwards compatibility and to make it clearer that the mask says which things an object will collide with.

[alex-boulay]

Do we still want the behavior to keep Categories number as target (1 to 16) or their integer values 2^category_number ?
like could I set by hand setCollisionmask(Shape1, Shape2) and it would be the same as setCollisionMask(3) ?
or should setCollisionMask(3) have to set the Category number 3 to true ?

[alterae]

i very strongly concur with OP here. flags/masks being inverted is a confusing footgun and makes no sense in light of the behavior of flags/masks in box2d itself

[xiaoandtell]

This needs to be fixed to match Box2D specs. It caused me trouble and wasted testing time until I found this open issue.
To add to the confusion, the code in src/libraries/box2d/dynamics\b2_world_callbacks.cpp and source/modules/physics/box2d/World.cpp matches the Box2D specs.
Can we get an ETA on the fix? Thanks

[slime73]

Right now I'm inclined to not change anything until we do a deeper rework of shape categories, because of this comment I left in the PR:

if someone wants an object to collide with everything or nearly everything, they'd need to specify around 16 parameters with the new API whereas with the old one it'd be much less.

The real issue is that the box2D C API's bit mask doesn't translate well to Lua which doesn't have idiomatic bit masks in the same manner. I'd like to do a better high level API (possibly with an option to drop down to actual bit masks) instead, in the future.

[slime73]

This needs to be fixed to match Box2D specs.

Also, just to clarify, while love.physics uses Box2D (2 currently, not 3) under the hood, it's not necessarily a bug if a love.physics API behaves differently from Box2D. Often it's just a different API design choice.

[xiaoandtell]

Right now I'm inclined to not change anything until we do a deeper rework of shape categories, because of this comment I left in the PR:

if someone wants an object to collide with everything or nearly everything, they'd need to specify around 16 parameters with the new API whereas with the old one it'd be much less.

As a user I found Love2D's inverted Box2d physics masking to be unfriendly in practice, with unexpected results, despite the stated intention to make things simpler. I suspect that they didn't test many use cases of colliding different bodies with multiple categories and different collision rules.
Fortunately the user can write a interface function to revert the logic back to the original box2d behavor by setting calling the love2d setmask function with all category numbers NOT specified.