You are most at risk from the smallest dust particles that you cannot see. That doesn’t mean they are not there.

Even when you are keeping the dust down, you are still likely to need a facemask.
It is easy to use a mask badly. This means it might not give you ANY protection. You should first be given a facefit test to make sure the mask is right for you. You then need to wear it correctly so that the really fine dust cannot get in through any gaps. Store masks properly. Don’t leave them in a pile of dirt!!