The pic isn't very clear. You sure its grit and not sawdust? Rubbing it between your fingers should tell you.
Is the mortar between the bricks colored? That pic looks more in line with deteriorating mortar. I have doors in my basement that do that. Old brick can deteriorate and crumble. The general movement of the house can grind away at it. Most of the crumbling brick in my house is due in part to them having been painted 30-50 years ago. I have soft clay bricks in my basement. They need to breath. Paint... paneling don't allow for that.