Have noticed the same thing; small rocks, stones, etc. If the panel is painted it could be repaired. If the color is molded in I'm not sure how the repair could be done to match the color - may be possible?
Thanks! It's a molded panel. Fred, what's the brand name of this plastic polish you use, and I assume it's found at automotive parts stores?