If your siding has small black dots that do not come off during cleaning but can be scraped off using your finger nail. You probably have artillery fungus on your siding. This cannot be removed from the siding without causing damage.
The presence of persistent, small, brown spots on cars, house siding, plants, and other surfaces may indicate the presence of the fungus Sphaerobolus stellatus. This cosmopolitan saprophyte is also called the "artillery fungus" or "sphere thrower" since it forcibly ejects its brown spore masses for considerable distances. This fungus grows on wood and bark chips, dying and decaying wood, and dung. Although it is not pathogenic to plants or woody ornamental, it is a nuisance to homeowners.