A robots.txt file is the first thing most crawlers read. Use it to block staging areas, thin pages, and duplicate URLs while keeping your important content open to indexing. Keep rules simple and test them before deploying.