Articles tagged "robots-txt"

robots.txt, sitemap, and JSON-LD: three files, three jobs

They answer three different questions, and the classic mistake — blocking a page in robots.txt to deindex it — does the exact opposite.