Robots.txt is a text file which contains parameters of the site indexing for web crawlers. It is usually used to ignore pages in the search engine results.
Robots.txt tab is found in the site settings: Settings → Robots.txt.
If you want to exclude the whole site from web indexing (from all search engines), copy the text below in the File content field:
If you want to exclude only a separate page, copy:
/page/ is the URL of the page you need to exclude. URL is shown in the page settings.
To see the page URL, click the “…” button on the page thumbnail and select Settings.
Copy the link from the Page URL and paste.
For example: Disallow: /page_copy2/
- Learn more about creating a robots.txt file in this Google article.
- If your website has a page available on multiple URLs or some similar pages, then search engines can consider them as duplicates.
- If this occurs, a search engine chooses only one page to show in the search results. Such page is called canonical.
- If you want to specify any page as canonical, then switch the toggle on.