3.18 Include Files That Are Automatically Expected

Search engines and browsers regularly examine websites, requesting specific files by default (they expect them to exist). If the files don’t exist, this will lead to potential errors and emissions being caused when they could be created, especially as the files offer SEO, user-experience, and other benefits to visitors.

Criteria

Expected File Formats: Include the favicon.ico, robots.txt, opensearch.xml, site.webmanifest, and sitemap.xml documents.

Impact

Low

Effort

Low

Benefits

Environmental:
Search engines or browsers request certain files by default, ensuring they are in place will reduce loading errors, and may provide efficiency enhancements in how visitors find or interact with a site.
Accessibility:
OpenSearch enables the browser’s default search box rather than a custom solution to be integrated with your website search, which may aid accessibility as it encourages the use of a browser native component (and / or keyboard shortcuts) rather than a website or application which may suit certain accessibility requirements better.
Performance:
Files that are expected will produce HTTP requests, ensuring they are met will satisfy the products making them and potentially reduce the requests once they are discovered.
Economic:
Robots and Sitemap files can be utilized by search engines to help make your website more findable, this could lead to more visitors and potentially more customers as a result.
Conversion:
Robots can be used to target specific search engines, helping to ensure content is correctly indexed to get a good placement so that visitors can find you easily.

GRI

materials: Low
energy: Low
water: Low
emissions: Low

Example

Code:
```
User-agent: *
Disallow: /cgi-bin/
```
About /robots.txt.

Resources

View the Guideline