RFC 9309: Robots Exclusion Protocol
This document specifies and extends the "Robots Exclusion Protocol"           method originally defined by Martijn Koster in 1994 for service owners           to control how content served by their services may be accessed, if at           all, by automatic clients known as crawlers. Specifically, it adds           definition language for the protocol, instructions for handling           errors, and instructions for caching.