jwm.robotstxtΒΆ
Wrapped googlebot bindings for handling robots.txt files.
Modules
This file implements the standard defined by the Robots Exclusion Protocol (REP) internet draft (I-D). https://www.rfc-editor.org/rfc/rfc9309.html. |
Back to top