jwm.robotstxt.googlebot.ParseRobotsTxt

jwm.robotstxt.googlebot.ParseRobotsTxt(robots_body: str, parse_callback: jwm.robotstxt.googlebot.RobotsParseHandler) → None

Parses body of a robots.txt and emits parse callbacks. This will accept typical typos found in robots.txt, such as ‘disalow’.

Note, this function will accept all kind of input but will skip everything that does not look like a robots directive.

Wrapper Note (jwm.robotstxt): If using with RobotsMatcher make sure to call the InitUserAgentsAndPath first to initialise the internal user agents and path properties.