A module for parsing and making sense of robots.txt files.
@push.rocks/smartrobots a module for making sense of robots.txt Install To install @push.rocks/sm...