• drphungky@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    2 months ago

    Anyone using bots that are openly bots would be limiting their shit to robots.txt. Anyone who is out trawling the whole website is probably obfuscating what they’re doing by spoofing new user agents.