User-agent: Scrapy Disallow: / User-agent: * Disallow: /?s=* Disallow: */?* User-agent: * Disallow: /store$ Disallow: /store/$ Disallow: /wp-content/ Disallow: /*/action~*/