User-Agent: * Disallow: /flash/ Disallow: /styles/ Disallow: /images/ Disallow: /javascript/ Disallow: /webservices/ Allow: / # Allow everything Crawl-delay: 3 # Set a delay of 3 seconds between requests Visit-time: 0000-0600 # Suggest visiting times between 12am and 6am UTC (GMT) Request-rate: 20/1m # Set the request rate to 20 requests every 1 minute (1 request per 3 seconds)