User-agent: * Crawl-delay: 2 Allow: / Disallow: /temp* Allow: *temperature* Disallow: /cgi-bin/ Request-rate: 1/5 # maximum rate is one page every 5 seconds Visit-time: 0001-2358 # only visit between 00:01 and 23:58 UTC (GMT)