#robots.txt #this file stops search engines from visiting certain pages #specifically list pages which are empty and #would interfere with getting the site listed in some search engines #note max number of characters is 2000 #see http://www.robotstxt.org/wc/exclusion.html for syntax User-agent: * Disallow: /english/listening/nc1c/ Disallow: /english/listening/nc2a/ Disallow: /english/listening/nc3plus/ Disallow: /english/reading/nc3plus/ Disallow: /english/writing/nc2a/ Disallow: /english/writing/nc3plus/ Disallow: /maths/number/nc1c/ Disallow: /maths/number/nc1b/ Disallow: /maths/number/nc1a/ Disallow: /maths/number/nc2c/ Disallow: /maths/number/nc2b/ Disallow: /maths/number/nc2a/ Disallow: /maths/number/nc3plus/ Disallow: /maths/shape/p4/ Disallow: /maths/shape/p5/ Disallow: /maths/shape/nc1a/ Disallow: /maths/shape/nc2c/ Disallow: /maths/shape/nc2b/ Disallow: /science/life/ Disallow: /artic/artic/preverbal/ Disallow: /artic/artic/prosody/ Disallow: /artic/artic/vowels/ Disallow: /otherareas/ Disallow: /tap/menu/workinprogress.html/ #allow the W3C link checker to see everything though User-Agent: W3C-checklink Disallow: