# # robots.txt # # First, this is the # robots.txt from http://www.wikipedia.org/ for "other" robots # Thanks Wikipedians! # # Please note: There are a lot of pages on this site, and there are # some misbehaved spiders out there that go _way_ too fast. If you're # irresponsible, your access to the site may be blocked. # User-agent: Orthogaffe Disallow: / # Crawlers that are kind enough to obey, but which we'd rather not have # unless they're feeding search engines. User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # Sorry, wget in its recursive mode is a frequent problem. # Please read the man page and use it properly; there is a # --wait option you can use to set the delay between hits, # for instance. # User-agent: wget Disallow: / # # The 'grub' distributed client has been *very* poorly behaved. # User-agent: grub-client Disallow: / # # Doesn't follow robots.txt anyway, but... # User-agent: k2spider Disallow: / # # Hits many times per second, not acceptable # http://www.nameprotect.com/botinfo.html User-agent: NPBot Disallow: / # A capture bot, downloads gazillions of pages with no public benefit # http://www.webreaper.net/ User-agent: WebReaper Disallow: / #robots que estudian el seo de tu site User-agent: MJ12bot Disallow: / User-agent: spbot Disallow: / #robots que no se sabe que hacen user-agent: sitebot disallow: / #bloqueado el 19-12-12 seobot SISTRIX Crawler user-agent: Sistrix disallow: / # allow google image bot to search all images User-agent: Googlebot-Image Disallow: User-agent: Mediapartners-Google* Disallow: User-agent: * Disallow: /elvirtual/ Disallow: /copiadseguretat/ Disallow: /disseny/ Disallow: /inclusions/ Disallow: /utilitats/ Disallow: /conversaciones/admin/ Disallow: /conversaciones/backups/ Disallow: /conversaciones/cache/ Disallow: /conversaciones/Contrib/ Disallow: /conversaciones/db/ Disallow: /conversaciones/docs/ Disallow: /conversaciones/images/ Disallow: /conversaciones/includes/ Disallow: /conversaciones/language/ Disallow: /conversaciones/modules/ Disallow: /conversaciones/profilcp/ Disallow: /conversaciones/stats_mod/ Disallow: /conversaciones/templates/ Disallow: /foro/admin/ Disallow: /foro/backups/ Disallow: /foro/cache/ Disallow: /foro/Contrib/ Disallow: /foro/db/ Disallow: /foro/docs/ Disallow: /foro/images/ Disallow: /foro/includes/ Disallow: /foro/language/ Disallow: /foro/modules/ Disallow: /foro/profilcp/ Disallow: /foro/stats_mod/ Disallow: /foro/templates/ Disallow: /foro/foro/admin/ Disallow: /foro/foro/backups/ Disallow: /foro/foro/cache/ Disallow: /foro/foro/Contrib/ Disallow: /foro/foro/db/ Disallow: /foro/foro/docs/ Disallow: /foro/foro/images/ Disallow: /foro/foro/includes/ Disallow: /foro/foro/language/ Disallow: /foro/foro/modules/ Disallow: /foro/foro/profilcp/ Disallow: /foro/foro/stats_mod/ Disallow: /foro/foro/templates/ Disallow: /libro/bin/ Disallow: /libro/config/ Disallow: /libro/docs/ Disallow: /libro/extensions/ Disallow: /libro/images/ Disallow: /libro/includes/ Disallow: /libro/languages/ Disallow: /libro/locale/ Disallow: /libro/maintenance/ Disallow: /libro/math/ Disallow: /libro/serialized/ Disallow: /libro/skins/ Disallow: /libro/t/ Disallow: /libro/tests/ Disallow: /foro/wikifida/bin/ Disallow: /foro/wikifida/config/ Disallow: /foro/wikifida/docs/ Disallow: /foro/wikifida/extensions/ Disallow: /foro/wikifida/images/ Disallow: /foro/wikifida/includes/ Disallow: /foro/wikifida/languages/ Disallow: /foro/wikifida/locale/ Disallow: /foro/wikifida/maintenance/ Disallow: /foro/wikifida/math/ Disallow: /foro/wikifida/serialized/ Disallow: /foro/wikifida/skins/ Disallow: /foro/wikifida/tests/ Disallow: /planet/wp-admin/ Disallow: /planet/wp-content/ Disallow: /planet/wp-includes/ Disallow: /temas/wp-admin/ Disallow: /temas/wp-includes/ Disallow: /*/feed/ Disallow: /*/trackback/ Disallow: /*/category/ Disallow: /*tag=* Disallow: /inter_noticiesplus.php Disallow: /temas/elvirtual.php Disallow: /2007/0* Disallow: /2007/1* Disallow: /2008/0* Disallow: /2008/1* Disallow: /2009/0* Disallow: /2009/1* Disallow: /2010/0* Disallow: /2010/1* Disallow: /2011/0* Disallow: /2011/1* Disallow: /2012/0* Disallow: /2012/1* Disallow: /page/ Disallow: /atom Disallow: /rdf Disallow: /rss Disallow: /aviso-legal Disallow: /cookies