# For format, see http://info.webcrawler.com/mak/projects/robots/norobots.html User-agent: * Disallow: /users/ # this is just a link to /u Disallow: /projects/nfsv4/linux/kernel-patches/ # large and uninteresting Disallow: /projects/itss/admin/ Disallow: /projects/itss-all/ # no permissions! Disallow: /u/rwash # it no longer exists, but crawlers still try to go there Disallow: /u/itoi # no longer exists