I was wondering if it was enough for me to just add a file in my WWW folder named robots.txt (with this inside it:

User-agent: *
Disallow: /

) to prevent Google and other search-bots from indexing my site? Or does this require a www.brandeis.edu site admin to edit a robots.txt file in a root of brandeis.edu to have this achieved?

>according to Google, the robots.txt needs to be placed in the root of the site.

>my current location (perhaps useless) is http://people.brandeis.edu/~wrussian/robots.txt

>if it does require one of the admins to add my ID to the robots.txt file on brandeis.edu, could you please do so?

Thank you.

If you like, I can add your site to the global robots.txt file on people.brandeis.edu. This may not achieve the effect you're looking for, though. While GoogleBot and other well-behaved robots will read the file and respect its contents, there are many, many nefarious bots operating on the web today. These bots will not only ignore the contents of the file, they will specifically use it to index directories that aren't linked from anywhere.

Furthermore, Google and company won't necessarily stop listing your site because robots.txt has changed. They may simply stop crawling it looking for changes, and you'd have to wait months for it to expire from their respective caches.

In my opinion, the best way to keep a low profile is to not publish links to the site anywhere else on the web. If the search bots don't know about your site, they can't index it. To reverse the damage already done, each search engine has its own process for getting pages removed from its index. You'll orobably have to do each one individually. Unfortunately, getting content out of search engines is simply not an easy process.

I have never linked to my Brandeis Webspace from anywhere else. I would put my name on a homework, and now if i Google my name, this homework comes up. As i understand it, this is the role of a bot, to crawl the web and search for data.

Thanks for the interesting reading. I will look at it in depth after the finals are over :-)

That seems like a bad place for them unless they are meant to be totally public. Hiding them from robots with robots.txt will not change the fact that they are readable by anyone with a UNet account.

That's one of the (many) reasons Brandeis offers a course management system: collaboration amongst students in a class is a good thing, but making developmental and formative assignments available for the world to see for all of time is generally a bad thing.

I couldn't figure out for sure where Google got your page from, but here's an interesting page. If this page came before Google indexed your site, then probably Google got it from there. This is also potentially a good example of the kind of malicious robots you have to worry about. The people who run medical-papers.com seem to be interested in ripping off people's content for advertising money. If they run their own spider, there's no reason they would want it to respect robots.txt.

this is why i thought it would be good to have a robots file. but it makes it difficult for me to add this file, since Elliot has to do it, and also it make it difficult for me to remove already existing indexes, since i again have to be able to access the root.

i don't even have a slightest clue why, my .css would be there. i guess their bots searched all .edu domains for hw5.css

what would you recommend to be the best way to password protect the files, or even the whole folder? should i create a rar of my files?

Some instructions for passwording a directory in your UNet webspace are at

http://my.brandeis.edu/bboard/q-and-a-fetch-msg?msg_id=00004w

and

http://www.brandeis.edu/webservices/htaccess.html

Thanks everyone.

I am going to look into the webmaster tools at Google to see how I could remove my pages from their index.

Quick question, if now most of my webspace is protected by the .htaccess file, would Google consider my site as a source for indexing, even though they can not read it? (meaning, if say they have cached a file/folder and now they can not read it, would it be considered as 401 and be removed in like 6 months?)

Again, thank you everyone for such useful responses.

Also, you might have personal info in essays that perhaps you do not want revealed to anyone who googles your name or user id.

Finally, future employers might find something they would pass judgment upon, you just never know.

thats my take. Use good judgment on what needs to be private and what is a good resource to share.

Jonathan: I might be worried about the possibility of someone stealing your work, submitting it as their own and potentially getting you in trouble as being complicit in cheating. That shouldn't happen if the authorities involved have a clue, but that's not always the case.

Depending on the quality of the work, it might actually improve your chances with employers. They're not just looking for bad stuff on the web to disqualify applicants, but also good stuff to make people stand out more.

Matthew: As Kiril points out, you can do that with htaccess files. In addition to the links he posted, you might also want to check out the full documentation courtesy of Apache.

Would it also work to use the .htaccess file to grant access to a directory only if the IP falls within Brandeis's range?

Jonathan wrote:

What are everyone's thoughts about posting assignments and papers online