* robots.txt
@ 1999-03-09 15:04 Gerald Pfeifer
[not found] ` < Pine.GSO.4.10.9902150100390.9254-100000@alphard.dbai.tuwien.ac.at >
1999-03-31 23:46 ` robots.txt Gerald Pfeifer
0 siblings, 2 replies; 4+ messages in thread
From: Gerald Pfeifer @ 1999-03-09 15:04 UTC (permalink / raw)
To: Jason Molenda, egcs
I noticed that we don't (yet) have a robots.txt file for the egcs web
server.
How about the following for now?
User-Agent: *
Disallow: /cgi-bin/
If there are no objections, I'll commit that in a couple of days...
Gerald
--
Gerald Pfeifer (Jerry) Vienna University of Technology
pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/
^ permalink raw reply [flat|nested] 4+ messages in thread[parent not found: < Pine.GSO.4.10.9902150100390.9254-100000@alphard.dbai.tuwien.ac.at >]
* Re: robots.txt [not found] ` < Pine.GSO.4.10.9902150100390.9254-100000@alphard.dbai.tuwien.ac.at > @ 1999-03-11 14:11 ` Gerald Pfeifer 1999-03-31 23:46 ` robots.txt Gerald Pfeifer 0 siblings, 1 reply; 4+ messages in thread From: Gerald Pfeifer @ 1999-03-11 14:11 UTC (permalink / raw) To: Jason Molenda, egcs; +Cc: egcs-patches On Wed, 10 Mar 1999, Gerald Pfeifer wrote: > How about the following for now? I have now installed the following robots.txt file for our egcs.cygnus.com site as wwwdocs/htdocs/robots.txt. ---- cut ---- # See http://info.webcrawler.com/mak/projects/robots/norobots.html # for information about the file format. # Contact egcs@egcs.cygnus.com for questions. User-Agent: * Disallow: /cgi-bin/ ---- cut ---- Thanks to Jason for his comments... Gerald -- Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/ ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: robots.txt 1999-03-11 14:11 ` robots.txt Gerald Pfeifer @ 1999-03-31 23:46 ` Gerald Pfeifer 0 siblings, 0 replies; 4+ messages in thread From: Gerald Pfeifer @ 1999-03-31 23:46 UTC (permalink / raw) To: Jason Molenda, egcs; +Cc: egcs-patches On Wed, 10 Mar 1999, Gerald Pfeifer wrote: > How about the following for now? I have now installed the following robots.txt file for our egcs.cygnus.com site as wwwdocs/htdocs/robots.txt. ---- cut ---- # See http://info.webcrawler.com/mak/projects/robots/norobots.html # for information about the file format. # Contact egcs@egcs.cygnus.com for questions. User-Agent: * Disallow: /cgi-bin/ ---- cut ---- Thanks to Jason for his comments... Gerald -- Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/ ^ permalink raw reply [flat|nested] 4+ messages in thread
* robots.txt 1999-03-09 15:04 robots.txt Gerald Pfeifer [not found] ` < Pine.GSO.4.10.9902150100390.9254-100000@alphard.dbai.tuwien.ac.at > @ 1999-03-31 23:46 ` Gerald Pfeifer 1 sibling, 0 replies; 4+ messages in thread From: Gerald Pfeifer @ 1999-03-31 23:46 UTC (permalink / raw) To: Jason Molenda, egcs I noticed that we don't (yet) have a robots.txt file for the egcs web server. How about the following for now? User-Agent: * Disallow: /cgi-bin/ If there are no objections, I'll commit that in a couple of days... Gerald -- Gerald Pfeifer (Jerry) Vienna University of Technology pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/ ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~1999-03-31 23:46 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
1999-03-09 15:04 robots.txt Gerald Pfeifer
[not found] ` < Pine.GSO.4.10.9902150100390.9254-100000@alphard.dbai.tuwien.ac.at >
1999-03-11 14:11 ` robots.txt Gerald Pfeifer
1999-03-31 23:46 ` robots.txt Gerald Pfeifer
1999-03-31 23:46 ` robots.txt Gerald Pfeifer
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).