Google has added another option to their Robots Exclusion Protocol (REP). Website owners now have the ability to flag webpages to be dropped out of the Google index on a specific date and time. This additional flexibility will allow website owners to expire content that has a specific shelf life and will be useful on websites that run specials online, sell products that are seasonal in nature, and so on. An example of the new tag and its syntax is included below. This example is the same as seen on Google’s Official Blog.
Archive for robots.txt
-
Unavailable_After Tag Now Available
Jason Barrett
29Aug
-
Yahoo!’s Content Filtering
Terri Greene
Yahoo recently introduced the ‘robot-no content’ tag which allows users to specify what content they do not want the search engine spiders to crawl due to irrelevance. This ensures that the search engine crawler will only look at relevant information. This will help bridge the gap of having a website that caters to users or search engines. With this content filtering, it could be possible to have a website that does both. This new concept will help webmasters in optimizing their sites better. Sometimes, there’s content on the site that isn’t quite relevant to the main content of the site; for instance, privacy statements, mission statements, terms and conditions, etc. Instead, you can get the Yahoo search engine crawler to only look at the content in which you would want them to by applying this ‘robot-no content’ tag to certain parts of the page that indicate to Yahoo’s search engine crawler what parts of the page are unrelated to the main content and are only useful to visitors.
Read the full article…22Jun
-
SES 2007 | NY | Day Two
Andrew Doak
Either there are a lot more people here today or I was in the wrong sessions yesterday. I actually had to sit on the floor (cross legged) along with a packed group listening about Duplicate Content & Multiple Site Issues. Now I know what you are thinking “should have gotten there earlier and not 10 minutes into the presentations…” I was busy taking photos of all the participants checking out our very cool booth. If you are reading this every morning to stay up to date on all the happenings at SES 2007, and you are actually here stop by and check us out – booth 2014.
11Apr
