Blogs1 - 6 of 6 recent posts for tag:"robots txt"
13
Nov
2011
Should I block Googlebot from crawling a dynamically generated calendar?

90 days ago by Dustin Salmons

Should I deny an access to a dynamically generated vBulletin calendar via robots.txt? Whenever I open my Who's Online?" page, I can always se...

Brand Personalization - brandpersonalization.com · 2 references

If I don’t need to block crawlers, should I create a robots.txt file?

90 days ago by Dustin Salmons

If I don't need to block crawlers, should I create a robots.txt file? Is it better to have a blank robots.txt file, a robots.txt that contains Us...

Brand Personalization - brandpersonalization.com · 2 references

08
Nov
2011
Bing Webmaster Tools Update: Deeper Crawl Details and More // technology

96 days ago by Klaas

Bing is rolling out a long list of new features and improvements for its Webmaster Tools offering. The updates focus on providing webmasters with the ability to "share more data" with the search engine and that's exactly what it has done. As Bing closes in on 30 percent search market share, many of ...

Joseph Klaas - josephklaas.com · Rank: 70,551 · 2 references

28
Aug
2011
Robots.txt – A Real Helper When Optimizing Your Site // technology

167 days ago by Alfredo

Today I’ll show you how to use the file robots.txt. But first, let’s get why, in fact, we need the file robots.txt. The robots.txt file is to specify the search of search engines which files and folders you do not want to be indexed. It should be located in the root directory of a site. Very small h ...

Home Business Consulting - home-biz-consulting.com · 22 references

17
Jul
2011
List of search engines

209 days ago by admin

For good results from search engines, your web pages more important than achieving a higher ranking in search results. How can this be? The main factor that search engines depends on the structure of the links on your website. By following a few simple rules, your most important web pages receive th ...

10
May
2011
Ignore SEO, GoogleBot Will Sort it All Out for You // technology

278 days ago by Klaas

How to Handle Duplicate Content Here is a fun webmaster help video from March 10th of 2010, answering the following question: "If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using ...

Joseph Klaas - josephklaas.com · Rank: 70,551 · 2 references