Optimizing/Customizing Sharepoint Search Crawling

- 1 answer

With SharePoint Server 2007, there is also a Search Feature and a Crawler. However, the Crawler is somewhat limited in that it only supports Basic Auth when crawling external sites and that there is no way to tell it to ignore no-index,no-follow attributes.

Now, there is a site i'd like to index, unfortunately this site uses it's own Authentication System, and it uses no-index,no-follow on the pages. As I control that site, i can remove the Attributes, but it's a PITA to do so. Also, it does not solve the Authentication issue.

So I just wonder if it's possible to extend Sharepoint's Crawler somehow?



The limitation of MOSS crawling sites with different forms authentication should have been addressed in MOSS SP1. :

Here's a link to a post which describes how to get the hotfix for pre-SP1 MOSS to enable the crawling of sites with forms authentication:

Hope that helps!