Andrew Connell [MVP MOSS]
1543 Posts |  42 Articles |  4753 Comments
.NET  |  MCMS  |  SharePoint  |  Office System
SharePoint Quick Links
Article Categories
Archives
Post Categories


Add to Technorati Favorites

[via Enterprise Search Blog]

This entry is cross posted from my guest blog entry on the Microsoft SharePoint Enterprise Search Blog.

One of the more discussed topics I’ve seen (and struggled with myself) is around the concept of obtaining the total number of hits in a search results when working with the MOSS API. For instance, when I search for “sales forecast” in my SharePoint site, I want to not only see a set of paged results, 10 hits per page, but also see that my search found 127 matches. Those of you who’ve worked with the SharePoint Search Web Parts know this is a piece of cake using the Search Core Results, Search Paging and Search Statistics Web Parts.

But what if you need to roll your own solution? How can you get the same data out of your search query using the MOSS 2007 Search API? At first this can be a bit tricky but this post will hopefully show you how to knock it down to being a trivial task.

Executing a search query against the SharePoint API has you working with two objects that implement the abstract class Microsoft.Office.Server.Search.Query.Query: Microsoft.Office.Server.Search.Query.KeywordQuery and Microsoft.Office.Server.Search.Query.FullTextSqlQuery. The former KeywordQuery is useful for simple queries whereas the latter FullTextSqlQuery is much more powerful. Both implement the Execute() method which executes the defined query and returns back a collection of results as type Microsoft.Office.Server.Search.Query.ResultTableCollection. Using this object, you can get the specific results you are interested in. For instance to get the relevant results use the following to get an instance of a specific Microsoft.Office.Server.Search.Query.ResultTable:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))
{
  query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",
  ResultTableCollection results = query.Execute();
  ResultTable relevantResults = results[ResultType.RelevantResults];
  // do work with the results
}

Simple enough, but the project requires much more than that as usual. What we need to do is page the results to show only 15 items per page. No problem… let’s just modify that query a bit to set the Query.StartRow & Query.RowLimit properties of the query to say what page we’re on and tell SharePoint how many results we want to get back. Take for instance if we’re on page 2 of the results… we want to start with the 16th hit as 1-15 were on page 1:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))
{
  query.StartRow = 16;
  query.RowLimit = 15;
  query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc);
  ResultTableCollection results = query.Execute();
  ResultTable relevantResults = results[ResultType.RelevantResults];
  // do work with the results
}

Again… pretty straight forward. Now is where it gets a bit tricky. You need to show links to provide paging… but in order to do that you need a good idea what the total reset set of your search query because if there were only 43 hits, you don’t want to shot options to jump to page 9. The property that gives you the number you’re looking for is ResultTable.TotalResults. Now there’s something special about this guy: he doesn’t give you an exact number… he gives you an estimate. Why an estimate? Quite simply, with all the security trimming and other complex logic inherit to search algorithms, it’s just too expensive to get a specific number. Sites like Live.com can do this because they don’t have to concern themselves with the security trimming of hits.

But this is not all… there’s another property you should pay attention to: Query.TotalRowsExactMinimum. This property tells SharePoint this is the minimum number of hits to be included in the search. It’s used to generate the estimate of total results. Think of it like a hint to search… saying “you only have to work this hard on this query.” Most search implementations only show the next few paging options… they don’t show ALL the options. For instance, if you’re on page 5, your paging control may show the following:

«Previous« 2 3 4 5 6 7 8 »Next»

In this case, you don’t need for search to find ALL the results… you only need it to determine how many more page options you want to show to see if you’re going to show too many or too few. In the above example, you have an additional 3 pages of results you want to show. Continuing on this example, you have a result set of 15 and you have an additional 3 pages you want to show, the Query.TotalResultsExactMinimum property would be 45 as it already is going to factor into the equation the Query.StartRow property:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))
{
  query.StartRow = 16;
  query.RowLimit = 15;
  // TotalRowsExactMinimum = [number of pages to show] * [page size]
  query.TotalRowsExactMinimum = 45;

  query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc);
  ResultTableCollection results = query.Execute();
  ResultTable relevantResults = results[ResultType.RelevantResults];
  // do work with the results
}

That’s all there really is to it! One parting word of advice: use the Query.TotalRowsExactMinimum property with care as the higher its set, the greater performance impact there will be on each search query executed.

A special shout out & thanks to Puneet Narula @ Microsoft for helping uncover this very helpful nugget of info.

Technorati Tags: ,,,
posted on Monday, May 26, 2008 10:08 AM

Feedback

# re: Understanding Total Hits & Paging in the MOSS 2007 Search API 5/27/2008 5:03 AM Dirk
Gravatar Hi I am having some interesting behaviour with some of this code.

When I use the ORDER BY clause in my sql statement the results that I get back have any rows that have null for the value I ordered removed.

So if I am trying to retrieve a list of people that I know should come back with Mr Jones and Mr Smith, but Mr Smith does not have a telephone number and this is what I am ordering on (ORDER BY Telephone) Mr Smith is removed from the search results.

Is there a way around this?

Thanks

# re: Understanding Total Hits & Paging in the MOSS 2007 Search API 5/27/2008 11:12 AM Deepak Aggarwal
Gravatar Hi Andrew,

Nice Post!!!

We have implemented the same solution with in our custom search implemntation for one of your client. It is working fine and meeting all the requirmetns. But one thing we noticed that if we use Sharepoint Search API to implement Search, the Search Usage Reports donot get generated. In the usage reporting it shows nothing. But if we fire the same query using Sharepoint Search Control, we got those reports. We tried to find lot of API's but still not able to get the answers. As a workaround we had put the "SearchResultControlEx " webparts on the search results page and made it visible false.
Could you please tell me is there any API available which can be used in the code itself?

Regards
Deepak

# re: Understanding Total Hits & Paging in the MOSS 2007 Search API 6/5/2008 6:07 AM chris
Gravatar Hi, I'm using the Query web service.

Is there a way to retrieve something like the TotalResults property after executing a query with the QueryService.QueryEx method?


# re: Understanding Total Hits & Paging in the MOSS 2007 Search API 1/27/2009 8:28 AM R Thomas
Gravatar Dirk,

With respect to null values and ordering try this article:

http://blogs.msdn.com/varun_malhotra/archive/2008/08/16/moss-search-with-order-by-clause-doesn-t-return-all-results.aspx

# re: Understanding Total Hits & Paging in the MOSS 2007 Search API 1/29/2009 7:00 AM R Thomas
Gravatar The “TotalRowsExactMinimum” (trem) doesn’t work as described here. I ran some tests with a small subset of data (using a where condition to pull back a subset of known data) e.g.

Item01
Item02
Item03
Item04
Item05
Item06
Item07

If I run a query with startRows=0 and rowlimit=10 and trem=10 then I get all the resuls back.

However if I try to implement paging say rowlimit=2 and startrows=0 and trem=2 I don’t get back Item01 and Item02 (I get back Item06 and Item07). At first I thought this was just down to ordering, but as I incremented through the pages I found results being duplicted and some hits not coming back at all!

I am convinced this is a bug (perhaps similar to audiences and CQWP) where the trem is being applied too early before the paging. The only workaround I currently have is to ensure trem is set to total results (i.e. run query with no paging first to get this value).

Andrew – when moderating this I am happy to be proven wrong but I can’t see how this can be anyting else but a bug! Maybe your MS contact can confirm (or perhpas this is by design :( )


# re: Understanding Total Hits & Paging in the MOSS 2007 Search API 2/3/2009 4:16 AM Eddy Z.
Gravatar Hello Andrew,

Thank you for this very useful post! I've been struggling with FullTextSQLQuery in WSS for quite a time. Thanks to this I managed to expand our solution en build my custom pager control and give unlimited results, instead of a maximum number of results (the higher the number, the worse the performance). First I used RowLimit to limit the maximum number of results and bound it directly to a gridview with standard paging control. But now I know better!

Greetings from the Netherlands,

Eddy

Post Feedback

Title:
Name:
Email:
(email will not be displayed)
Url:
Comments: 
Please add 1 and 7 and type the answer here:    
All Comments Are Filtered & Moderated
Unfortunately comment spammers are just too effecient and are constantly dirtying up blogs with irrelevant and unwanted comments trying to improve their standing on search engines. All comments on this blog are moderated. I do not censor comments, but I don't approve comments with vulger language or those soliciting products. Most of the time comments are approved within a few hours of being submitted with the only exception when I'm traveling.

Why are you asking for my email address?
The only reason I'm asking for your email address, which isn't required to submit a comment, is to provide a gravatar if you've created an account for yourself and associated your email address with a small image. If you have a gravatar created for the email address you submit, it will appear next to your comment. Otherwise nothing will appear.

What is a gravatar?
A gravatar is a "globally recognized avatar." You can get more information about gravatars, as well as create your own for free, at www.gravatar.com. You can also view my gravatar here.


Copyright © 2003 - 2010 Andrew Connell
Creative Commons License 
This work is licensed under a Creative Commons License
Site design by Heather Solomon.

 
 
MOSS WCM Training
Looking for MOSS 2007 WCM developer training? Look no further! I teach my 5-day hands-on and online WCM classes for developers I offer through my company: Critical Path Training.

Get more information on the WCM courses!