docbook-apps

Hi. I am fielding some questions about the search behavior in the
webhelp output. Is there an explanation of the behavior available
somewhere?

Specifically, I need to understand:

* How substrings are handled. Why does "locale" match "localeString"
but "crea" doesn't match "create"?

* Is there a way to search for strings that contain special characters
like periods. Can I search for "foo.bar" by escaping the period? Can I
remove the period from the list of special characters?

Thanks for your help. I have turned off stemming in case that matters.

Peter Desjardins

On 03/17/2011 02:51 PM, Peter Desjardins wrote:
> Hi. I am fielding some questions about the search behavior in the
> webhelp output. Is there an explanation of the behavior available
> somewhere?
>
> Specifically, I need to understand:
>
> * How substrings are handled. Why does "locale" match "localeString"
> but "crea" doesn't match "create"?
>
> * Is there a way to search for strings that contain special characters
> like periods. Can I search for "foo.bar" by escaping the period? Can I
> remove the period from the list of special characters?
>
> Thanks for your help. I have turned off stemming in case that matters.
>
> Peter Desjardins

Hi Peter,
Good point. That had occurred to me before, but then I forgot to put it
in the feature list, so I've just added the following to the feature
list[1]:

"Cause indexer not to tokenize punctuation if the punctuation isn't
followed by a space. So don't tokenize "build.xml" into "build" and
"xml". Instead just enter it as "build.xml". Evaluation how to handle
things that look like paths "foo/bar/baz.xml" Should we tokenize into
foo, bar, and baz.xml or not or put both in the index? "

So we should tokenize "build. xml" but not "build.xml". Unfortunately, I
can't suggest any workaround at the moment.

Thanks,
David

[1] http://docbook.xmlpress.net/tiki-index.php?page=WebHelp2011

Hi,

On Fri, Mar 18, 2011 at 1:21 AM, Peter Desjardins <peter.desjardins.us@
gmail.com> wrote:

> Hi. I am fielding some questions about the search behavior in the
> webhelp output. Is there an explanation of the behavior available
> somewhere?
>
> Specifically, I need to understand:
>
> * How substrings are handled. Why does "locale" match "localeString"
> but "crea" doesn't match "create"?
>

The stemmed root word of create/created/creating/creat is "creat", so all of
these words produce same output. Stemmed word of "crea" is "crea" itself
which is actually not a word! And, "localeString" matches to "localStr"
(Stemmers tend to remove suffixes such as -ing, -ed) and 'locale" matches to
"local". Were these produced same output for you?

You can check how it behaves by executing javascript command stemmer(string)
via Google Chrome's console or via FindBugs for Firefox.
Results:

stemmer("create")
"creat"
stemmer("crea")
"crea"
stemmer("localeString")
"localeStr"
stemmer("locale")
"local"


>
> * Is there a way to search for strings that contain special characters
> like periods. Can I search for "foo.bar" by escaping the period? Can I
> remove the period from the list of special characters?
>

As David said, this is something we need to fix. stemming does not play any
part here. i.e. stemmer("foo.bar") == "foo.bar". Here, foo and bar got
indexed separately.

Regards,
--Kasun


> Thanks for your help. I have turned off stemming in case that matters.
>
> Peter Desjardins
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: docbook-apps-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: docbook-apps-help@lists.oasis-open.org
>
>


--
~~~*******'''''''''''''*******~~~
Kasun Gajasinghe,
University of Moratuwa,
Sri Lanka.
Blog: http://kasunbg.blogspot.com
Twitter: http://twitter.com/kasunbg