Post series written by Matthew Lynch (Senior Consultant at Charteris specialising in eCommerce). Before joining Charteris Matthew worked for IBM on various eCommerce enablement projects across multiple industries from the late 1990’s through to the present day.
This series of blogs is about how product managers in an eCommerce business can influence and optimise the customer search experience on their website by tuning the on-site search engine.
The search engine doesn’t know what it doesn’t know
You may already have a facility that helps you improve on-site search for your customers.
It should have been setup with an initial “dictionary” to make sure that it works well at go live.
But, after go-live, how can you make sure it continues to work for you? After all, you’ve got new products coming on-line all the time and you want to ensure your customers can find them.
Search engines only know what they’re told
Without manual intervention the search engine is only as good as the in-built auto-correction function (to deal with typos, mis-spellings), the default dictionary, any content that it knows about and ‘domain-specific’ dictionaries which, if vendor supplied, are likely to be quite generic.
Search tools are not necessarily aware of common mis-spellings, typo’s, domain-specific acronyms / phrases, what customers will actually search for in real life or other common language in the industry sector. They will also not be aware of specific key phrases or made-up product names that exist in the product catalogue that have specific relevance.
Also, language use evolves over time, sometimes quite quickly (often as a result of creative marketing or journalistic minds), so there’s no way that a search engine is going to keep up-to-date without human intervention.
Regular search tuning improves the bottom line
Search tuning enhances the relevancy of results, reduces the number of zero results found and helps identify application configuration and design changes that may be required. The process of search tuning is also another way of keeping you in touch with customer trends. All of these things help to improve the customer experience ultimately leading to improved conversion, basket size and customer loyalty.
The business, not IT is best placed to perform search tuning
If your business is not tuning its on-site search engine, then it may not know how many customers are having trouble searching for your products on your site. Many people assume that the search engine belongs to and is maintained by IT. You might think that your SEO strategy and keyword-promotions ensure that customers can find your products using Google, Yahoo etc, but if you’ve got any direct-loaders (people who go straight to your site and then use your site’s search facility) then they need their searches to be looked after by the business, the guys who understand the products and how they are marketed and sold to the customer.
Consider who owns the customer relationship. The business doesn’t get IT guys to write the product titles, descriptions, retail reviews or marketing copy, so why should the business rely on IT, when search engines deal with words that customers are looking for in the very text that has been created by the business?
Use analytics to identify search tuning needs
In eCommerce, many people will have come across a “top-search-terms” report from the web-analytics tool. This, generated regularly, can form the basis for search tuning. For best use, it needs to report the popularity of search terms together with the average number of results returned and needs to show search terms with zero results. In addition, it is useful to have a report showing you search terms that have been autocorrected.
Tuning affects which words and phrases the search engine considers or ignores
Search tuning involves telling the search engine how to handle words and phrases that it may not ‘understand’ or that it may assign more or less relevance to than desired by the business.
Stop words
In most circumstances, there’s no point in a search engine searching for words that are so common that they appear in large numbers of product titles or descriptions. It results in too many, often irrelevant, results for the customer and wastes the search engine’s resources in matching them. Typically these words include ‘a’, ‘an’, ‘of’, ‘the’ ‘i’, ‘is’ etc but the business really ought to work out for itself which words are so common that the search engine should ignore them. This may include some words that are core to the product set being sold e.g. a entertainment retailer might get the search engine to ignore DVD, CD etc. Be wary of adopting the standard set provided by the search tool vendor. For example, a ‘stop word list’ provided by a search tool vendor may include the word ‘that’, but for a CD retailer, this would not be an appropriate stop word if it meant that it made searches for CDs by ‘Take That’ of lower relevance than results that don’t contain the word ‘that’. Try it, you’ll find there’s a band called “Take”.
Synonyms
It is estimated that 20-30% of all search terms used on the internet contain a mis-spelling or a typo. If that is the case and your search engine doesn’t manage to autocorrect them or the typo is on a made-up word such as a brand or product name, then your customers will get zero results, the dreaded dead-end.
The solution; find out the terms that customers are using that return zero or few results and setup a synonym to the term that does match products.
Some examples where you would use synonyms include :
- Numbers – equate 2 with ii and two
- Acronyms – to equate product-acronyms with the full product title
e.g. LOTR -> Lord of the Rings
- Common mis-spellings e.g. cart <-> kart
- Made-up product titles with spaces taken out e.g. (wii) motion plus = motionplus
- UK / US spellings (if not auto-corrected) e.g. color <-> colour, metre <-> meter,
Enforced or Automatic Phrasing
If your customers are getting too many results for certain search phrases, of which you find that many are irrelevant, then you should instruct your search engine to only consider results where there is an exact match to the phrase (words and their order).
e.g. "32gb ipod touch" - assuming you didn’t want the search engine to return 4/8/16gb ipods or ipod nanos in the results set)
Stemming
Good search engines take verbs, adjectives and adverbs that are used in a search term and condense them to their ‘root’ form or ‘stem’ (e.g. running -> run, richer -> rich, strongly -> strong) before comparing them to the content to be searched which has also been ‘stemmed’. Search engines often come with stemming built-in but if problems are identified using the search analytics, then additional stemming ‘synonyms’ or rules will need to be added. The process of getting to a word stem is language dependent and may require the skills of an experienced linguist if adjustments need to be made.
Search keyword / phrase Redirects
Sometimes it is desirable to redirect a customer to a hub or landing page that makes the best representation of a product or product set that the customer is looking for. This gives the customer a better experience than the presentation of a product list (which may well not show the full range of potential product type matches on page 1).
It is a common occurrence for customers to search for types of products e.g. Romance films, so a redirect for such types of search would be a good idea.
It is good practice to monitor for new occurrences of such searches and setup a search term redirect for the phrases used. Good search engines provide a facility for recording such redirect phrases with the desired destination such that the search engine provides redirect URL destinations to the eCommerce website when such phrases are matched.
In my next post we will look at what’s desirable to look for in people who will perform search tuning and how the tuning exercise can be organised
---------------------------------------------------
Contact – matthew.lynch@charteris.com
This series of blogs is about how product managers in an eCommerce business can influence and optimise the customer search experience on their website by tuning the on-site search engine.
The search engine doesn’t know what it doesn’t know
You may already have a facility that helps you improve on-site search for your customers.
It should have been setup with an initial “dictionary” to make sure that it works well at go live.
But, after go-live, how can you make sure it continues to work for you? After all, you’ve got new products coming on-line all the time and you want to ensure your customers can find them.
Search engines only know what they’re told
Without manual intervention the search engine is only as good as the in-built auto-correction function (to deal with typos, mis-spellings), the default dictionary, any content that it knows about and ‘domain-specific’ dictionaries which, if vendor supplied, are likely to be quite generic.
Search tools are not necessarily aware of common mis-spellings, typo’s, domain-specific acronyms / phrases, what customers will actually search for in real life or other common language in the industry sector. They will also not be aware of specific key phrases or made-up product names that exist in the product catalogue that have specific relevance.
Also, language use evolves over time, sometimes quite quickly (often as a result of creative marketing or journalistic minds), so there’s no way that a search engine is going to keep up-to-date without human intervention.
Regular search tuning improves the bottom line
Search tuning enhances the relevancy of results, reduces the number of zero results found and helps identify application configuration and design changes that may be required. The process of search tuning is also another way of keeping you in touch with customer trends. All of these things help to improve the customer experience ultimately leading to improved conversion, basket size and customer loyalty.
The business, not IT is best placed to perform search tuning
If your business is not tuning its on-site search engine, then it may not know how many customers are having trouble searching for your products on your site. Many people assume that the search engine belongs to and is maintained by IT. You might think that your SEO strategy and keyword-promotions ensure that customers can find your products using Google, Yahoo etc, but if you’ve got any direct-loaders (people who go straight to your site and then use your site’s search facility) then they need their searches to be looked after by the business, the guys who understand the products and how they are marketed and sold to the customer.
Consider who owns the customer relationship. The business doesn’t get IT guys to write the product titles, descriptions, retail reviews or marketing copy, so why should the business rely on IT, when search engines deal with words that customers are looking for in the very text that has been created by the business?
Use analytics to identify search tuning needs
In eCommerce, many people will have come across a “top-search-terms” report from the web-analytics tool. This, generated regularly, can form the basis for search tuning. For best use, it needs to report the popularity of search terms together with the average number of results returned and needs to show search terms with zero results. In addition, it is useful to have a report showing you search terms that have been autocorrected.
Tuning affects which words and phrases the search engine considers or ignores
Search tuning involves telling the search engine how to handle words and phrases that it may not ‘understand’ or that it may assign more or less relevance to than desired by the business.
Stop words
In most circumstances, there’s no point in a search engine searching for words that are so common that they appear in large numbers of product titles or descriptions. It results in too many, often irrelevant, results for the customer and wastes the search engine’s resources in matching them. Typically these words include ‘a’, ‘an’, ‘of’, ‘the’ ‘i’, ‘is’ etc but the business really ought to work out for itself which words are so common that the search engine should ignore them. This may include some words that are core to the product set being sold e.g. a entertainment retailer might get the search engine to ignore DVD, CD etc. Be wary of adopting the standard set provided by the search tool vendor. For example, a ‘stop word list’ provided by a search tool vendor may include the word ‘that’, but for a CD retailer, this would not be an appropriate stop word if it meant that it made searches for CDs by ‘Take That’ of lower relevance than results that don’t contain the word ‘that’. Try it, you’ll find there’s a band called “Take”.
Synonyms
It is estimated that 20-30% of all search terms used on the internet contain a mis-spelling or a typo. If that is the case and your search engine doesn’t manage to autocorrect them or the typo is on a made-up word such as a brand or product name, then your customers will get zero results, the dreaded dead-end.
The solution; find out the terms that customers are using that return zero or few results and setup a synonym to the term that does match products.
Some examples where you would use synonyms include :
- Numbers – equate 2 with ii and two
- Acronyms – to equate product-acronyms with the full product title
e.g. LOTR -> Lord of the Rings
- Common mis-spellings e.g. cart <-> kart
- Made-up product titles with spaces taken out e.g. (wii) motion plus = motionplus
- UK / US spellings (if not auto-corrected) e.g. color <-> colour, metre <-> meter,
Enforced or Automatic Phrasing
If your customers are getting too many results for certain search phrases, of which you find that many are irrelevant, then you should instruct your search engine to only consider results where there is an exact match to the phrase (words and their order).
e.g. "32gb ipod touch" - assuming you didn’t want the search engine to return 4/8/16gb ipods or ipod nanos in the results set)
Stemming
Good search engines take verbs, adjectives and adverbs that are used in a search term and condense them to their ‘root’ form or ‘stem’ (e.g. running -> run, richer -> rich, strongly -> strong) before comparing them to the content to be searched which has also been ‘stemmed’. Search engines often come with stemming built-in but if problems are identified using the search analytics, then additional stemming ‘synonyms’ or rules will need to be added. The process of getting to a word stem is language dependent and may require the skills of an experienced linguist if adjustments need to be made.
Search keyword / phrase Redirects
Sometimes it is desirable to redirect a customer to a hub or landing page that makes the best representation of a product or product set that the customer is looking for. This gives the customer a better experience than the presentation of a product list (which may well not show the full range of potential product type matches on page 1).
It is a common occurrence for customers to search for types of products e.g. Romance films, so a redirect for such types of search would be a good idea.
It is good practice to monitor for new occurrences of such searches and setup a search term redirect for the phrases used. Good search engines provide a facility for recording such redirect phrases with the desired destination such that the search engine provides redirect URL destinations to the eCommerce website when such phrases are matched.
In my next post we will look at what’s desirable to look for in people who will perform search tuning and how the tuning exercise can be organised
---------------------------------------------------
Contact – matthew.lynch@charteris.com
Posts
