Författarna försöker klustra sökloggar från AOL med hjälp av en bipartit graf mellan söksträngar och adresser och K-meansklustring i de resulterande komponenterna.

Metod

Metoden innehåller fyra steg:

1. Bygg en bipartit med söksträngar och adresser, där en kant finns om ensökning har föranlett ett klick till adressen.
2. Elimination av skräpkanter, för att kunna få ut komponenter.
3. Dimensionreducering av grannmatrisern i komponentenerna.
4. Klustring med K-means i komponenterna.

Resultat

Testdata är framtagen genom att plocka 40 k slumpvisa sökningar. Dessa klassificeras manuellt till 7 olika kategorier.

Efter de första stegen med den bipartita grafen finns det bara en komponent som är så stor att den är intressant att gå vidare med, och den innehåller 55 % av datan. Den komponenten klustras till fyra delar med K-means.

Resultaten utvärderas genom att precisionen för olika ämnen kontrolleras i varje kluster. Tre av fyra kluster hade något ämne som med betydligt högre precision än de andra.

Relevans för oss

Att klustra i AOL-loggarna är ju precis vad vi håller på med, så det är intressant att se hur de lyckas med. De har dock nöjt sig med att försöka passa in några få jättekluster i förutbestämda kategorier, något som vi bedömmer som ointressant. Något som däremot är intressant är att de har lagt ner ett stort arbete på att manuellt klassificera 40 k sökningar i sju kategorier, vilket ger oss en bild av hur proportionerna mellan dessa borde vara i våra kluster, det är möjligtvis något vi skulle kunna använda för utvärdering.

Att deras initiala steg med komponenter i en bipartit graf bara gav en stor komponent stämmer väl med våra erfarenheter av loggarna,  att det är skräpiga data som är svåra att slå isär.

Tracking

Två urval av träningsdata väljs ut, ett som innehåller artiklar om ämnet man är ute efter och ett som inte gör det.

Val av ämneskännetecken

För att kunna göra en cosine-jämförelse mellan ett ämne och en artikel behöver man ta fram ord som kännetecknar ämnet från artiklar i träningsdata. Författarna försökte med fyra olika metoder:

1. Ta alla ord i artiklarna.
2. För alla artiklar tilsammans, ta ut de n vanligaste orden.
3. För varje artikel, ta ut de n vanligaste orden.
4. Som 3, men man lägger iterativt till fler termer om de ger bättre resultat.

Metod fyra gav bäst resultat, men man valde metod tre eftersom den var marginellt sämre och betydligt mindre komplicerad.

Normalisering

Författarna försökte på olika sätt normalisera vektorn som representerar ämnet med hjälp av träningsdata, men gav upp det eftersom resultaten inte blev bättre.

Resultat

Bäst resultat verkade enkla metoder ge och författarna anser att deras resultat är konkurrenskraftiga.

Detection

Man försöker upptäcka ämnen med hjälp av algoritmen Single-linkage clustering och samma likhetsmått som tidigare. Algoritmen ger problematiska kedjefenomen, olika ämnen hålls ihop i samma kluster av enskilda artiklar som behandlar båda ämnena. Författarna verkar vara missnöjda och skyller resultaten på algoritmens brister.

In particular, the best way to find a frequently sought information goal in a search engine is to employ a frequently used query. For example, a good way to find http://www.webmd.com may be to issue the query webmd. In the authors’ study this was a fairly frequent query and information goal. Using the more frequent query medical questions page would yield more results making it more difficult to find the correct website. On the other hand, the much less frequent webmb would probably require the user to perform spelling correction before finding what he or she was looking for.

Note the connection between information goal and website. In their study, the authors have chosen to make these synonymous. Of course this need not be so. It may be true that it is more difficult to get to http://www.webmd.com by using medical questions page than by using webmb, but if the user wasn’t explicitly looking for http://www.webmd.com but instead was interested in finding the answer to a medical question then medical questions page could have been a better query to issue. This is not reflected in the study.

After having submitted a query the action of the user varies depending on the frequency of the query. Frequent queries more often lead to a click in the result list than do rare queries. Rare queries are more often followed by a requery than do frequent queries. Having clicked a frequently clicked result, the chance of clicking another result is smaller than if the clicked result had been rare. Conversely, the likelihood of a user issuing a requery after having clicked a rare result is greater than if he or she had clicked a frequent result.

According to the study, search engines are much better at finding answers to common information goals than they are at resolving rare goals. At first glance this seems reasonable enough.  A successful search engine is one which quickly finds the information that a user requires. Finding what most people want is almost as good as finding what everybody wants. The evidence of this is that for rare information goals, users typically need to reformulate their queries more times than they do for frequent goals.

To sum up, the paper implies that the frequency of the queries and clicked links in a session can help us discern to what degree the user was able to find what he or she was looking for.

Tillvägagångssätt

Klustringsprinciper

1. Om två sökningar innehåller samma eller liknande sökord så representerar de samma, eller liknande, informationsbehov.
2. Två sökningar är lika om de leder till vala av samma eller liknande dokument.

Princip ett räcker inte själv eftersom samma sökord kan representera olika behov.  Beräknad likhet innebär inte alltid semantisk likhet, detta gäller särskilt för korta söksträngar.

Princip två har svagheten att användare inte nödvändigtvis bara klickar på relevanta dokument och ett dokument kan innehålla information om flera ämnen.

Implementation

Data

Ur stora mängder loggar tar man ut sessioner som består av en söksträng och de dokument-klick som sökningen gav upphov till.

Algoritm

Man anser sig behöva en algoritm som klarar stora datamängder, inte kräver förutbestämt antal kluster osh som sorterar bort skräp. Valet faller på DBSCAN som har komplxitetetn O(nlogn).

Likhetsmått

För jämförelse av söksträngar kan man använda termlikhet (exempelvis cosine) eller stränglikhet (exempelvis Levenshtein).

Klicklikhet kan göras baserad på överlappande dokument, jämförbart med termlikheten, eller med hjälp av de kategorier som de klickade dokumenten tillhör.

Kombination av olika mått

Genom att kombinera olika mått kan man få bättre resultat än med ett enskilt mått. Man måste då välja parametrar för viktning av olika mått.

Utvärdering

Ur en månads loggar från Encarta, 22 GB, 2,7 M sessioner väljs 20 000 sessioner slumpmässigt för utvärdering.

Fyra olika kombinationer av likhetsmått används:

1. Termlikhet
2. Dokumentöverlapp
3. Termer kombinerat med dokumentöverlapp
4. Termer kombinerat med dokumentkategorier

Resultaten jämförs genom att titta på antal kluster och andel klustrade sessioner, med varierande likhetskrav.

Kvalitet

Kvalitetsutvärderingen sker genom att manuellt granska 100 slumpmässigt utvalda kluster.

Resultat

Resultaten påvisar fördelar med kombinationsmåt, särskilt gällande kvalitet. Kombinationsmåtten ger fler kluster och större andel klustrade sessioner, men karar inte lika höga likhetskrav som de enkla måtten.

Relevans för oss

Frågeställningarna är i stor utsträckning samma som vi ställs inför, men svaren inte alltid riktigt lika, förmodligen beroende på skillnader i datakvalitet.

Vi skulle inte kunna använda sessioner bestående av bara en söksträng och eventuella klick, de skulle bli för små och omöjliga att klustra. Jag tror inte heller att det är så internetanvändare använder en sökmotor, jag tror mer på modellen med flera sammanhängande frågor i jakt på information (Om det inte är en navigationssökning, men då är det ändå ointressant).

Vi har inte lika välformade dokument i klicklistorna och inga kategorier för dokumenten. Internet är väldigt stort och betydligt spretigare än Encarta och vi har bara tillgång till de domäner klicken leder till, inte exakta dokument. Våra metoder blir därför betydligt trubbigare.

Jag upplever att 20 000 sessioner är en ganska liten mängd att göra klustringar på, med det kan bero på att sökningar på Encarta är mer likriktade och möjliga dokument betydligt färre. De får förmodligen betydligt större överlapp även på små mängder sessioner.

Utvärderingsmetoderna är klart intressanta för oss när vi kommer dit förutsatt att vi inte själva kommer på något bättre. Kvalitetsmått genom manuell granskning av 100 kluster känns lite skakigt.

Det innebär att de parametrar som nu behöver sättas är startlikhet, stopplikhet och steglängd. Det sker även en utvärdering av klustringarna i varje steg som kan avgöra om man borde avbryta rekursionen innan stopplikheten är uppnådd, t.ex. på grund av klustrens storlek.

Resultaten ser hittills lovande ut. Ännu är ngen formell utvärdering utförd, men en stor andel av sessionerna klustras, och inget kluster blir så stort att det blir meningslöst. Ett exempel finns här.

Parallellt har Eskil utvecklat en klassifikation av sessioner så att vi kan slänga bort en stor del av de “navigational”-sessioner som vi är ganska ointresserade av. Men förhoppningsvis mer om det senare.

Parsing, session splitting and storage

We decided to cleanly separate the task of parsing AOL logs into sessions and the task of clustering the sessions. Although there were many reasons for this decision, the most important was that fetching sessions from a database seemed much more attractive than having to re-parse the log files every time we wanted to test our clustering algorithms.

The process of parsing the logs was fairly straight forward, although it is worth mentioning that they do contain some garbage and they don’t follow the bundled specification perfectly. Among other things we have assumed that multiple subsequent repetitions of a single query are in fact not independent queries but a single query with varying operations – liked next page, page refresh etc – performed on it.

When splitting sessions we’ve noted, as many others before us, that “time gap is not enough”. We’ve had the most success using a combination of time gap and Levenshtein measure. As a benchmark, we note where our splitter decides to split our test data. The algorithm inserts splits correctly roughly 75% of the time when compared to our gold standard.

Early on we played with the idea of storing sessions in a relational database like MySQL and have that run on a separate server. The performance was appalling and we decided to go with local instances of the flat Berkeley-DB database instead.

The whole process from reading logs to storing sessions now takes less than an hour. During this time roughly 20 million queries are combined into 8 million sessions.

Clustering

Our initial strategy when clustering was to use Bisecting K-means to partition the sessions. Unfortunately we had to rethink this when it became apparent to us that K-means would have trouble handling similarity based on such small amounts of text and URLs. Instead, we opted for the DBScan algorithm as Wen et al did in Query Clustering Using User Logs. We also hoped to use an existing implementation of the clustering algorithm and spare ourselves the work of implementing our own. Weka has a DBScan implementation but it doesn’t allow us the control over similarity that we require. Luckily the algorithm in itself isn’t very complicated and we managed to implement it ourselves without too much of a hassle.

While DBScan runs in n log n time, this must be multiplied by the time it takes to fetch all neighbours for a particular node. In our implementation this requires n^2 time. Initially we used a pre-calculated bit set for this purpose. This gave us run times comparable to Wen et al with 20000 sessions being clustered in the ballpark of 3 minutes. Unfortunately we have been forced to accept that clustering the full set of sessions is unfeasible due to time and memory constraints. This raises two questions that we will need to answer:

1. Is analysis of one or several samples enough to allow us to draw conclusions regarding the contents of the whole log? (Yes, probably).
2. How large a sample and how many of them do we need to ensure that the conclusion is reliable?

So far the results of clustering are difficult to understand. A single DBScan iteration with non-trivial parameters either discards 75% of the data or so as noise or has a single cluster, usually Google, Yahoo or MySpace in our samples, dominating all other clusters. We can’t seem to find a set of parameters that will allow us to include the noise and still keep a wide variety of clusters. This can be an indication that there isn’t a single set of parameters that is good for all concepts. Some concepts simply have a smaller level of internal similarity than do others. Here are the ideas we are working on right now to combat this problem:

• Incorporation of nominal phrases. New York City is the most frequent trigram in our data set, but we have yet to see a cluster describing it as a concept. Frequent trigrams probably indicate terms that should be treated as a whole in similarity calculations.
• Recursion. If one cluster dominates the others, why not attempt to split it by clustering the sessions in that cluster? This is what Bisecting K-means does. As a bonus of sorts this produces hierarchical results letting us identify Google Mail as a sub-cluster of Google.
• Excluding large amounts of data prior to clustering. We note that a large amount of sessions are what could be classed as navigational, where as we believe informational sessions hold the most interest. It is also probable that these two classes have different optimal parameters when clustering. Separating the two may result in better clustering results.

A query is considered navigational when a user has a particular Web page in mind and is primarily interested in visiting the page. Informational queries, on the other hand, refer to the queries where the user does not have a particular page in mind or intends to visit multiple pages to learn about a topic.

Incidentally there may be more than two categories. Previous research has had other categories as well, but these two are a good starting point that the researchers seem to agree on.

Is it possible to predict what category a query belongs to?

Lee, Liu and Cho set out to determine if it is possible to predict which of the two categories, navigational or informational, a query belongs to. Using a panel of 28 members of the UCLA Computer Science department they annotated those 50 most popular queries to Google that have been issued from within the department. The panel members voted on the best classification for each of the 50 queries. For most queries the panel was in agreement on classification, and the authors concluded that it indeed is possible to predict the query type.

It is unfortunate that the study examined only the most popular queries. As is often the case, simple, intuitive queries are usually the most frequent in a search system and it is not unreasonable to suspect that the chosen queries are much easier to classify than other, less frequent but more ambiguous, queries. It is not necessarily so that the ease of classification on these 50 popular queries is representative for all queries.

The authors discovered that the queries that were difficult to classify were those about software or people. After this discovery they decided to discard these queries and only work with those that were easy to classify. They note that query routing can be used to invoke special treatment for this type of difficult queries.

Automating the process

In automating the process, Lee, Liu and Cho looked at past user-click behavior and anchor-link distribution. Past user-click behavior studies what links have been clicked for a given query in the past. For the given query, the authors create a histogram of what links have been clicked for a given query. The characteristics of the distribution are assumed to tell us something of the classification of the query. If a majority of the clicks have the same destination link then this query is likely to be navigational in nature. If, on the other hand, the distribution is flat the query probably has an informational intent.

Mathematically, this can be expressed through several measures. The authors experimented with mean and median values along with measures of skewness and kurtosis. Simply put, skewness tells us how much the sorted distribution leans to the left. Kurtosis is a measure of how much the distribution tends to peak or how flat it is. In both cases a higher value is expected to mean a higher probability of navigational intent in the query than do lower values.

Another feature is how many clicks in average the query resulted in. A single click probably means navigational, where as we would expect informational queries to require several clicks.

Anchor-link distribution is based on link data from sites on the Web. For each query a distribution is compiled from links on the web that contain the query in the anchor description. Needless to say this requires massive amounts of data from highly informative and authoritative web sites. As with past user-click distributions the mean, median, skewness and kurtosis measures provide good candidate features.

Results show that when using past user-click behavior mean, median, skewness and kurtosis are equally good at predicting the query type with about 80% accuracy. Using average number of clicks as a feature yields the same accuracy. All of this is with bias toward informational queries. The results for anchor-link distribution resulted in 75% for each feature but with bias toward navigational queries. Combining features from both anchor-link and past user-click behavior got them 90% accuracy.

As far as data goes, the AOL logs are nice to play with because they contain authentic data that shows how real people use search on the web. The log contains over 20 million queries issued by 650 000 users over three months of time. The data is split over ten files with each row formatted according to

{UserID, Query, QueryTime, ClickedRank, DestinationDomainUrl}

Our primary domain of interest is in private networks such as corporate intranets, and we expect that there are many differences between the AOL logs and those that we intend to study. In particular we assume that search in private networks is more domain-oriented and focused than web search. Users searching for information at a medical university are probably interested in chemical compounds or what’s for lunch at the local cantina, but are probably not looking to use the intranet search application to buy a car. The web, on the other hand, boasts information on very diverse topics and we expect that users will probably be looking for all of the above and much more.

On the whole, we believe the AOL logs are useful for the core of our research needs. After all, the framework of parsing the logs, dividing them into sessions and clustering them should be common to any search engine implementation, regardless of domain.

Some general information about the the logs can be found in the Wikipedia article AOL search data scandal. Chronicle of AOL Search Query Log Release Incident has a timeline over the events of the AOL log scandal.

Are timeouts good enough?

So how do we go about finding sessions? In their paper Beyond the Session Timeout: Automatic Hierarchical Segmentation of Search Topics in Query Logs, Rosie Jones and Kristina Lisa Klinkner provide a short background to this problem before presenting their own methods and findings. They note that most recent attempts have been based on temporal features. The theory is that if enough time has passed between two subsequent queries then it is likely that the user’s task at hand isn’t the same for the two queries. There is some evidence to support this, but as Jones and Klinkner point out in the title of their paper temporal features by themselves simply aren’t enough.

As an example, take #4578, a fictive user. Like any other user, #4578 often uses her preferred search engine to find information on whatever she may be up to at the moment. #4578’s actions are stored in a query log and left for us to investigate. By simply inspecting the queries we can in most cases determine where #4578 shifts the topic of her searches. Sometimes though, and often due to the ambiguous nature of queries, we cannot do this. In those cases the timestamp can help us. If there was a short pause between the two queries then perhaps #4578 is simply reformulating herself. However, a large gap can lead us to suspect that she has either found what she was looking for or  given up, and so is now looking for something different.

So yes, timestamps are useful in finding boundaries between search sessions. Unfortunately there isn’t a single time gap that will allow us to confidently determine where sessions start and end. The reason is that this metric, just like the query string itself, is highly ambiguous. Does a long gap really indicate that the user has stopped searching for something and begun afresh? Yes, but only if we choose that gap to be hours or even days. And if we do, then we will miss all those occasions where #4578 finds what she is looking for and immediately switches to the next task.

The hierarchical model

Jones and Klinkner contribute to the problem of session detection in two ways. For starters, they propose a hierarchical model for query logs. Using their terminology, a Search Session is the set of all queries in the log issued by the same user, for example by #4578. Sessions contain Search Missions. These constitute the user’s intentions or needs, for example the need to have a suit cleaned. Finally, missions may contain Search Goals. Search goals are subtasks that need to be carried out in order to achieve an intention or need, for example finding a decent dry cleaner, and finding a service that can plot the route to the cleaner’s on a map.

Jones and Klinkner acknowledge that the term Search Session has traditionally been used to describe those queries submitted by a single user and sharing the same intent, but point out that not even among previous writers has there existed a common definition. For the sake of conformity we will continue to let Search Session mean the set of queries a user issues for a given intent, that is, what Jones and Klinkner define as a Search Goal.

If timeouts aren’t enough, what else is needed?

Their second contribution is a classifier that is able to tag search sessions with up to 95% accuracy. This is to be compared with the best result of those classifiers based solely on temporal data and able to tag sessions at only 70% accuracy. In their effort to develop a better classifier, Jones and Klinkner have looked at the following features.

• Temporal features, such as the time gap between subsequent queries are still useful in combination with other features
• Word and character edit features, such as the number of words the queries have in common
• Query log sequence features, like frequently collocated pairs of words in the query log as a whole
• Web search features like having clicked links in common, or the overall similarity of the queries’ result sets.

One of the most interesting results of their investigation is the great impact of Levenshtein distance as a similarity feature. Levenshtein distance, or the edit distance as it is also known, is the number of operations required to transform one string into another. Levenshtein distance could, by itself, manage to yield a 90% accuracy score for tagging sessions. Seeing as to how easy the Levenshtein distance algorithm is to implement, this would probably serve as a good start for our own classifier.

A query log is a sequence of records of users interacting with an information retrieval system such as a search engine. Ultimately, the success of such a system is measured by how well its users can use it to find what they’re looking for. In this context query logs can help us evaluate how well the system is performing. If we can somehow detect what users are looking for, and if we can measure how well they seem to be able to find this information in the system, then we can measure how successful the system is. The same methods can also be used to identify areas where the system is not performing well, and where efforts from administrators and information managers can help improve it. Together this makes the analyzing of query logs a key tool when administering an information retrieval system.

What are users looking for?

Two popular metrics when analyzing query logs are most frequent queries and most frequent 0-hit queries. A list of most frequent queries is assumed to tell us what people are looking for in the system. A list of most frequent 0-hit queries tells us what people are looking for and not finding. A good system administrator or information manager will want to know what frequent queries aren’t being answered so that he or she can take steps to add the information or make it more accessible.

However, merely counting occurrences of queries isn’t good enough. The problem is that there may be many queries describing a given need. How do we know which queries represent the same need? At the same time a query may be relevant to more than one need. How do we know which of several different needs a query represents? Thus when compiling a list of the most frequent queries, we are actually producing a list of the most entered query strings, when we would much rather learn the actual information needs of the user.

How do we go about doing this? We have already assumed that there is a connection between the information need of a user and the queries he or she issues to the search system. Based on previous research we believe that studying search sessions will yield better clues as to what the user was looking for than studying plain search queries. A search session contains all the queries that a single user issued to the system in order to fill a single information need. Furthermore, if the query log provides such information, we can extract additional clues from which documents the user clicked during the search session.

Two sessions containing similar queries and sharing clicked documents are probably similar also, with respect to the information needs of the two users. Using this similarity we can divide the set of search sessions in the query log into clusters, where sessions in the same cluster reflect the same need. Large clusters imply information needs that many users are trying to fill, which is exactly what we are looking for.

Our contribution

Methods like the ones above have been outlined in a few recent scientific articles. We aim to implement a query log topic detection system based on these, but we also hope to contribute extensions to topic detection that we believe to be useful when evaluating an information retrieval system.

First, we intend to investigate if it is possible to evaluate how well the information retrieval system satisfies the information needs of the users. For example, if many users are searching for a topic that can be roughly described as “pharmacy sleeping pills”, then we would like to know if they were able to find relevant information regarding this topic. We hope to gain understanding of this by analyzing each search session and detemining how likely it was that the user successfully found what he or she was looking for. Through this we can then draw conclusions for each topic based on the ratio of successful and unsuccessful sessions.

Second, we will present temporal statistics for the search topics. By doing this we can show the level of popularity of topics in a given time frame, and also track changes to topic popularity over time. This can be combined with our first contribution so that it is possible to track how well a topic in the system performs with regards to the share of sessions that managed to find relevant information. In the event that popular topics aren’t being covered by the system, administrators and information managers may take steps to improve this rating. Temporal statistics will be useful in proving that their effort did indeed have effect (or the opposite, if they were unsuccessful).