Cross-Language Information Retrieval - Resource based Approach

This post is intended to discuss how to handle traditional information retrieval in cross-language setting. This environment mainly allows for measuring the similarity between the information need (query) and information (indexed documents) across languages. The cross-language similarity estimation techniques can be generally categorised in two sets, depending on the similarity space: i) vector space, and ii) latent space. The former models aim to calculate the similarity in the vector space i.e. the terms are matched across languages using some external resources like bilingual dictionaries, transliteration systems, wordnets etc. The latter models measure the similarity in latent concept space. This post aims to review the former (vector space based) approach. Future post will include some discussion on latent space models to address CLIR.

This post assumes that the reader has basic knowledge about vector space based models like TF-IDF weighting schemes and cosine similarity. Also, the awareness of basic information retrieval (IR) modules like indexing and retrieval is assumed. If you are not sure about it, just go through a couple of pages at this link. The focus will be to explore more on the cross-language part of the IR problem.

As briefly discussed, in vector space the documents are compared in the original high dimensional space. In order to handle cross-language similarity either a query or the documents are translated to the language of comparison. This post considers the query-translation based approach where query is translated to the language of comparison. How the cross-language variant of IR system differs from its mono-lingual counterpart can be seen in the following figure.  

As can be seen in the Figure above, there is an additional module called "Query Translation Module" in CL variant. This module may be supported by cross-language resources like Bilingual dictionary, transliteration system or a complete machine-translation system like Google Translate API. In order to facilitate these systems, you may also require a morphological analyser and or stemmer. Usually you pass the query through a term-pipeline same as the documents when they were indexed. Below is more details on the term-pipeline.

1. Indexing

 First of all you want to index the source collection using a standard IR toolkit like Lucene, Terrier, Lemur, Xapian etc. There are different ways to index the collection with different indexing configurations. The most common ways are
 -  either to remove stop-words or not
 -  either to stem the terms or not, if yes then with what stemmer,
 - what should be considered a unit of terms like single-word-gram, n-word-gram or n-character-gram.

Hint for n-gram indexing
If your favourite IR library does not have n-gram (word/character) tokeniser than you can safely achieve this by a quick pre-processing script. For example, if you want to index 2-word-grams, joint each consecutive words to appear as 2-word grams by connecting them and putting a delimiter (in this case, #) between the terms just for the sake of readability.


Input = This is an example of n-grams.
Processed Input = This#is is#an an#example example#of of#ngrams

Don't forget to tokenise the query also the same way!


2. Query Translation

To achieve this, you first have to decide what you want to use and an availability of that for that particular language. If you have the access and the luxury, then you can get your query completely translated to the language of comparison by a complete MT system like Google Translate, Bing Translator, Apertium etc. Usually it is costly (also monetarily). Alternatively, you can use cross-lingual resources and systems like dictionaries (hand-made or statistically trained), transliteration system to normalise the language. Morphological analyser usually helps to look-up in the dictionary and to transliterate. The best way to load a dictionary in the code is to load as a HashMap or something similar for quick look-up. Sometimes this independent modules are connected in a sequential way, for example, if the term is not found in the dictionary then transliterate and so on. But it completely depends on the application and your expectation.

3. Retrieval (Ranking)

Finally you are ready to measure the similarity in the language of comparison. The vector space based models like TF-IDF, BM25 (probabilistic) and Language Models are the most popular ones. Usually all the toolkits have infrastructure to retrieve the document with its score based on the model you selected.

All these steps of IR system with Terrier are explained in the slides and the code is publicly available here.

I hope with this post you can start your basic CLIR system using one of your favourite IR toolkit.

Plagiarism Detection

Today the results of the 4th International Competition on Plagiarism Detection Challenge PAN : http://pan.webis.de are declared. We [ I and my collogues] at DA-IICT are regular participants of the competition. Many people ask me that why are you working on this and its a solved problem so what is the scope of research in there?

Well there are pretty good challenges which still require good amount of research. Plagiariser may not be writing the document as a verbatim copy of some other text. He/She may copy the text or matter with slight or high modifications starting from simple addition or deletion of words to more complex manual paraphrasing. With the good translation facilities available, a plagiariser may also translate the contents from source documents. Another challenge may be what to consider as source documents, in some cases you may have to take whole web as source documents. There are plenty of commercial plagiarism detectors available but they are good only when the plagiarised passages are exact copies from the source documents.

Plagiarism Detection has always been a very computationally complex and resource-hungry task. Because it involves sentence to sentence matches [Please don't consider matches as exact matches] and sometimes the documents being compared are some MBs big of raw text.

With the last years proceedings of PAN, it can be seen that many people have tried very innovative ideas to handle above challenges. But translation seems to be still a big challenge.

From translation I can remember of another big challenge. The machine translation technology for Indian languages is still in its infancy phase. You may have read the news of google has extended support for more 5 indian languages in its translation facility but still the quality of machine translation is very bad compared to European languages. You may try it there using some complex or long english statement and trying to translate it in hindi or so.

One such interesting challenge is going to happen in FIRE at IIT Bombay and the task name is CL!TR and its webpage is http://users.dsic.upv.es/grupos/nle/fire-workshop-clitr.html.

F.A.L.T.U : A Plagiarised movie from Hollywood movie 'Accepted' : Who will detect Plagiarism?

Those who have seen movie F.A.L.TU. may want to know the fact that it strongly resembles with the Hollywood Flick 'Accepted [2006]'.






"A high school slacker who's rejected by every school he applies to opts to create his own institution of higher learning, the South Harmon Institute of Technology, on a rundown piece of property near his hometown." [IMDB Page of 'Accepted']

Don't these lines make you think of the plot of Bollywood movie F.A.L.T.U? Yes its true, its the same story with a little makeover and in my opinion, they had to do those changes because they were inefficient even to reproduce the same story :D . Well I seriously pity on the debutant director who starts the career from such a plagiarised story and talks about creativity and passion in the movie.

This is not the only plagiarised movie, The list is long and includes some of the big names like 'Gajni' from 'Memento', 'Partner' from 'Hitch' . Sometimes songs are copied from some other songs, and our Anu Malik and Pritam are famous for that.

Plagiarism is prominent also in text, like we use someone else's work or text without explicitly acknowledging the person or source. “You know the feeling while reading an article on a topic related to your own research, you gradually become aware of a strong similarity between the ideas expressed and those you published some time ago. A quick glance at the references cited shows you that your article isn’t mentioned. ”[Garfield, 1980]. Here, as I have given the reference to Garfield so it is not Plagiarism! You may find it very frequent in student reports, even there are very funny cases like professor publishing book from the Ph.D thesis of his/her student.

As one of my research area is Natural Language Processing and somewhat plagiarism detection in text, this kind of incidents provoke me to write something about it. Plagiarism Detection has much moral and social impact than merely a profitable application. Because we help directly the society by making life of the plagiariser hard and promote the original work at the same time make the environment more ethical and authentic.

I sign off with a hope not to come across such cases again!

Should we make machine Intelligent?

Well some days ago I heard from someone that teaching process is changing so rapidly that there will not be any pen and paper in use. I remember my primary school days, when my teacher used to say all these stuff about the (future) technology. I used to feel so thrilled about it at that age thinking 'Yes.. it would be the great day!'

Now when I am an M.Tech student with specialization 'Machine Intelligence' , where we are taught to make machine intelligent so that one day a human can be replaced by a machine at many places. Well let me define what we mean by 'Machine Intelligence' for those who are new to this term, cause my mother asks me this question twice in a month. I will try to make it clear using an example. If you put one apple and one orange in front of your computer, turn on the camera for it and ask it to identify which is an orange? It will see the objects through a camera but probably you will get no answer, because your machine can not make decisions i.e. it is not intelligent enough to "infer". If you perform the same exercise with your 4yr old niece, she would spontaneously come up with the correct answer. Well, here one branch of machine intelligence 'Pattern Recognition' plays a role. Here we make these things easy for a machine. We can provide it some shapes and colour data to make decisions. Still there isn't any technology that can let your machine smell the object and perhaps taste the object!

In 'Machine Intelligence' we are directly in race with the Nature. Though we have started just 50-60 years back in this race, we have reached quite far but still a long way to go. At many places we have replaced the human effort. Like, where you had to keep a door man to open the gate only when, yours or your relatives', cars come, There are automatic Door opener with the system installed with desired Car number plates data that "if any car with provided numbers comes then only open the door" kind of instructions.

I see a danger in doing so. Population of many countries is increasing with a rapid rate. Then how will be the scene after 50 years when we would have enough machines to do all the work. What shall people do if there is a machine to do that. And always human labour is costlier than machine so I am sure what would industries prefer.

Other concern I see is, some conventional methods, i think, are still needed. Like, I find teaching on the board more effective than that through slides... Rough work/drafting with pencil and paper... Chapati made by your mom.. printed photos than digital on screen photos and blah blah..

So sometimes I get confused, should we stop this race or continue with increasing speed. Though as a student I will continue doing my assignments  still this topic is rolling in my mind quite often.