The Fact Ranking Challenge continues....

Yes, you might remember our experiment with fact ranking [1,2,3,4]. After almost 4 months, this is the current state of intermediate results (cf. below).

To make it short: YES, there has been some progress. NO, it is not sufficient so far.

Therefore, PLEASE HELP!
If you have already started to work with the application [3], PLEASE CONTINUE!
If you don't know the application [3], PLEASE START!
PLEASE PARTICIPATE!



We know that is is not an easy task and we also know that it takes time. Therefore, WE REALLY DO APPRECIATE YOUR HELP VERY MUCH!!

Please keep on playing and help us to gather more data!
Please tell all your family and friends to support us!
Please tell all your colleagues and fellow students to support us!

THANK YOU VERY MUCH!

[1] The Fact Ranking Quiz Application, http://s16a.org/fr/
[2] Help us with a Research Problem, July 30, 2014
[3] The Importance of Relevance - Intermediate Results, Aug 19, 2014
[4] More intermediate Results from our Fact Ranking Challenge, Sep. 2, 2014


...and here are the statistics:

Number of users who participated: 465

Sum of concepts done: 2410

485 unique concepts are covered (out of 541). 

Average concepts done per user: 5.183

295 times concepts were skipped (relevance in Step2 hasn't been changed for any of the facts).

CONCEPTS DONE:

0 concepts were done by 97 users. 

1 concepts were done by 93 users. 

2 concepts were done by 78 users. 

3 concepts were done by 49 users. 

4 concepts were done by 33 users. 

5 concepts were done by 26 users. 

6 concepts were done by 10 users. 

7 concepts were done by 7 users. 

8 concepts were done by 9 users. 

9 concepts were done by 8 users. 

10 concepts were done by 8 users. 

11 concepts were done by 6 users. 

12 concepts were done by 2 users. 

13 concepts were done by 1 users. 

14 concepts were done by 5 users. 

15 concepts were done by 3 users. 

16 concepts were done by 3 users. 

17 concepts were done by 1 users. 

18 concepts were done by 1 users. 

19 concepts were done by 2 users. 

20 concepts were done by 3 users. 

21 concepts were done by 2 users. 

23 concepts were done by 1 users. 

24 concepts were done by 2 users. 

25 concepts were done by 1 users. 

26 concepts were done by 1 users. 

31 concepts were done by 1 users. 

36 concepts were done by 1 users. 

40 concepts were done by 1 users. 

42 concepts were done by 2 users. 

56 concepts were done by 1 users. 

58 concepts were done by 1 users. 

60 concepts were done by 1 users. 

64 concepts were done by 1 users. 

68 concepts were done by 1 users. 

70 concepts were done by 1 users. 

88 concepts were done by 1 users. 

201 concepts were done by 1 users. 

EDUCATION:

highschool : 47 users.

phd : 71 users.

other : 26 users.

bachelors : 105 users.

masters : 213 users.

AGE:

33+ :  261 users.

19-25 :  72 users.

26-32 :  122 users.

0-18 :  7 users.

GENDER:

female :  111 users.

male :  351 users.

COUNTRY OF ORIGIN:

Angola :  1 users.

Belarus :  1 users.

Portugal :  4 users.

Philippines :  2 users.

Morocco :  3 users.

Greece :  5 users.

Ukraine :  3 users.

Indonesia :  4 users.

Afghanistan :  2 users.

Sri Lanka :  1 users.

Italy :  15 users.

Iraq :  2 users.

India :  44 users.

France :  14 users.

Denmark :  1 users.

Latvia :  1 users.

Pakistan :  4 users.

Syrian Arab Republic :  2 users.

Montenegro :  1 users.

Armenia :  1 users.

Mexico :  2 users.

Canada :  1 users.

Brazil :  10 users.

Venezuela :  1 users.

Croatia :  2 users.

Macedonia, The Former Yugoslav Republic of :  1 users.

Romania :  2 users.

Western Sahara :  1 users.

Algeria :  5 users.

Sweden :  2 users.

United States :  24 users.

Serbia :  10 users.

Nigeria :  3 users.

Estonia :  1 users.

Spain :  9 users.

Taiwan, Republic of China :  2 users.

Ireland :  1 users.

Russian Federation :  12 users.

Israel :  1 users.

Colombia :  3 users.

Switzerland :  2 users.

Azerbaijan :  2 users.

Kenya :  2 users.

Yemen :  1 users.

Malaysia :  2 users.

Viet Nam :  1 users.

Australia :  5 users.

Peru :  1 users.

Albania :  1 users.

South Africa :  2 users.

Tunisia :  1 users.

Netherlands :  9 users.

China :  3 users.

Somalia :  1 users.

Slovenia :  1 users.

Finland :  3 users.

Lithuania :  1 users.

Austria :  7 users.

Sudan :  1 users.

United Kingdom :  15 users.

Egypt :  2 users.

Bahamas :  1 users.

Hungary :  1 users.

Belgium :  1 users.

Poland :  6 users.

Iran, Islamic Republic of :  2 users.

Bulgaria :  3 users.

Norway :  1 users.

Germany :  177 users.

New Zealand :  3 users.

COUNTRY OF RESIDENCE:

United Arab Emirates :  1 users.

null :  0 users.

Belarus :  1 users.

Portugal :  2 users.

Philippines :  1 users.

Morocco :  2 users.

Greece :  3 users.

Ukraine :  1 users.

Indonesia :  3 users.

Luxembourg :  1 users.

Sri Lanka :  1 users.

Italy :  14 users.

Iraq :  1 users.

India :  32 users.

France :  17 users.

Jordan :  1 users.

Denmark :  1 users.

Latvia :  1 users.

Pakistan :  3 users.

Syrian Arab Republic :  1 users.

Oman :  1 users.

Turkey :  1 users.

Czech Republic :  1 users.

Armenia :  1 users.

Canada :  4 users.

Brazil :  9 users.

Croatia :  1 users.

Romania :  3 users.

Algeria :  6 users.

Sweden :  3 users.

United States :  25 users.

Serbia :  5 users.

Nigeria :  1 users.

Saudi Arabia :  1 users.

Estonia :  2 users.

Spain :  7 users.

Taiwan, Republic of China :  1 users.

Ireland :  3 users.

Russian Federation :  6 users.

Israel :  2 users.

Colombia :  2 users.

Switzerland :  8 users.

Azerbaijan :  2 users.

Kenya :  2 users.

Norfolk Island :  1 users.

Yemen :  1 users.

Malaysia :  2 users.

Australia :  7 users.

Peru :  1 users.

Albania :  1 users.

South Africa :  2 users.

Netherlands :  13 users.

China :  1 users.

Somalia :  1 users.

Slovenia :  1 users.

Gambia :  1 users.

Finland :  3 users.

Lithuania :  1 users.

Austria :  6 users.

United Kingdom :  17 users.

Egypt :  1 users.

Bahamas :  1 users.

Belgium :  3 users.

Poland :  7 users.

Singapore :  2 users.

Iran, Islamic Republic of :  1 users.

Bulgaria :  3 users.

Norway :  1 users.

Germany :  197 users.

New Zealand :  3 users.

Overall average confidence of users about the seen concepts: 2.616

NO. OF USERS PER CONCEPT:

1 users for 67 concepts

2 users for 123 concepts

3 users for 108 concepts

4 users for 86 concepts

5 users for 73 concepts

6 users for 37 concepts

7 users for 11 concepts

8 users for 7 concepts

9 users for 2 concepts

10 users for 2 concepts

18 users for 1 concepts

On AVERAGE there are 3.398 users per concept.

NO. OF ANSWERS PER CONCEPT (STEP 1):

On AVERAGE there are 5.002 answers per concept.

NONSENSE STATEMENTS:

TOTAL number of nonsense sentences = 1771

More intermediate Results from our Fact Ranking Challenge

Fact Ranking Challenge

Again, here is an update about our currently gathered data about our fact ranking experiment.

We started the original challenge about 5 weeks ago and now are able to present you some more intermediate results [1]. Nevertheless, the challenge is still running. Therefore, please distribute, participate, advertise, and help us to generate a fully fledged ground truth for fact ranking [2]. All data will be made publicly available for further research.

To determine the importance of a fact is of utmost importance, if you want to properly understand the content of information. Usually, you have a rich variety of possible interpretations of information. To determine the proper interpretation, you are going to use the context, i.e. further available information. So, the question develops from "what is important?" to "what is important with regard of this specific context?".

Current Intermediate Statistics (from Aug 28):

Number of users who participated: 388

Sum of concepts done: 1736

446 unique concepts are covered (out of 541). 

Average concepts done per user: 4.47

CONCEPTS DONE:

0 concepts were done by 79 users. 

1 concepts were done by 79 users. 

2 concepts were done by 68 users. 

3 concepts were done by 41 users. 

4 concepts were done by 25 users. 

5 concepts were done by 22 users. 

6 concepts were done by 8 users. 

7 concepts were done by 7 users. 

8 concepts were done by 9 users. 

9 concepts were done by 6 users. 

10 concepts were done by 8 users. 

11 concepts were done by 4 users. 

12 concepts were done by 2 users. 

13 concepts were done by 1 users. 

14 concepts were done by 4 users. 

15 concepts were done by 3 users. 

16 concepts were done by 2 users. 

19 concepts were done by 1 users. 

20 concepts were done by 3 users. 

21 concepts were done by 2 users. 

22 concepts were done by 1 users. 

23 concepts were done by 1 users. 

24 concepts were done by 2 users. 

25 concepts were done by 1 users. 

31 concepts were done by 1 users. 

38 concepts were done by 2 users. 

40 concepts were done by 1 users. 

42 concepts were done by 1 users. 

58 concepts were done by 1 users. 

59 concepts were done by 1 users. 

62 concepts were done by 1 users. 

64 concepts were done by 1 users. 

EDUCATION:

highschool : 41 users.

phd : 57 users.

other : 22 users.

bachelors : 93 users.

masters : 174 users.

AGE:

33+ :  216 users.

19-25 :  68 users.

26-32 :  98 users.

0-18 :  5 users.

GENDER:

female :  90 users.

male :  297 users.

COUNTRY OF ORIGIN:

Angola :  1 users.

Belarus :  1 users.

Portugal :  4 users.

Philippines :  2 users.

Morocco :  3 users.

Greece :  5 users.

Ukraine :  3 users.

Indonesia :  3 users.

Sri Lanka :  1 users.

Italy :  13 users.

Iraq :  1 users.

India :  40 users.

France :  11 users.

Latvia :  1 users.

Pakistan :  3 users.

Syrian Arab Republic :  1 users.

Montenegro :  1 users.

Armenia :  1 users.

Mexico :  2 users.

Brazil :  10 users.

Venezuela :  1 users.

Croatia :  2 users.

Macedonia, The Former Yugoslav Republic of :  1 users.

Romania :  2 users.

Western Sahara :  1 users.

Algeria :  4 users.

Sweden :  2 users.

United States :  21 users.

Serbia :  8 users.

Nigeria :  2 users.

Estonia :  1 users.

Spain :  8 users.

Taiwan, Republic of China :  2 users.

Ireland :  1 users.

Israel :  1 users.

Russian Federation :  9 users.

Colombia :  3 users.

Switzerland :  1 users.

Azerbaijan :  1 users.

Kenya :  2 users.

Yemen :  1 users.

Malaysia :  2 users.

Viet Nam :  1 users.

Australia :  4 users.

Peru :  1 users.

Albania :  1 users.

South Africa :  2 users.

Netherlands :  8 users.

China :  2 users.

Somalia :  1 users.

Slovenia :  1 users.

Finland :  3 users.

Lithuania :  1 users.

Austria :  6 users.

Sudan :  1 users.

United Kingdom :  15 users.

Egypt :  2 users.

Bahamas :  1 users.

Hungary :  1 users.

Poland :  4 users.

Iran, Islamic Republic of :  2 users.

Bulgaria :  3 users.

Norway :  1 users.

Germany :  140 users.

New Zealand :  3 users.

COUNTRY OF RESIDENCE:

United Arab Emirates :  1 users.

Belarus :  1 users.

Portugal :  2 users.

Philippines :  1 users.

Morocco :  2 users.

Greece :  3 users.

Ukraine :  1 users.

Indonesia :  2 users.

Luxembourg :  1 users.

Sri Lanka :  1 users.

Italy :  11 users.

India :  31 users.

France :  13 users.

Jordan :  1 users.

Denmark :  1 users.

Latvia :  1 users.

Pakistan :  3 users.

Oman :  1 users.

Turkey :  1 users.

Czech Republic :  1 users.

Armenia :  1 users.

Canada :  3 users.

Brazil :  9 users.

Croatia :  1 users.

Romania :  3 users.

Algeria :  4 users.

Sweden :  3 users.

United States :  23 users.

Serbia :  4 users.

Nigeria :  1 users.

Saudi Arabia :  1 users.

Estonia :  2 users.

Spain :  7 users.

Taiwan, Republic of China :  1 users.

Ireland :  3 users.

Israel :  2 users.

Russian Federation :  2 users.

Colombia :  2 users.

Switzerland :  7 users.

Azerbaijan :  2 users.

Kenya :  2 users.

Norfolk Island :  1 users.

Yemen :  1 users.

Malaysia :  2 users.

Australia :  6 users.

Peru :  1 users.

Albania :  1 users.

South Africa :  1 users.

Netherlands :  12 users.

Somalia :  1 users.

Slovenia :  1 users.

Gambia :  1 users.

Finland :  3 users.

Lithuania :  1 users.

Austria :  5 users.

United Kingdom :  17 users.

Egypt :  1 users.

Bahamas :  1 users.

Belgium :  2 users.

Poland :  5 users.

Singapore :  1 users.

Iran, Islamic Republic of :  1 users.

Bulgaria :  3 users.

Norway :  1 users.

Germany :  154 users.

New Zealand :  2 users.

Overall confidence of users about the seen concepts: 2.687

NO. OF USERS PER CONCEPT:

On AVERAGE there are 2.18 users per concept.

NO. OF ANSWERS PER CONCEPT (STEP 1):

On AVERAGE there are 4.71 answers per concept.

NONSENSE STATEMENTS:

TOTAL number of nonsense sentences = 1371

Hint: You might wonder about the impressive high scores on the top of the list? Well, actually points are given exponentially, i.e. the longer you play, the more points you will score per processed concept.

References:
[1] Help Us with a Research Problem, July 30, 2914
[2] Fact Ranking Web-Application, http://s16a.org/fr/

The Importance of Relevance - Intermediate Results

Current Highscore List of our fact ranking challenge

In my last post, we invited you to take part in our research challenge, which was about creating a ground truth for fact ranking algorithms. To determine the importance of a fact is of utmost importance, if you want to properly understand the content of information. Usually, you have a rich variety of possible interpretations of information. To determine the proper interpretation, you are going to use the context, i.e. further available information. So, the question develops from "what is important?" to "what is important with regard of this specific context?".

We started the original challenge about 3 weeks ago and now are able to present you first intermediate results [1]. Nevertheless, the challenge is still running. Therefore, please distribute, participate, advertise, and help us to generate a fully fledged ground truth for fact ranking [2]. All data will be made publicly available for further research.

Current Intermediate Statistics:
Number of users who participated: 110 (Thanks to all you you!!!)
Number of overall processed concepts: 509
Overall 200 unique concepts are covered (out of 541).

Average concepts processed per user: 4.63

Detailed number of processed concepts per user:
0 concepts were done by 15 users.
1 concepts were done by 27 users.
2 concepts were done by 21 users.
3 concepts were done by 12 users.
4 concepts were done by 7 users.
5 concepts were done by 8 users.
6 concepts were done by 2 users.
7 concepts were done by 1 users.
8 concepts were done by 4 users.
9 concepts were done by 1 users.
10 concepts were done by 2 users.
11 concepts were done by 2 users.
14 concepts were done by 1 users.
16 concepts were done by 1 users.
20 concepts were done by 1 users.
22 concepts were done by 1 users.
25 concepts were done by 1 users.
31 concepts were done by 1 users.
53 concepts were done by 2 users.

Participant statistics:

EDUCATION:
highschool : 7 users.
bachelors : 28 users.
masters : 47 users.
phd : 23 users.

other : 5 users. 

AGE:
33+ : 43 users.
26-32 : 34 users.
19-25 : 33 users.

GENDER:
female : 25 users.
male : 85 users.

COUNTRY OF ORIGIN:
United States : 9 users.
Serbia : 6 users.
Spain : 2 users.
Ukraine : 1 users.
Russian Federation : 3 users.
Colombia : 1 users.
Italy : 6 users. India : 2 users.
France : 3 users.
Malaysia : 1 users.
Australia : 1 users.
Albania : 1 users.
China : 1 users.
Pakistan : 1 users.
Finland : 1 users.
Austria : 2 users.
Montenegro : 1 users.
United Kingdom : 8 users.
Brazil : 3 users.
Poland : 2 users.
Iran, Islamic Republic of : 1 users.
Macedonia, The Former Yugoslav Republic of : 1 users.
Croatia : 1 users.
Germany : 49 users.
Algeria : 1 users.
New Zealand : 1 users.
Sweden : 1 users.

Overall confidence of users about the seen concepts: 2.585

NO. OF USERS PER CONCEPT:
On AVERAGE there are 1.295 users per concept.

NO. OF ANSWERS PER CONCEPT (STEP 1):
On AVERAGE there are 4.825 answers per concept.

We will keep you posted about the results.
Please distribute, participate, advertise, and help us to generate a fully fledged ground truth for fact ranking.

Hint: You might wonder about the impressive high scores on the top of the list? Well, actually points are given exponentially, i.e. the longer you play, the more points you will score per processed concept.

References:
[1] Help Us with a Research Problem, July 30, 2914
[2] Fact Ranking Web-Application, http://s16a.org/fr/

Help Us with a Research Problem

As you might know, we already have tried previously to let the public participate in our research. Last time, we have had developed several games (with a purpose). This time, unfortunately it is not a game, simply because the development of a good game is really expensive. But, let's get to the point. What is the task all about, where you can help us....?

You know, my research group is working on semantic technologies. Semantics in that sense means, we are trying to (automatically) understand what information (or data) is all about and what is the meaning of it. Sometimes, information is ambiguous. This makes it difficult to understand, because you have to solve ambiguities with the help of context.

On the other hand, sometimes you have various different information about a subject. How do you determine, which information or fact is more important or relevant than another? Just a quick example. Let's assume we have the following two facts:

(1) Albert Einstein is a physicist.
(2) Albert Einstein is a Vegetarian.

Which of the two facts is more important or relevant? Yes, this is difficult to answer, simply because the truth often lies in the eye of the beholder. For a vegetarian, maybe the second fact is more important. But, what about the most common opinion? What would the mainstream think? Probably, most people would say that fact (1) in general is more important.

So, what we are doing is that we develop heuristics that determine the importance of facts (relative to other facts). To get an idea about the quality of our heuristics, we have to do an evaluation, i.e. somebody has to decide whether the decision of the heuristics was wrong or right. Unfortunately, there does not exist a ground truth for this task called "fact ranking". Therefore, we are about to create a new ground truth that later will be publicly available and open for all researchers.

This ground truth is achieved with the little 'voting' application that you will find here [1]. You just have to register with the tool and then the task will be explained to you in detail. We took 500 popular concepts from Wikipedia and you have (1) to think about the most important facts about these concepts that come to your mind and then (2) rate the (new) facts presented to you according to their relevance. There is no right or wrong answer. Just vote as you think it seems right for you. Afterwards, we will aggregate all votes from all participants to determine the general (mainstream) relevance of the presented facts.

You might interrupt your rating of the presented facts at any time you like and continue later. To make it a bit more interesting, you can also score points and of course there is a highscore list. We would really appreciate your help in this task. Please do also spread the word. The more participants, the more valid our ground truth will be.

We know that this is a difficult and sometimes rather boring task. The more we would be really grateful for your assistance!

[1] Fact Ranking Web-Application, http://s16a.org/fr/

Who Knows Movies - The 2nd Round

Play it

We are happy to announce that our paper

Andreas Thalhammer, Magnus Knuth and Harald Sack: "Evaluating Entity Summarizations Using a Game-Based Ground Truth" 

has been accepted for the Evaluations and Experiments Track of ISWC 2012. We want to thank all of you very much, who played our WhoKnowsMovies? game, which allowed us to collect the necessary data for this publication. And since we also want provide updated statistics for the final version of our paper, you are very welcome to play a little bit more... :)

There has been a little upgrade for the game including more questions and we need more data to achieve a more reliable proof of our assumption that our proposed fact ranking for entity summarization really is better than a random choice.

Therefore, please help us and play the game. Test your knowledge about movies! Can you challenge the highscore? 

P.S. The already gathered (anonymized) data is available at http://www.yovisto.com/labs/iswc2012/.