The Importance of Relevance - Intermediate Results

Current Highscore List of our fact ranking challenge

In my last post, we invited you to take part in our research challenge, which was about creating a ground truth for fact ranking algorithms. To determine the importance of a fact is of utmost importance, if you want to properly understand the content of information. Usually, you have a rich variety of possible interpretations of information. To determine the proper interpretation, you are going to use the context, i.e. further available information. So, the question develops from "what is important?" to "what is important with regard of this specific context?".

We started the original challenge about 3 weeks ago and now are able to present you first intermediate results [1]. Nevertheless, the challenge is still running. Therefore, please distribute, participate, advertise, and help us to generate a fully fledged ground truth for fact ranking [2]. All data will be made publicly available for further research.

Current Intermediate Statistics:
Number of users who participated: 110 (Thanks to all you you!!!)
Number of overall processed concepts: 509
Overall 200 unique concepts are covered (out of 541).

Average concepts processed per user: 4.63

Detailed number of processed concepts per user:
0 concepts were done by 15 users.
1 concepts were done by 27 users.
2 concepts were done by 21 users.
3 concepts were done by 12 users.
4 concepts were done by 7 users.
5 concepts were done by 8 users.
6 concepts were done by 2 users.
7 concepts were done by 1 users.
8 concepts were done by 4 users.
9 concepts were done by 1 users.
10 concepts were done by 2 users.
11 concepts were done by 2 users.
14 concepts were done by 1 users.
16 concepts were done by 1 users.
20 concepts were done by 1 users.
22 concepts were done by 1 users.
25 concepts were done by 1 users.
31 concepts were done by 1 users.
53 concepts were done by 2 users.

Participant statistics:

EDUCATION:
highschool : 7 users.
bachelors : 28 users.
masters : 47 users.
phd : 23 users.

other : 5 users. 

AGE:
33+ : 43 users.
26-32 : 34 users.
19-25 : 33 users.

GENDER:
female : 25 users.
male : 85 users.

COUNTRY OF ORIGIN:
United States : 9 users.
Serbia : 6 users.
Spain : 2 users.
Ukraine : 1 users.
Russian Federation : 3 users.
Colombia : 1 users.
Italy : 6 users. India : 2 users.
France : 3 users.
Malaysia : 1 users.
Australia : 1 users.
Albania : 1 users.
China : 1 users.
Pakistan : 1 users.
Finland : 1 users.
Austria : 2 users.
Montenegro : 1 users.
United Kingdom : 8 users.
Brazil : 3 users.
Poland : 2 users.
Iran, Islamic Republic of : 1 users.
Macedonia, The Former Yugoslav Republic of : 1 users.
Croatia : 1 users.
Germany : 49 users.
Algeria : 1 users.
New Zealand : 1 users.
Sweden : 1 users.

Overall confidence of users about the seen concepts: 2.585

NO. OF USERS PER CONCEPT:
On AVERAGE there are 1.295 users per concept.

NO. OF ANSWERS PER CONCEPT (STEP 1):
On AVERAGE there are 4.825 answers per concept.

We will keep you posted about the results.
Please distribute, participate, advertise, and help us to generate a fully fledged ground truth for fact ranking.

Hint: You might wonder about the impressive high scores on the top of the list? Well, actually points are given exponentially, i.e. the longer you play, the more points you will score per processed concept.

References:
[1] Help Us with a Research Problem, July 30, 2914
[2] Fact Ranking Web-Application, http://s16a.org/fr/

Harald's Original Miscellany - The Truth about Football

The England National football team, 1893, photo: wikipedia

Well, it's the time of the Worldcup 2014. Why should I bother you with the peculiarities of authors and writers, when we can also have a look on Football! As you might remember, we had about 15,328 individuals in Wikipedia classified as authors [1]. What do you think, how many footballers are there compared to authors? Well there's a huge difference: 162,597 referenced footballer players, i.e. about 10 times as much as authors [2]. Maybe you think now there might be an overlap. How many football players are also listed as authors? I am sorry to disappoint you, but there is no overlap. No football player is also listed as being an author.

Fact No. 1: Footballers are no authors, and vice versa.

So you might wonder, what other categories these football players are in to get a better overview about what we are talking about. Interestingly, when looking at the most popular categories, you will soon find the large number of expatriates among those players. The Top5 expatriate nationalities among football players are: Brazil, Argentina, Russia, France, Serbia [3]. If you look at the bottom of the list, you will find the more or less exotic combinations, such as e.g. Hungarian expatriates in Uzbekistan, or Cameroonian expatriates in Venezuela, or even German expatriates in the Netherlands ;-)

Fact No. 2: Only a few Hungarian football players emigrate to Uzbekistan.

And in which countries they prefer to emigrate? The Top5 countries for football players to emigrate are: England, Germany, Spain, Italy, France [4]. At least for France the statistics seems to be balanced somehow, while England and Germany are the leading nations to attract foreign football players all around the world. Very interesting also the bottom of the list, where as the "least attractive" countries Gambia, Guam, Nepal, or Antigua and Barbuda are listed.

Fact No.3: French football players seem to be undecided whether to stay or leave the country.

But what about the categories that do not have a direct relationship to football in the first place? Let's filter out these categories and let's have a look on what football players are up to.

alternative professions of football players	#players
http://dbpedia.org/class/yago/Director110014939	279
http://dbpedia.org/class/yago/Migrant110314952	271
http://dbpedia.org/class/yago/National109625401	250
http://dbpedia.org/class/yago/Intellectual109621545	237
http://dbpedia.org/class/yago/Alumnus109786338	234
http://dbpedia.org/class/yago/Scholar110557854	234
http://dbpedia.org/class/yago/Striker110663996	134
http://dbpedia.org/class/yago/Worker109632518	73
http://dbpedia.org/class/yago/SkilledWorker110605985	64
http://dbpedia.org/class/yago/Unfortunate109630641	56
http://dbpedia.org/class/yago/Trainer110722575	55
http://dbpedia.org/class/yago/Coach109931640	54
http://dbpedia.org/class/yago/Communicator109610660	51
http://dbpedia.org/class/yago/Serviceman110582746	51
http://dbpedia.org/class/yago/Expert109617867	43
http://dbpedia.org/class/yago/Observer110369528	33
http://dbpedia.org/class/yago/Cricketer109977326	32
http://dbpedia.org/class/yago/Adult109605289	28
http://dbpedia.org/class/yago/Victim110752093	28
http://dbpedia.org/class/yago/Rival110533013	27
http://dbpedia.org/class/yago/Official110372076	26
http://dbpedia.org/class/yago/Adjudicator109769636	26
http://dbpedia.org/class/yago/MilitaryOfficer110317007	25
http://dbpedia.org/class/yago/EnlistedPerson110058777	25
http://dbpedia.org/class/yago/Soldier110622053	25
http://dbpedia.org/class/yago/Referee110514429	25
http://dbpedia.org/class/yago/BadPerson109831962	23
http://dbpedia.org/class/yago/Wrongdoer109633969	23
http://dbpedia.org/class/yago/Writer110794014	21
http://dbpedia.org/class/yago/Professional110480253	21
http://dbpedia.org/class/yago/EnglishCricketers	20
http://dbpedia.org/class/yago/OlympicBronzeMedalistsForGermany	18
http://dbpedia.org/class/yago/Creator109614315	18
http://dbpedia.org/class/yago/AsianGamesGoldMedalistsForIran	16
http://dbpedia.org/class/yago/OlympicGoldMedalistsForTheUnitedStates	16
http://dbpedia.org/class/yago/Presenter110466387	16
http://dbpedia.org/class/yago/Survivor110681194	16
http://dbpedia.org/class/yago/Entertainer109616922	16
http://dbpedia.org/class/yago/Performer110415638	16
http://dbpedia.org/class/yago/Captain109893191	15
http://dbpedia.org/class/yago/CommissionedMilitaryOfficer109943239	15
http://dbpedia.org/class/yago/CommissionedOfficer109942970	15
http://dbpedia.org/class/yago/Artist109812338	15
http://dbpedia.org/class/yago/OlympicSilverMedalistsForParaguay	14
http://dbpedia.org/class/yago/OlympicSilverMedalistsForPoland	14
http://dbpedia.org/class/yago/Journalist110224578	14
http://dbpedia.org/class/yago/Principal110474950	13
http://dbpedia.org/class/yago/Criminal109977660	13
http://dbpedia.org/class/yago/OlympicGoldMedalistsForArgentina	12
http://dbpedia.org/class/yago/OlympicBronzeMedalistsForItaly	12

Please find the entire list in the references [5].

Fact No. 4: There are more intellectuals among football players than bad persons.

Wow, 237 football players are also categorized as being intellectuals, while 23 football players are listed as "bad persons". But, in this statistics, we will also find 21 writers(!) among the football players, as well as 15 artists, 14 journalists, and 13 criminals.
Further down the list, you will also find

• 12 politicians, 
• 8 identical twins, 
• 8(!) head of state, 
• 7 musicians, 
• 4 comedians (I'll bet there are more...), 
• 4 scientists, 
• 3 singers, 
• 2 mammals,
• 2 Gentleman Cricketeers,
• 2 gambling addicts,
• 2 aviators,
• 2 painters,
• 1 UFO conspiracy theorists,
• 1 bank robber,
• 1 rapper,
• 1 plumber,

etc.

So what does this tell us about football players?

Fact No. 5: There are more politicians among football players than comedians.

Well, in general, and according to Wikipedia, football players most times stick to their original profession. While emigrating here and there sometimes, there are only a few among them who actually have a second career outside of their original profession. Please note that we did not follow categories like football manager, football trainer, football coach, etc. Anyway, we finally did find also some authors among them....

Fact No. 6: There are writers among the football players...although they are not listed as authors.

to be continued....

Please find the full tables with all the results listed here in the References:

[1] total number of authors (?author rdf:type dbpedia-owl:Writer .)
[2] total number of football players (?player rdf:type dbpedia-owl:SoccerPlayer .)
[3] expatriate football players by home country
[4] expatriate football players by emigration country
[5] occupations of football players other than football

Who knows Movies? Another Game with a Purpose for the Semantic Web

Play it

Of course you will remember our last Quiz Game 'Who Knows' (cf. the post below), where we have used DBpedia facts to generate questionaires for a game with a purpose. The very first application of this game was to put to evaluate some newly developed entity/property heuristics via crowdsourcing. Then, we realized that the gathered data also had some collateral benefits such as the detection of inconsistencies and flaws within the Linked Data resources that we used for generating the questions.

Now, we are looking into another task: entity summarization. Entity summarization means that we are trying to wrap up only the most important facts that determine a distinct entity. Just think of the Google knowledge graph that displays entity summaries from Freebase. Thus, Entity summarization of course is rather similar to relevance ranking of facts.

To generate a ground truth for evaluation of entity summarization heuristics, we adapted our quiz game WhoKnows? to become WhoKnows?Movies! We are cooperating with Andreas Thalhammer from University of Innsbruck with this task and we have restricted the domain of questions to popular movies, adapted the gereration of questionaires by utilizing the Freebase knowledge Base. Now we have to gather data....

Therefore, please help us and play the game. Test your knowledge about movies! Can you challenge the highscore? 


P.S. Of course all the data gathered will be anonymized and made publicly avaible.

New Challenge! Playing WhoKnows? to develop a new Teflon Pan

Some of you already may know our --serious-- fun game WhoKnows? (N. Ludwig, J. Waitelonis, M. Knuth, H. Sack: WhoKnows? - Evaluating Linked Data Heuristics with a Quiz that Cleans Up DBpedia) that has been presented inter alia at this year's ESWC 2011 (cf. picture from poster session).

WhoKnows? is a quiz game based on the DBpedia dataset; while answering the quickies the player produces data that can be used for the ranking of facts from the underlying knowledge base. Furthermore, the player has the possibility to mark strange questions often originating from inconsistencies that we want to identify this way.

Now, as a next step we want to apply the collected data for the development of an expert finder and user interest profile recommendation system. For this we would appreciate a larger data set that allows us to rate the expertise of several users in various domains. If you like to contribute this research, you can do this easily by playing WhoKnows? on Facebook. In order to make a sound statement about your expertise, we need at least about thirty questions answered and of course the more the better.

Of course all gathered data will be anonymized before analysis and evaluation.

Don't forget it's really fun and educational together!
Your help really is appreciated. Thank you for playing!

P.S. We would be pleased to inform you about the final results, if you are interested in. Just send us an e-mail.

There are more Things in Heaven and Earth... - DBPedia Link Graph Analysis Revisited

In the course of our ongoing work with Linked Open Data, we recently made some analysis on the graph structure of DBPedia data. For this we only took under consideration the original link graph (aka 'wikilinks'), where we did some cleanup first, such as, e.g., resolving redirects, etc.

As a side effect, we had to compute in-degree and out-degree of all DBPedia entities according to wikilinks, ... and we discovered some more or less surprising facts (thanks to Nadine):

The entity with the hightest out-degree (i.e. number of outgoing links) currently is:

http://dbpedia.org/resource/List_of_places_in_Afghanistan

with 7.147 outlinks (after cleanup, and remember it's wikilinks and not typed links of DBPedia)

The entity with the highest in-degree (i.e. number of incoming links) currently is:

http://dbpedia.org/resource/Living_people

with 440.151 inlinks (after cleanup)

While the 2nd one (living people) seemed pretty clear to me, the first (Afghanistan places...) was a bit of a surprise (as also are trilobytes...). For all the explorers among us, I have included the Top Ten list of incoming and outgoing wikilinks, each with indegree and associated outdegree...

Top Ten Incoming	in	out
http://dbpedia.org/resource/Living_people	440151	0
http://dbpedia.org/resource/United_States	385407	963
http://dbpedia.org/resource/France	124206	759
http://dbpedia.org/resource/England	123223	1320
http://dbpedia.org/resource/United_Kingdom	121203	1152
http://dbpedia.org/resource/List_of_sovereign_states	114086	465
http://dbpedia.org/resource/Canada	105849	523
http://dbpedia.org/resource/Germany	103382	889
http://dbpedia.org/resource/Animal	98680	236
http://dbpedia.org/resource/World_War_II	93555	771
http://dbpedia.org/resource/Association_football	90673	196

Top Ten Outgoing	in	out
http://dbpedia.org/resource/list_of_places_in_afghanistan	9	7147
http://dbpedia.org/resource/Flora_of_New_South_Wales	917	6819
http://dbpedia.org/resource/List_of_municipalities_of_Brazil	1	5503
http://dbpedia.org/resource/Index_of_India-related_articles	4	5369
http://dbpedia.org/resource/Area_codes_in_Germany	6	5360
http://dbpedia.org/resource/IUCN_Red_List_vulnerable_species_%28Plantae%29	0	5172
http://dbpedia.org/resource/List_of_trilobites	6	5102
http://dbpedia.org/resource/List_of_Social_Democratic_Party_of_Germany_members	24	5078
http://dbpedia.org/resource/List_of_French_words_of_Germanic_origin	9	5010
http://dbpedia.org/resource/Index_of_Thailand-related_articles	4	4831

But there are more interesting things to discover ... stay tuned!