Implications of inter-rater agreement on a student information retrieval evaluation
| Titel: | Implications of inter-rater agreement on a student information retrieval evaluation |
|---|---|
| Verfasser: | ; ; |
| Beteiligt: | ; ; ; |
| Körperschaft: | |
| Veröffentlicht: | Kassel, 2010 |
| Umfang: | Online-Ressource, 1-7 S. |
| Format: | E-Book |
| Sprache: | Englisch |
| Schriftenreihe/ mehrbändiges Werk: |
LWA 2010. Lernen, Wissen & Adaptivität ; workshop proceedings ; Bd. 2010/5 Kasseler Informatikschriften (KIS) ; Bd. 2010/5 |
X
alg: 45288363 001A $06000:06-09-19 001B $01999:13-10-24 $t03:58:45.000 001D $06000:06-09-19 001U $0utf8 001X $00 002@ $0Oax 002C $aText $btxt $2rdacontent 002D $aComputermedien $bc $2rdamedia 002E $aOnline-Ressource $bcr $2rdacarrier 003@ $0452883636 004E/07 $0oai:gesis.izsoz.de:document/46667 004U $0urn:nbn:de:0168-ssoar-46667-7 006G $01192136276 006U $019,O09 $f20190803 007A $01192136276 $aDNB 009Q $uhttps://nbn-resolving.org/urn:nbn:de:0168-ssoar-46667-7 $xR 009Q $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $xH 010@ $aeng 011@ $a2010 $n2010 021A $aImplications of inter-rater agreement on a student information retrieval evaluation 028A $aSchaer $BVerfasser $dPhilipp $4aut 028C $aMayr $BVerfasser $dPhilipp $4aut 028C $aMutschke $BVerfasser $dPeter $4aut 028C $aAtzmüller $BHerausgeber $dMartin $4edt 028C $aBenz $BHerausgeber $dDominik $4edt 028C $aHotho $BHerausgeber $dAndreas $4edt 028C $aStumme $BHerausgeber $dGerd $4edt 029F $aUniversität Kassel, Knowledge and Data Engineering Group $BHerausgeber $4edt 033A $pKassel 034D $aOnline-Ressource, 1-7 S. 036E $aLWA 2010. Lernen, Wissen & Adaptivität ; workshop proceedings $lBd. 2010/5 036E/01 $aKasseler Informatikschriften (KIS) $lBd. 2010/5 037A $aPostprint 037A $abegutachtet 037A $aIn: Atzmüller, Martin (Hg.), Benz, Dominik (Hg.), Hotho, Andreas (Hg.), Stumme, Gerd (Hg.): LWA 2010 : Lernen, Wissen & Adaptivität ; workshop proceedings. 2010. S. 1-7 045E $c020 047I $aAbstract: This paper is about an information retrieval evaluation on three different retrieval-supporting services. All three services were designed to compensate typical problems that arise in metadata- driven Digital Libraries, which are not adequately handled by a simple tf-idf based retrieval. The services are: (1) a co-word analysis based query expansion mechanism and re-ranking via (2) Bradfordizing and (3) author centrality. The services are evaluated with relevance assessments conducted by 73 information science students. Since the students are neither information professionals nor domain experts the question of inter-rater agreement is taken into consideration. Two important implications emerge: (1) the inter- rater agreement rates were mainly fair to moderate and (2) after a data-cleaning step which erased the assessments with poor agreement rates the evaluation data shows that the three retrieval services returned disjoint but still relevant result sets. (author's abstract) lok: 45288363 3 exp: 45288363 3 1 #EPN 201B/01 $025-02-21 $t22:56:41.427 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736478 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 5 exp: 45288363 5 1 #EPN 201B/01 $025-02-21 $t22:56:41.430 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736486 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 8 exp: 45288363 8 1 #EPN 201B/01 $025-02-21 $t22:56:41.432 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736494 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 10 exp: 45288363 10 1 #EPN 201B/01 $025-02-21 $t22:56:41.435 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736508 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 11 exp: 45288363 11 1 #EPN 201B/01 $025-02-21 $t22:56:41.437 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736516 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 13 exp: 45288363 13 1 #EPN 201B/01 $025-02-21 $t22:56:41.440 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736524 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 20 exp: 45288363 20 1 #EPN 201B/01 $025-02-21 $t22:56:41.443 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736532 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 21 exp: 45288363 21 1 #EPN 201B/01 $025-02-21 $t22:56:41.446 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736540 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 23 exp: 45288363 23 1 #EPN 201B/01 $025-02-21 $t22:56:41.448 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736559 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 24 exp: 45288363 24 1 #EPN 201B/01 $025-02-21 $t22:56:41.451 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736567 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 25 exp: 45288363 25 1 #EPN 201B/01 $025-02-21 $t22:56:41.454 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736575 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 36 exp: 45288363 36 1 #EPN 201B/01 $013-10-24 $t03:58:45.730 201C/01 $013-10-24 201U/01 $0utf8 203@/01 $01277843813 208@/01 $a13-10-24 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 49 exp: 45288363 49 1 #EPN 201B/01 $025-02-21 $t22:56:41.456 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736583 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 51 exp: 45288363 51 1 #EPN 201B/01 $025-02-21 $t22:56:41.461 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736605 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 59 exp: 45288363 59 1 #EPN 201B/01 $025-02-21 $t22:56:41.467 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736621 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 74 exp: 45288363 74 1 #EPN 201B/01 $013-10-24 $t03:58:45.738 201C/01 $013-10-24 201U/01 $0utf8 203@/01 $01277843821 208@/01 $a13-10-24 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 76 exp: 45288363 76 1 #EPN 201B/01 $025-02-21 $t22:56:41.469 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $0114373663X 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 107 exp: 45288363 107 1 #EPN 201B/01 $025-02-21 $t22:56:41.472 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736648 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 108 exp: 45288363 108 1 #EPN 201B/01 $025-02-21 $t22:56:41.474 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736656 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 204 exp: 45288363 204 1 #EPN 201B/01 $025-02-21 $t22:56:41.480 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736664 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH lok: 45288363 205 exp: 45288363 205 1 #EPN 201B/01 $025-02-21 $t22:56:41.482 201C/01 $025-02-21 201U/01 $0utf8 203@/01 $01143736672 208@/01 $a25-02-21 $bl 209S/01 $S0 $uhttp://www.ssoar.info/ssoar/handle/document/46667 $XH
| LEADER | 00000cam a22000002c 4500 | ||
|---|---|---|---|
| 001 | 452883636 | ||
| 003 | DE-603 | ||
| 005 | 20241013035845.0 | ||
| 007 | cr|||||||||||| | ||
| 008 | 190906s2010 xx |||| o| u00||u|eng | ||
| 015 | |a 19,O09 |2 dnb | ||
| 016 | 7 | |2 DE-101 |a 1192136276 | |
| 024 | 7 | |a urn:nbn:de:0168-ssoar-46667-7 |2 urn | |
| 028 | 5 | 2 | |a oai:gesis.izsoz.de:document/46667 |
| 035 | |a (DE-599)DNB1192136276 | ||
| 040 | |a DE-603 |b ger |c DE-603 |d DE-603 | ||
| 041 | |a eng | ||
| 084 | |a 020 |q DE-101 |2 sdnb | ||
| 100 | 1 | |a Schaer, Philipp |e Verfasser |4 aut | |
| 245 | 0 | 0 | |a Implications of inter-rater agreement on a student information retrieval evaluation |
| 264 | 1 | |a Kassel |c 2010 | |
| 300 | |a Online-Ressource, 1-7 S. | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 490 | 0 | |a LWA 2010. Lernen, Wissen & Adaptivität ; workshop proceedings |v Bd. 2010/5 | |
| 490 | 0 | |a Kasseler Informatikschriften (KIS) |v Bd. 2010/5 | |
| 500 | |a Postprint | ||
| 500 | |a begutachtet | ||
| 500 | |a In: Atzmüller, Martin (Hg.), Benz, Dominik (Hg.), Hotho, Andreas (Hg.), Stumme, Gerd (Hg.): LWA 2010 : Lernen, Wissen & Adaptivität ; workshop proceedings. 2010. S. 1-7 | ||
| 520 | |a Abstract: This paper is about an information retrieval evaluation on three different retrieval-supporting services. All three services were designed to compensate typical problems that arise in metadata- driven Digital Libraries, which are not adequately handled by a simple tf-idf based retrieval. The services are: (1) a co-word analysis based query expansion mechanism and re-ranking via (2) Bradfordizing and (3) author centrality. The services are evaluated with relevance assessments conducted by 73 information science students. Since the students are neither information professionals nor domain experts the question of inter-rater agreement is taken into consideration. Two important implications emerge: (1) the inter- rater agreement rates were mainly fair to moderate and (2) after a data-cleaning step which erased the assessments with poor agreement rates the evaluation data shows that the three retrieval services returned disjoint but still relevant result sets. (author's abstract) | ||
| 700 | 1 | |a Mayr, Philipp |e Verfasser |4 aut | |
| 700 | 1 | |a Mutschke, Peter |e Verfasser |4 aut | |
| 700 | 1 | |a Atzmüller, Martin |e Herausgeber |4 edt | |
| 700 | 1 | |a Benz, Dominik |e Herausgeber |4 edt | |
| 700 | 1 | |a Hotho, Andreas |e Herausgeber |4 edt | |
| 700 | 1 | |a Stumme, Gerd |e Herausgeber |4 edt | |
| 710 | 2 | |a Universität Kassel, Knowledge and Data Engineering Group |e Herausgeber |4 edt | |
| 856 | |u https://nbn-resolving.org/urn:nbn:de:0168-ssoar-46667-7 |x Resolving-System | ||
| 856 | |u http://www.ssoar.info/ssoar/handle/document/46667 |x Verlag |z kostenfrei | ||
| 924 | 1 | |9 603 |a (DE-603)1143736478 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736486 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736494 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736508 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736516 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736524 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736532 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736540 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736559 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736567 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736575 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1277843813 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736583 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736605 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736621 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1277843821 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)114373663X |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736648 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736656 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736664 |b DE-603 |c HES |d d | |
| 924 | 1 | |9 603 |a (DE-603)1143736672 |b DE-603 |c HES |d d | |
