Reducing the number of non-naïve participants in Mechanical Turk samples
Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness....
Ausführliche Beschreibung
Autor*in: |
Ethan A. Meyers [verfasserIn] Alexander C. Walker [verfasserIn] Jonathan A. Fugelsang [verfasserIn] Derek J. Koehler [verfasserIn] |
---|
Format: |
E-Artikel |
---|---|
Sprache: |
Englisch |
Erschienen: |
2020 |
---|
Schlagwörter: |
---|
Übergeordnetes Werk: |
In: Methods in Psychology - Elsevier, 2021, 3(2020), Seite 100032- |
---|---|
Übergeordnetes Werk: |
volume:3 ; year:2020 ; pages:100032- |
Links: |
---|
DOI / URN: |
10.1016/j.metip.2020.100032 |
---|
Katalog-ID: |
DOAJ013701479 |
---|
LEADER | 01000caa a22002652 4500 | ||
---|---|---|---|
001 | DOAJ013701479 | ||
003 | DE-627 | ||
005 | 20230310060413.0 | ||
007 | cr uuu---uuuuu | ||
008 | 230226s2020 xx |||||o 00| ||eng c | ||
024 | 7 | |a 10.1016/j.metip.2020.100032 |2 doi | |
035 | |a (DE-627)DOAJ013701479 | ||
035 | |a (DE-599)DOAJ45a2316901f847e8bf25776402229943 | ||
040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
041 | |a eng | ||
050 | 0 | |a BF1-990 | |
100 | 0 | |a Ethan A. Meyers |e verfasserin |4 aut | |
245 | 1 | 0 | |a Reducing the number of non-naïve participants in Mechanical Turk samples |
264 | 1 | |c 2020 | |
336 | |a Text |b txt |2 rdacontent | ||
337 | |a Computermedien |b c |2 rdamedia | ||
338 | |a Online-Ressource |b cr |2 rdacarrier | ||
520 | |a Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. | ||
650 | 4 | |a Mechanical Turk | |
650 | 4 | |a Nonnaivete | |
650 | 4 | |a HIT restriction | |
650 | 4 | |a Data quality | |
653 | 0 | |a Psychology | |
700 | 0 | |a Alexander C. Walker |e verfasserin |4 aut | |
700 | 0 | |a Jonathan A. Fugelsang |e verfasserin |4 aut | |
700 | 0 | |a Derek J. Koehler |e verfasserin |4 aut | |
773 | 0 | 8 | |i In |t Methods in Psychology |d Elsevier, 2021 |g 3(2020), Seite 100032- |w (DE-627)1689632550 |x 25902601 |7 nnns |
773 | 1 | 8 | |g volume:3 |g year:2020 |g pages:100032- |
856 | 4 | 0 | |u https://doi.org/10.1016/j.metip.2020.100032 |z kostenfrei |
856 | 4 | 0 | |u https://doaj.org/article/45a2316901f847e8bf25776402229943 |z kostenfrei |
856 | 4 | 0 | |u http://www.sciencedirect.com/science/article/pii/S2590260120300199 |z kostenfrei |
856 | 4 | 2 | |u https://doaj.org/toc/2590-2601 |y Journal toc |z kostenfrei |
912 | |a GBV_USEFLAG_A | ||
912 | |a SYSFLAG_A | ||
912 | |a GBV_DOAJ | ||
912 | |a GBV_ILN_11 | ||
912 | |a GBV_ILN_20 | ||
912 | |a GBV_ILN_22 | ||
912 | |a GBV_ILN_23 | ||
912 | |a GBV_ILN_24 | ||
912 | |a GBV_ILN_31 | ||
912 | |a GBV_ILN_39 | ||
912 | |a GBV_ILN_40 | ||
912 | |a GBV_ILN_60 | ||
912 | |a GBV_ILN_62 | ||
912 | |a GBV_ILN_63 | ||
912 | |a GBV_ILN_65 | ||
912 | |a GBV_ILN_69 | ||
912 | |a GBV_ILN_70 | ||
912 | |a GBV_ILN_73 | ||
912 | |a GBV_ILN_74 | ||
912 | |a GBV_ILN_95 | ||
912 | |a GBV_ILN_105 | ||
912 | |a GBV_ILN_110 | ||
912 | |a GBV_ILN_151 | ||
912 | |a GBV_ILN_161 | ||
912 | |a GBV_ILN_206 | ||
912 | |a GBV_ILN_213 | ||
912 | |a GBV_ILN_224 | ||
912 | |a GBV_ILN_230 | ||
912 | |a GBV_ILN_285 | ||
912 | |a GBV_ILN_293 | ||
912 | |a GBV_ILN_602 | ||
912 | |a GBV_ILN_2001 | ||
912 | |a GBV_ILN_2003 | ||
912 | |a GBV_ILN_2005 | ||
912 | |a GBV_ILN_2006 | ||
912 | |a GBV_ILN_2007 | ||
912 | |a GBV_ILN_2008 | ||
912 | |a GBV_ILN_2009 | ||
912 | |a GBV_ILN_2010 | ||
912 | |a GBV_ILN_2011 | ||
912 | |a GBV_ILN_2014 | ||
912 | |a GBV_ILN_2015 | ||
912 | |a GBV_ILN_2020 | ||
912 | |a GBV_ILN_2021 | ||
912 | |a GBV_ILN_2025 | ||
912 | |a GBV_ILN_2026 | ||
912 | |a GBV_ILN_2027 | ||
912 | |a GBV_ILN_2034 | ||
912 | |a GBV_ILN_2038 | ||
912 | |a GBV_ILN_2044 | ||
912 | |a GBV_ILN_2048 | ||
912 | |a GBV_ILN_2049 | ||
912 | |a GBV_ILN_2050 | ||
912 | |a GBV_ILN_2055 | ||
912 | |a GBV_ILN_2056 | ||
912 | |a GBV_ILN_2059 | ||
912 | |a GBV_ILN_2061 | ||
912 | |a GBV_ILN_2064 | ||
912 | |a GBV_ILN_2086 | ||
912 | |a GBV_ILN_2088 | ||
912 | |a GBV_ILN_2106 | ||
912 | |a GBV_ILN_2110 | ||
912 | |a GBV_ILN_2112 | ||
912 | |a GBV_ILN_2122 | ||
912 | |a GBV_ILN_2129 | ||
912 | |a GBV_ILN_2143 | ||
912 | |a GBV_ILN_2152 | ||
912 | |a GBV_ILN_2153 | ||
912 | |a GBV_ILN_2190 | ||
912 | |a GBV_ILN_2232 | ||
912 | |a GBV_ILN_2336 | ||
912 | |a GBV_ILN_2470 | ||
912 | |a GBV_ILN_2507 | ||
912 | |a GBV_ILN_4012 | ||
912 | |a GBV_ILN_4035 | ||
912 | |a GBV_ILN_4037 | ||
912 | |a GBV_ILN_4112 | ||
912 | |a GBV_ILN_4125 | ||
912 | |a GBV_ILN_4126 | ||
912 | |a GBV_ILN_4242 | ||
912 | |a GBV_ILN_4249 | ||
912 | |a GBV_ILN_4251 | ||
912 | |a GBV_ILN_4305 | ||
912 | |a GBV_ILN_4306 | ||
912 | |a GBV_ILN_4307 | ||
912 | |a GBV_ILN_4313 | ||
912 | |a GBV_ILN_4322 | ||
912 | |a GBV_ILN_4323 | ||
912 | |a GBV_ILN_4324 | ||
912 | |a GBV_ILN_4325 | ||
912 | |a GBV_ILN_4326 | ||
912 | |a GBV_ILN_4333 | ||
912 | |a GBV_ILN_4334 | ||
912 | |a GBV_ILN_4335 | ||
912 | |a GBV_ILN_4338 | ||
912 | |a GBV_ILN_4367 | ||
912 | |a GBV_ILN_4393 | ||
912 | |a GBV_ILN_4700 | ||
951 | |a AR | ||
952 | |d 3 |j 2020 |h 100032- |
author_variant |
e a m eam a c w acw j a f jaf d j k djk |
---|---|
matchkey_str |
article:25902601:2020----::euighnmeonnaeatcpnsneh |
hierarchy_sort_str |
2020 |
callnumber-subject-code |
BF |
publishDate |
2020 |
allfields |
10.1016/j.metip.2020.100032 doi (DE-627)DOAJ013701479 (DE-599)DOAJ45a2316901f847e8bf25776402229943 DE-627 ger DE-627 rakwb eng BF1-990 Ethan A. Meyers verfasserin aut Reducing the number of non-naïve participants in Mechanical Turk samples 2020 Text txt rdacontent Computermedien c rdamedia Online-Ressource cr rdacarrier Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. Mechanical Turk Nonnaivete HIT restriction Data quality Psychology Alexander C. Walker verfasserin aut Jonathan A. Fugelsang verfasserin aut Derek J. Koehler verfasserin aut In Methods in Psychology Elsevier, 2021 3(2020), Seite 100032- (DE-627)1689632550 25902601 nnns volume:3 year:2020 pages:100032- https://doi.org/10.1016/j.metip.2020.100032 kostenfrei https://doaj.org/article/45a2316901f847e8bf25776402229943 kostenfrei http://www.sciencedirect.com/science/article/pii/S2590260120300199 kostenfrei https://doaj.org/toc/2590-2601 Journal toc kostenfrei GBV_USEFLAG_A SYSFLAG_A GBV_DOAJ GBV_ILN_11 GBV_ILN_20 GBV_ILN_22 GBV_ILN_23 GBV_ILN_24 GBV_ILN_31 GBV_ILN_39 GBV_ILN_40 GBV_ILN_60 GBV_ILN_62 GBV_ILN_63 GBV_ILN_65 GBV_ILN_69 GBV_ILN_70 GBV_ILN_73 GBV_ILN_74 GBV_ILN_95 GBV_ILN_105 GBV_ILN_110 GBV_ILN_151 GBV_ILN_161 GBV_ILN_206 GBV_ILN_213 GBV_ILN_224 GBV_ILN_230 GBV_ILN_285 GBV_ILN_293 GBV_ILN_602 GBV_ILN_2001 GBV_ILN_2003 GBV_ILN_2005 GBV_ILN_2006 GBV_ILN_2007 GBV_ILN_2008 GBV_ILN_2009 GBV_ILN_2010 GBV_ILN_2011 GBV_ILN_2014 GBV_ILN_2015 GBV_ILN_2020 GBV_ILN_2021 GBV_ILN_2025 GBV_ILN_2026 GBV_ILN_2027 GBV_ILN_2034 GBV_ILN_2038 GBV_ILN_2044 GBV_ILN_2048 GBV_ILN_2049 GBV_ILN_2050 GBV_ILN_2055 GBV_ILN_2056 GBV_ILN_2059 GBV_ILN_2061 GBV_ILN_2064 GBV_ILN_2086 GBV_ILN_2088 GBV_ILN_2106 GBV_ILN_2110 GBV_ILN_2112 GBV_ILN_2122 GBV_ILN_2129 GBV_ILN_2143 GBV_ILN_2152 GBV_ILN_2153 GBV_ILN_2190 GBV_ILN_2232 GBV_ILN_2336 GBV_ILN_2470 GBV_ILN_2507 GBV_ILN_4012 GBV_ILN_4035 GBV_ILN_4037 GBV_ILN_4112 GBV_ILN_4125 GBV_ILN_4126 GBV_ILN_4242 GBV_ILN_4249 GBV_ILN_4251 GBV_ILN_4305 GBV_ILN_4306 GBV_ILN_4307 GBV_ILN_4313 GBV_ILN_4322 GBV_ILN_4323 GBV_ILN_4324 GBV_ILN_4325 GBV_ILN_4326 GBV_ILN_4333 GBV_ILN_4334 GBV_ILN_4335 GBV_ILN_4338 GBV_ILN_4367 GBV_ILN_4393 GBV_ILN_4700 AR 3 2020 100032- |
spelling |
10.1016/j.metip.2020.100032 doi (DE-627)DOAJ013701479 (DE-599)DOAJ45a2316901f847e8bf25776402229943 DE-627 ger DE-627 rakwb eng BF1-990 Ethan A. Meyers verfasserin aut Reducing the number of non-naïve participants in Mechanical Turk samples 2020 Text txt rdacontent Computermedien c rdamedia Online-Ressource cr rdacarrier Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. Mechanical Turk Nonnaivete HIT restriction Data quality Psychology Alexander C. Walker verfasserin aut Jonathan A. Fugelsang verfasserin aut Derek J. Koehler verfasserin aut In Methods in Psychology Elsevier, 2021 3(2020), Seite 100032- (DE-627)1689632550 25902601 nnns volume:3 year:2020 pages:100032- https://doi.org/10.1016/j.metip.2020.100032 kostenfrei https://doaj.org/article/45a2316901f847e8bf25776402229943 kostenfrei http://www.sciencedirect.com/science/article/pii/S2590260120300199 kostenfrei https://doaj.org/toc/2590-2601 Journal toc kostenfrei GBV_USEFLAG_A SYSFLAG_A GBV_DOAJ GBV_ILN_11 GBV_ILN_20 GBV_ILN_22 GBV_ILN_23 GBV_ILN_24 GBV_ILN_31 GBV_ILN_39 GBV_ILN_40 GBV_ILN_60 GBV_ILN_62 GBV_ILN_63 GBV_ILN_65 GBV_ILN_69 GBV_ILN_70 GBV_ILN_73 GBV_ILN_74 GBV_ILN_95 GBV_ILN_105 GBV_ILN_110 GBV_ILN_151 GBV_ILN_161 GBV_ILN_206 GBV_ILN_213 GBV_ILN_224 GBV_ILN_230 GBV_ILN_285 GBV_ILN_293 GBV_ILN_602 GBV_ILN_2001 GBV_ILN_2003 GBV_ILN_2005 GBV_ILN_2006 GBV_ILN_2007 GBV_ILN_2008 GBV_ILN_2009 GBV_ILN_2010 GBV_ILN_2011 GBV_ILN_2014 GBV_ILN_2015 GBV_ILN_2020 GBV_ILN_2021 GBV_ILN_2025 GBV_ILN_2026 GBV_ILN_2027 GBV_ILN_2034 GBV_ILN_2038 GBV_ILN_2044 GBV_ILN_2048 GBV_ILN_2049 GBV_ILN_2050 GBV_ILN_2055 GBV_ILN_2056 GBV_ILN_2059 GBV_ILN_2061 GBV_ILN_2064 GBV_ILN_2086 GBV_ILN_2088 GBV_ILN_2106 GBV_ILN_2110 GBV_ILN_2112 GBV_ILN_2122 GBV_ILN_2129 GBV_ILN_2143 GBV_ILN_2152 GBV_ILN_2153 GBV_ILN_2190 GBV_ILN_2232 GBV_ILN_2336 GBV_ILN_2470 GBV_ILN_2507 GBV_ILN_4012 GBV_ILN_4035 GBV_ILN_4037 GBV_ILN_4112 GBV_ILN_4125 GBV_ILN_4126 GBV_ILN_4242 GBV_ILN_4249 GBV_ILN_4251 GBV_ILN_4305 GBV_ILN_4306 GBV_ILN_4307 GBV_ILN_4313 GBV_ILN_4322 GBV_ILN_4323 GBV_ILN_4324 GBV_ILN_4325 GBV_ILN_4326 GBV_ILN_4333 GBV_ILN_4334 GBV_ILN_4335 GBV_ILN_4338 GBV_ILN_4367 GBV_ILN_4393 GBV_ILN_4700 AR 3 2020 100032- |
allfields_unstemmed |
10.1016/j.metip.2020.100032 doi (DE-627)DOAJ013701479 (DE-599)DOAJ45a2316901f847e8bf25776402229943 DE-627 ger DE-627 rakwb eng BF1-990 Ethan A. Meyers verfasserin aut Reducing the number of non-naïve participants in Mechanical Turk samples 2020 Text txt rdacontent Computermedien c rdamedia Online-Ressource cr rdacarrier Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. Mechanical Turk Nonnaivete HIT restriction Data quality Psychology Alexander C. Walker verfasserin aut Jonathan A. Fugelsang verfasserin aut Derek J. Koehler verfasserin aut In Methods in Psychology Elsevier, 2021 3(2020), Seite 100032- (DE-627)1689632550 25902601 nnns volume:3 year:2020 pages:100032- https://doi.org/10.1016/j.metip.2020.100032 kostenfrei https://doaj.org/article/45a2316901f847e8bf25776402229943 kostenfrei http://www.sciencedirect.com/science/article/pii/S2590260120300199 kostenfrei https://doaj.org/toc/2590-2601 Journal toc kostenfrei GBV_USEFLAG_A SYSFLAG_A GBV_DOAJ GBV_ILN_11 GBV_ILN_20 GBV_ILN_22 GBV_ILN_23 GBV_ILN_24 GBV_ILN_31 GBV_ILN_39 GBV_ILN_40 GBV_ILN_60 GBV_ILN_62 GBV_ILN_63 GBV_ILN_65 GBV_ILN_69 GBV_ILN_70 GBV_ILN_73 GBV_ILN_74 GBV_ILN_95 GBV_ILN_105 GBV_ILN_110 GBV_ILN_151 GBV_ILN_161 GBV_ILN_206 GBV_ILN_213 GBV_ILN_224 GBV_ILN_230 GBV_ILN_285 GBV_ILN_293 GBV_ILN_602 GBV_ILN_2001 GBV_ILN_2003 GBV_ILN_2005 GBV_ILN_2006 GBV_ILN_2007 GBV_ILN_2008 GBV_ILN_2009 GBV_ILN_2010 GBV_ILN_2011 GBV_ILN_2014 GBV_ILN_2015 GBV_ILN_2020 GBV_ILN_2021 GBV_ILN_2025 GBV_ILN_2026 GBV_ILN_2027 GBV_ILN_2034 GBV_ILN_2038 GBV_ILN_2044 GBV_ILN_2048 GBV_ILN_2049 GBV_ILN_2050 GBV_ILN_2055 GBV_ILN_2056 GBV_ILN_2059 GBV_ILN_2061 GBV_ILN_2064 GBV_ILN_2086 GBV_ILN_2088 GBV_ILN_2106 GBV_ILN_2110 GBV_ILN_2112 GBV_ILN_2122 GBV_ILN_2129 GBV_ILN_2143 GBV_ILN_2152 GBV_ILN_2153 GBV_ILN_2190 GBV_ILN_2232 GBV_ILN_2336 GBV_ILN_2470 GBV_ILN_2507 GBV_ILN_4012 GBV_ILN_4035 GBV_ILN_4037 GBV_ILN_4112 GBV_ILN_4125 GBV_ILN_4126 GBV_ILN_4242 GBV_ILN_4249 GBV_ILN_4251 GBV_ILN_4305 GBV_ILN_4306 GBV_ILN_4307 GBV_ILN_4313 GBV_ILN_4322 GBV_ILN_4323 GBV_ILN_4324 GBV_ILN_4325 GBV_ILN_4326 GBV_ILN_4333 GBV_ILN_4334 GBV_ILN_4335 GBV_ILN_4338 GBV_ILN_4367 GBV_ILN_4393 GBV_ILN_4700 AR 3 2020 100032- |
allfieldsGer |
10.1016/j.metip.2020.100032 doi (DE-627)DOAJ013701479 (DE-599)DOAJ45a2316901f847e8bf25776402229943 DE-627 ger DE-627 rakwb eng BF1-990 Ethan A. Meyers verfasserin aut Reducing the number of non-naïve participants in Mechanical Turk samples 2020 Text txt rdacontent Computermedien c rdamedia Online-Ressource cr rdacarrier Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. Mechanical Turk Nonnaivete HIT restriction Data quality Psychology Alexander C. Walker verfasserin aut Jonathan A. Fugelsang verfasserin aut Derek J. Koehler verfasserin aut In Methods in Psychology Elsevier, 2021 3(2020), Seite 100032- (DE-627)1689632550 25902601 nnns volume:3 year:2020 pages:100032- https://doi.org/10.1016/j.metip.2020.100032 kostenfrei https://doaj.org/article/45a2316901f847e8bf25776402229943 kostenfrei http://www.sciencedirect.com/science/article/pii/S2590260120300199 kostenfrei https://doaj.org/toc/2590-2601 Journal toc kostenfrei GBV_USEFLAG_A SYSFLAG_A GBV_DOAJ GBV_ILN_11 GBV_ILN_20 GBV_ILN_22 GBV_ILN_23 GBV_ILN_24 GBV_ILN_31 GBV_ILN_39 GBV_ILN_40 GBV_ILN_60 GBV_ILN_62 GBV_ILN_63 GBV_ILN_65 GBV_ILN_69 GBV_ILN_70 GBV_ILN_73 GBV_ILN_74 GBV_ILN_95 GBV_ILN_105 GBV_ILN_110 GBV_ILN_151 GBV_ILN_161 GBV_ILN_206 GBV_ILN_213 GBV_ILN_224 GBV_ILN_230 GBV_ILN_285 GBV_ILN_293 GBV_ILN_602 GBV_ILN_2001 GBV_ILN_2003 GBV_ILN_2005 GBV_ILN_2006 GBV_ILN_2007 GBV_ILN_2008 GBV_ILN_2009 GBV_ILN_2010 GBV_ILN_2011 GBV_ILN_2014 GBV_ILN_2015 GBV_ILN_2020 GBV_ILN_2021 GBV_ILN_2025 GBV_ILN_2026 GBV_ILN_2027 GBV_ILN_2034 GBV_ILN_2038 GBV_ILN_2044 GBV_ILN_2048 GBV_ILN_2049 GBV_ILN_2050 GBV_ILN_2055 GBV_ILN_2056 GBV_ILN_2059 GBV_ILN_2061 GBV_ILN_2064 GBV_ILN_2086 GBV_ILN_2088 GBV_ILN_2106 GBV_ILN_2110 GBV_ILN_2112 GBV_ILN_2122 GBV_ILN_2129 GBV_ILN_2143 GBV_ILN_2152 GBV_ILN_2153 GBV_ILN_2190 GBV_ILN_2232 GBV_ILN_2336 GBV_ILN_2470 GBV_ILN_2507 GBV_ILN_4012 GBV_ILN_4035 GBV_ILN_4037 GBV_ILN_4112 GBV_ILN_4125 GBV_ILN_4126 GBV_ILN_4242 GBV_ILN_4249 GBV_ILN_4251 GBV_ILN_4305 GBV_ILN_4306 GBV_ILN_4307 GBV_ILN_4313 GBV_ILN_4322 GBV_ILN_4323 GBV_ILN_4324 GBV_ILN_4325 GBV_ILN_4326 GBV_ILN_4333 GBV_ILN_4334 GBV_ILN_4335 GBV_ILN_4338 GBV_ILN_4367 GBV_ILN_4393 GBV_ILN_4700 AR 3 2020 100032- |
allfieldsSound |
10.1016/j.metip.2020.100032 doi (DE-627)DOAJ013701479 (DE-599)DOAJ45a2316901f847e8bf25776402229943 DE-627 ger DE-627 rakwb eng BF1-990 Ethan A. Meyers verfasserin aut Reducing the number of non-naïve participants in Mechanical Turk samples 2020 Text txt rdacontent Computermedien c rdamedia Online-Ressource cr rdacarrier Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. Mechanical Turk Nonnaivete HIT restriction Data quality Psychology Alexander C. Walker verfasserin aut Jonathan A. Fugelsang verfasserin aut Derek J. Koehler verfasserin aut In Methods in Psychology Elsevier, 2021 3(2020), Seite 100032- (DE-627)1689632550 25902601 nnns volume:3 year:2020 pages:100032- https://doi.org/10.1016/j.metip.2020.100032 kostenfrei https://doaj.org/article/45a2316901f847e8bf25776402229943 kostenfrei http://www.sciencedirect.com/science/article/pii/S2590260120300199 kostenfrei https://doaj.org/toc/2590-2601 Journal toc kostenfrei GBV_USEFLAG_A SYSFLAG_A GBV_DOAJ GBV_ILN_11 GBV_ILN_20 GBV_ILN_22 GBV_ILN_23 GBV_ILN_24 GBV_ILN_31 GBV_ILN_39 GBV_ILN_40 GBV_ILN_60 GBV_ILN_62 GBV_ILN_63 GBV_ILN_65 GBV_ILN_69 GBV_ILN_70 GBV_ILN_73 GBV_ILN_74 GBV_ILN_95 GBV_ILN_105 GBV_ILN_110 GBV_ILN_151 GBV_ILN_161 GBV_ILN_206 GBV_ILN_213 GBV_ILN_224 GBV_ILN_230 GBV_ILN_285 GBV_ILN_293 GBV_ILN_602 GBV_ILN_2001 GBV_ILN_2003 GBV_ILN_2005 GBV_ILN_2006 GBV_ILN_2007 GBV_ILN_2008 GBV_ILN_2009 GBV_ILN_2010 GBV_ILN_2011 GBV_ILN_2014 GBV_ILN_2015 GBV_ILN_2020 GBV_ILN_2021 GBV_ILN_2025 GBV_ILN_2026 GBV_ILN_2027 GBV_ILN_2034 GBV_ILN_2038 GBV_ILN_2044 GBV_ILN_2048 GBV_ILN_2049 GBV_ILN_2050 GBV_ILN_2055 GBV_ILN_2056 GBV_ILN_2059 GBV_ILN_2061 GBV_ILN_2064 GBV_ILN_2086 GBV_ILN_2088 GBV_ILN_2106 GBV_ILN_2110 GBV_ILN_2112 GBV_ILN_2122 GBV_ILN_2129 GBV_ILN_2143 GBV_ILN_2152 GBV_ILN_2153 GBV_ILN_2190 GBV_ILN_2232 GBV_ILN_2336 GBV_ILN_2470 GBV_ILN_2507 GBV_ILN_4012 GBV_ILN_4035 GBV_ILN_4037 GBV_ILN_4112 GBV_ILN_4125 GBV_ILN_4126 GBV_ILN_4242 GBV_ILN_4249 GBV_ILN_4251 GBV_ILN_4305 GBV_ILN_4306 GBV_ILN_4307 GBV_ILN_4313 GBV_ILN_4322 GBV_ILN_4323 GBV_ILN_4324 GBV_ILN_4325 GBV_ILN_4326 GBV_ILN_4333 GBV_ILN_4334 GBV_ILN_4335 GBV_ILN_4338 GBV_ILN_4367 GBV_ILN_4393 GBV_ILN_4700 AR 3 2020 100032- |
language |
English |
source |
In Methods in Psychology 3(2020), Seite 100032- volume:3 year:2020 pages:100032- |
sourceStr |
In Methods in Psychology 3(2020), Seite 100032- volume:3 year:2020 pages:100032- |
format_phy_str_mv |
Article |
institution |
findex.gbv.de |
topic_facet |
Mechanical Turk Nonnaivete HIT restriction Data quality Psychology |
isfreeaccess_bool |
true |
container_title |
Methods in Psychology |
authorswithroles_txt_mv |
Ethan A. Meyers @@aut@@ Alexander C. Walker @@aut@@ Jonathan A. Fugelsang @@aut@@ Derek J. Koehler @@aut@@ |
publishDateDaySort_date |
2020-01-01T00:00:00Z |
hierarchy_top_id |
1689632550 |
id |
DOAJ013701479 |
language_de |
englisch |
fullrecord |
<?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>01000caa a22002652 4500</leader><controlfield tag="001">DOAJ013701479</controlfield><controlfield tag="003">DE-627</controlfield><controlfield tag="005">20230310060413.0</controlfield><controlfield tag="007">cr uuu---uuuuu</controlfield><controlfield tag="008">230226s2020 xx |||||o 00| ||eng c</controlfield><datafield tag="024" ind1="7" ind2=" "><subfield code="a">10.1016/j.metip.2020.100032</subfield><subfield code="2">doi</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-627)DOAJ013701479</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)DOAJ45a2316901f847e8bf25776402229943</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-627</subfield><subfield code="b">ger</subfield><subfield code="c">DE-627</subfield><subfield code="e">rakwb</subfield></datafield><datafield tag="041" ind1=" " ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="050" ind1=" " ind2="0"><subfield code="a">BF1-990</subfield></datafield><datafield tag="100" ind1="0" ind2=" "><subfield code="a">Ethan A. Meyers</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">Reducing the number of non-naïve participants in Mechanical Turk samples</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="c">2020</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="a">Text</subfield><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="a">Computermedien</subfield><subfield code="b">c</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="a">Online-Ressource</subfield><subfield code="b">cr</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants.</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Mechanical Turk</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Nonnaivete</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">HIT restriction</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Data quality</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Psychology</subfield></datafield><datafield tag="700" ind1="0" ind2=" "><subfield code="a">Alexander C. Walker</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="700" ind1="0" ind2=" "><subfield code="a">Jonathan A. Fugelsang</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="700" ind1="0" ind2=" "><subfield code="a">Derek J. Koehler</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="773" ind1="0" ind2="8"><subfield code="i">In</subfield><subfield code="t">Methods in Psychology</subfield><subfield code="d">Elsevier, 2021</subfield><subfield code="g">3(2020), Seite 100032-</subfield><subfield code="w">(DE-627)1689632550</subfield><subfield code="x">25902601</subfield><subfield code="7">nnns</subfield></datafield><datafield tag="773" ind1="1" ind2="8"><subfield code="g">volume:3</subfield><subfield code="g">year:2020</subfield><subfield code="g">pages:100032-</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doi.org/10.1016/j.metip.2020.100032</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doaj.org/article/45a2316901f847e8bf25776402229943</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">http://www.sciencedirect.com/science/article/pii/S2590260120300199</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="856" ind1="4" ind2="2"><subfield code="u">https://doaj.org/toc/2590-2601</subfield><subfield code="y">Journal toc</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_USEFLAG_A</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">SYSFLAG_A</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_DOAJ</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_11</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_20</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_22</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_23</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_24</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_31</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_39</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_40</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_60</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_62</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_63</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_65</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_69</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_70</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_73</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_74</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_95</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_105</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_110</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_151</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_161</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_206</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_213</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_224</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_230</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_285</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_293</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_602</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2001</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2003</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2005</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2006</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2007</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2008</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2009</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2010</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2011</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2014</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2015</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2020</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2021</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2025</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2026</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2027</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2034</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2038</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2044</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2048</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2049</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2050</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2055</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2056</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2059</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2061</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2064</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2086</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2088</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2106</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2110</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2112</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2122</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2129</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2143</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2152</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2153</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2190</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2232</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2336</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2470</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2507</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4012</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4035</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4037</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4112</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4125</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4126</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4242</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4249</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4251</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4305</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4306</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4307</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4313</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4322</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4323</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4324</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4325</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4326</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4333</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4334</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4335</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4338</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4367</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4393</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4700</subfield></datafield><datafield tag="951" ind1=" " ind2=" "><subfield code="a">AR</subfield></datafield><datafield tag="952" ind1=" " ind2=" "><subfield code="d">3</subfield><subfield code="j">2020</subfield><subfield code="h">100032-</subfield></datafield></record></collection>
|
callnumber-first |
B - Philosophy, Psychology, Religion |
author |
Ethan A. Meyers |
spellingShingle |
Ethan A. Meyers misc BF1-990 misc Mechanical Turk misc Nonnaivete misc HIT restriction misc Data quality misc Psychology Reducing the number of non-naïve participants in Mechanical Turk samples |
authorStr |
Ethan A. Meyers |
ppnlink_with_tag_str_mv |
@@773@@(DE-627)1689632550 |
format |
electronic Article |
delete_txt_mv |
keep |
author_role |
aut aut aut aut |
collection |
DOAJ |
remote_str |
true |
callnumber-label |
BF1-990 |
illustrated |
Not Illustrated |
issn |
25902601 |
topic_title |
BF1-990 Reducing the number of non-naïve participants in Mechanical Turk samples Mechanical Turk Nonnaivete HIT restriction Data quality |
topic |
misc BF1-990 misc Mechanical Turk misc Nonnaivete misc HIT restriction misc Data quality misc Psychology |
topic_unstemmed |
misc BF1-990 misc Mechanical Turk misc Nonnaivete misc HIT restriction misc Data quality misc Psychology |
topic_browse |
misc BF1-990 misc Mechanical Turk misc Nonnaivete misc HIT restriction misc Data quality misc Psychology |
format_facet |
Elektronische Aufsätze Aufsätze Elektronische Ressource |
format_main_str_mv |
Text Zeitschrift/Artikel |
carriertype_str_mv |
cr |
hierarchy_parent_title |
Methods in Psychology |
hierarchy_parent_id |
1689632550 |
hierarchy_top_title |
Methods in Psychology |
isfreeaccess_txt |
true |
familylinks_str_mv |
(DE-627)1689632550 |
title |
Reducing the number of non-naïve participants in Mechanical Turk samples |
ctrlnum |
(DE-627)DOAJ013701479 (DE-599)DOAJ45a2316901f847e8bf25776402229943 |
title_full |
Reducing the number of non-naïve participants in Mechanical Turk samples |
author_sort |
Ethan A. Meyers |
journal |
Methods in Psychology |
journalStr |
Methods in Psychology |
callnumber-first-code |
B |
lang_code |
eng |
isOA_bool |
true |
recordtype |
marc |
publishDateSort |
2020 |
contenttype_str_mv |
txt |
container_start_page |
100032 |
author_browse |
Ethan A. Meyers Alexander C. Walker Jonathan A. Fugelsang Derek J. Koehler |
container_volume |
3 |
class |
BF1-990 |
format_se |
Elektronische Aufsätze |
author-letter |
Ethan A. Meyers |
doi_str_mv |
10.1016/j.metip.2020.100032 |
author2-role |
verfasserin |
title_sort |
reducing the number of non-naïve participants in mechanical turk samples |
callnumber |
BF1-990 |
title_auth |
Reducing the number of non-naïve participants in Mechanical Turk samples |
abstract |
Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. |
abstractGer |
Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. |
abstract_unstemmed |
Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants. |
collection_details |
GBV_USEFLAG_A SYSFLAG_A GBV_DOAJ GBV_ILN_11 GBV_ILN_20 GBV_ILN_22 GBV_ILN_23 GBV_ILN_24 GBV_ILN_31 GBV_ILN_39 GBV_ILN_40 GBV_ILN_60 GBV_ILN_62 GBV_ILN_63 GBV_ILN_65 GBV_ILN_69 GBV_ILN_70 GBV_ILN_73 GBV_ILN_74 GBV_ILN_95 GBV_ILN_105 GBV_ILN_110 GBV_ILN_151 GBV_ILN_161 GBV_ILN_206 GBV_ILN_213 GBV_ILN_224 GBV_ILN_230 GBV_ILN_285 GBV_ILN_293 GBV_ILN_602 GBV_ILN_2001 GBV_ILN_2003 GBV_ILN_2005 GBV_ILN_2006 GBV_ILN_2007 GBV_ILN_2008 GBV_ILN_2009 GBV_ILN_2010 GBV_ILN_2011 GBV_ILN_2014 GBV_ILN_2015 GBV_ILN_2020 GBV_ILN_2021 GBV_ILN_2025 GBV_ILN_2026 GBV_ILN_2027 GBV_ILN_2034 GBV_ILN_2038 GBV_ILN_2044 GBV_ILN_2048 GBV_ILN_2049 GBV_ILN_2050 GBV_ILN_2055 GBV_ILN_2056 GBV_ILN_2059 GBV_ILN_2061 GBV_ILN_2064 GBV_ILN_2086 GBV_ILN_2088 GBV_ILN_2106 GBV_ILN_2110 GBV_ILN_2112 GBV_ILN_2122 GBV_ILN_2129 GBV_ILN_2143 GBV_ILN_2152 GBV_ILN_2153 GBV_ILN_2190 GBV_ILN_2232 GBV_ILN_2336 GBV_ILN_2470 GBV_ILN_2507 GBV_ILN_4012 GBV_ILN_4035 GBV_ILN_4037 GBV_ILN_4112 GBV_ILN_4125 GBV_ILN_4126 GBV_ILN_4242 GBV_ILN_4249 GBV_ILN_4251 GBV_ILN_4305 GBV_ILN_4306 GBV_ILN_4307 GBV_ILN_4313 GBV_ILN_4322 GBV_ILN_4323 GBV_ILN_4324 GBV_ILN_4325 GBV_ILN_4326 GBV_ILN_4333 GBV_ILN_4334 GBV_ILN_4335 GBV_ILN_4338 GBV_ILN_4367 GBV_ILN_4393 GBV_ILN_4700 |
title_short |
Reducing the number of non-naïve participants in Mechanical Turk samples |
url |
https://doi.org/10.1016/j.metip.2020.100032 https://doaj.org/article/45a2316901f847e8bf25776402229943 http://www.sciencedirect.com/science/article/pii/S2590260120300199 https://doaj.org/toc/2590-2601 |
remote_bool |
true |
author2 |
Alexander C. Walker Jonathan A. Fugelsang Derek J. Koehler |
author2Str |
Alexander C. Walker Jonathan A. Fugelsang Derek J. Koehler |
ppnlink |
1689632550 |
callnumber-subject |
BF - Psychology |
mediatype_str_mv |
c |
isOA_txt |
true |
hochschulschrift_bool |
false |
doi_str |
10.1016/j.metip.2020.100032 |
callnumber-a |
BF1-990 |
up_date |
2024-07-03T19:05:58.019Z |
_version_ |
1803585908139622400 |
fullrecord_marcxml |
<?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>01000caa a22002652 4500</leader><controlfield tag="001">DOAJ013701479</controlfield><controlfield tag="003">DE-627</controlfield><controlfield tag="005">20230310060413.0</controlfield><controlfield tag="007">cr uuu---uuuuu</controlfield><controlfield tag="008">230226s2020 xx |||||o 00| ||eng c</controlfield><datafield tag="024" ind1="7" ind2=" "><subfield code="a">10.1016/j.metip.2020.100032</subfield><subfield code="2">doi</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-627)DOAJ013701479</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)DOAJ45a2316901f847e8bf25776402229943</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-627</subfield><subfield code="b">ger</subfield><subfield code="c">DE-627</subfield><subfield code="e">rakwb</subfield></datafield><datafield tag="041" ind1=" " ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="050" ind1=" " ind2="0"><subfield code="a">BF1-990</subfield></datafield><datafield tag="100" ind1="0" ind2=" "><subfield code="a">Ethan A. Meyers</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">Reducing the number of non-naïve participants in Mechanical Turk samples</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="c">2020</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="a">Text</subfield><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="a">Computermedien</subfield><subfield code="b">c</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="a">Online-Ressource</subfield><subfield code="b">cr</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">Using participants who have been previously exposed to experimental stimuli (referred to as non-naïveté) can reduce effect sizes. The workforce of Amazon's Mechanical Turk is particularly vulnerable to this problem and solutions are usually cost and time inefficient and of mixed effectiveness. In response to this problem and its currently underwhelming solutions, we tested various participant recruitment strategies designed to recruit participants naïve to frequently used experimental stimuli. We collected samples using maximum HIT restrictions (50 for Experiment 1 and 2, 500 for Experiment 2) and TurkPrime's (now CloudResearch) naiveté feature and compared them to samples recruited with standard restrictions (95% HIT approval rating). In these comparisons, we replicated past findings where using nonnaïve (vs. naïve) participants has been shown to reduce effect sizes and affect performance on a variety of tasks (e.g., the Cognitive Reflection Test, a Public Goods Game). We demonstrate that restricting by the maximum number of HITs heavily reduces the number of “experienced” research subjects in samples but necessitates some sacrifice in data quality and collection speed. We discuss the pragmatics of our method, its limitations, and future directions for solving the problem of non-naïveté on Mechanical Turk. For those looking to avoid this issue, we recommend setting a maximum HIT restriction of 50 when recruiting participants.</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Mechanical Turk</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Nonnaivete</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">HIT restriction</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Data quality</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Psychology</subfield></datafield><datafield tag="700" ind1="0" ind2=" "><subfield code="a">Alexander C. Walker</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="700" ind1="0" ind2=" "><subfield code="a">Jonathan A. Fugelsang</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="700" ind1="0" ind2=" "><subfield code="a">Derek J. Koehler</subfield><subfield code="e">verfasserin</subfield><subfield code="4">aut</subfield></datafield><datafield tag="773" ind1="0" ind2="8"><subfield code="i">In</subfield><subfield code="t">Methods in Psychology</subfield><subfield code="d">Elsevier, 2021</subfield><subfield code="g">3(2020), Seite 100032-</subfield><subfield code="w">(DE-627)1689632550</subfield><subfield code="x">25902601</subfield><subfield code="7">nnns</subfield></datafield><datafield tag="773" ind1="1" ind2="8"><subfield code="g">volume:3</subfield><subfield code="g">year:2020</subfield><subfield code="g">pages:100032-</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doi.org/10.1016/j.metip.2020.100032</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doaj.org/article/45a2316901f847e8bf25776402229943</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">http://www.sciencedirect.com/science/article/pii/S2590260120300199</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="856" ind1="4" ind2="2"><subfield code="u">https://doaj.org/toc/2590-2601</subfield><subfield code="y">Journal toc</subfield><subfield code="z">kostenfrei</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_USEFLAG_A</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">SYSFLAG_A</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_DOAJ</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_11</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_20</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_22</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_23</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_24</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_31</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_39</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_40</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_60</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_62</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_63</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_65</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_69</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_70</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_73</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_74</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_95</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_105</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_110</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_151</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_161</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_206</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_213</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_224</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_230</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_285</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_293</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_602</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2001</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2003</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2005</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2006</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2007</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2008</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2009</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2010</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2011</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2014</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2015</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2020</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2021</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2025</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2026</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2027</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2034</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2038</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2044</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2048</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2049</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2050</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2055</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2056</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2059</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2061</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2064</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2086</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2088</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2106</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2110</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2112</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2122</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2129</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2143</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2152</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2153</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2190</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2232</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2336</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2470</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_2507</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4012</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4035</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4037</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4112</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4125</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4126</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4242</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4249</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4251</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4305</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4306</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4307</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4313</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4322</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4323</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4324</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4325</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4326</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4333</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4334</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4335</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4338</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4367</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4393</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">GBV_ILN_4700</subfield></datafield><datafield tag="951" ind1=" " ind2=" "><subfield code="a">AR</subfield></datafield><datafield tag="952" ind1=" " ind2=" "><subfield code="d">3</subfield><subfield code="j">2020</subfield><subfield code="h">100032-</subfield></datafield></record></collection>
|
score |
7.399972 |