3:40 - 40 participants gives a 15% margin of error and 95% confidence level (binary metrics)

Big five trait scores for 307,313 people from many different countries.

And remember, keeping screeners under 12 questions is the magic number to prevent attrition.

“significant” p-value ≠ “significant” finding: The significance of statistical evidence for the true X (i.e., statistical significance of the p-value for the estimate of the true X) says absolutely nothing about the practical/scientific significance of the true X. That is, significance of evidence is not evidence of significance. Increasing your sample size in no way increases the practical/scientific significance of your practical/scientific hypothesis. “significant” p-value = “discernible” finding: The significance of statistical evidence for the true X does tell us how well the estimate can discern the true X. That is, significance of evidence is evidence of discernibility. Increasing your sample size does increase how well your finding can discern your practical/scientific hypothesis.

Opt-in samples are about half as accurate as probability-based panels

This is SenseMaker in its most simple form, usually structured to have an open (non-hypothesis) question (commonly referred to as a ‘prompting question’) to collect a micro-narrative at the start. This is then followed by a range of triads (triangles), dyads (sliders), stones canvases, free text questions and multiple choice questions. The reason or value for using Sensemaker: Open free text questions are used at the beginning as a way of scanning for diversity of narratives and experiences. This is a way to remain open to ‘unknown unknowns’. The narrative is then followed by signifier questions that allow the respondent to add layers of meaning and codification to the narrative (or experience) in order to allow for mixed methods analysis, to map and explore patterns.

Info, research, questionnaires/scales, info on application to specific topics

With this thread, you’ll learn 9 lessons: 1. The Data Quality Framework 2. The Participant Relationship 3. Perfect introductions 4. Instructions that work 5. Helpful signposting 6. An enjoyable experience 7. Getting feedback 8. Progressive piloting 9. Types of quality control

Qualtrics recommendation is to use the commitment request as that performed the best. However, the textual and factual attention checks also performed better than the control.

Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.

A survey’s completion rate is one of its most important data quality measures. There are quite a few published studies examining web survey completion rate through experimental approaches. In this study, we expand the existing literature by examining the predictors of web survey completion rate using 25,080 real-world web surveys conducted by a single online panel. Our findings are consistent with the literature on some dimensions, such as finding a negative relationship between completion rate and survey length and question difficulty. Also, surveys without progress bars have higher completion rates than surveys with progress bars. This study also generates new insights into survey design features, such as the impact of the first question type and length on completion rate. More: https://twitter.com/nielsmede/status/1576234663341064192?s=20&t=kSwdGBBuVv1yiqo1lE4vbw

The Data Playbook is 120 exercises, games, scenarios, slides and checklists to assist you and your teams on your data journey. The social learning content is designed for teams to have discussions and activities across the data lifecycle in short 30 minute to 1 hour sessions.

Broadly, these feedback surveys can be categorised into five groups: the pointless; the self-important; the immoral; the demanding; and the downright weird:

EMERGE (Evidence-based Measures of Empowerment for Research on Gender Equality) is a project focused on gender equality and empowerment measures to monitor and evaluate health programs and to track progress on UN Sustainable Development Goal (SDG) 5: To Achieve Gender Equality and Empower All Girls. As reported by UN Women (2018), only 2 of the 14 SDG 5 indicators have accepted methodologies for measurement and data widely available. Of the remaining 12, 9 are indicators for which data are collected and available in only a limited number of countries. This assessment suggests notable measurement gaps in the state of gender equality and empowerment worldwide. EMERGE aims to improve the science of gender equality and empowerment measurement by identifying these gaps through the compilation and psychometric evaluation of available measures and supporting scientifically rigorous measure development research in India.

The Meta-Analysis Learning Information Center (MALIC) believes in equitably providing cutting-edge and up-to-date techniques in meta-analysis to researchers in the social sciences, particularly those in education and STEM education.

awful example of landing page!

For HCI survey research broadly, we recommend using a question similar to the first question in [2]’s measure (as quoted in [3]) – “Are you…?” with three response options: “man,” “woman,” “something else: specify [text box]” – and allowing respondents to choose multiple options. This question will not identify all trans participants [3], but is inclusive to non-binary and trans people and will identify gender at a level necessary for most HCI research. To reduce trolling, we recommend providing the fill-in-theblank text box as a second step only for those respondents who choose the “something else” option.

I love it that one of my students suggested we change the default “Other (please specify“) option to “Not Listed (please specify)“ in a demographic survey. Explicitly *not* “othering“ participants while still asking for the info we want. Any implied failure is on us, not them.

40 participants is an appropriate number for most quantitative studies, but there are cases where you can recruit fewer users.

In our work at BehaviourWorks Australia (BWA) we are frequently asked ‘What does the research say about getting audience Y to do behaviour X?’. When our partners need an urgent answer we often provide it using a Rapid Review. In this article I explain Rapid Reviews, why you should do them, and a process that you can follow to conduct one. What is a Rapid Review? Rapid Reviews are “a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner” [1]. Indeed, with sufficient resources (e.g., multiple staff working simultaneously) you can do a Rapid Review in less than a day. The outputs of these reviews are, of course, brief and descriptive, but they can be very useful where rapid evidence is needed, for example, in addressing COVID-19. Rapid Reviews can therefore provide detailed research within reduced timeframes and also meet most academic requirements by being standardised and reproducible. They are often, but not always, publishable in peer-reviewed academic journals.

The Research Methods Knowledge Base is a comprehensive web-based textbook that addresses all of the topics in a typical introductory undergraduate or graduate course in social research methods. It covers the entire research process including: formulating research questions; sampling (probability and nonprobability); measurement (surveys, scaling, qualitative, unobtrusive); research design (experimental and quasi-experimental); data analysis; and, writing the research paper. It also addresses the major theoretical and philosophical underpinnings of research including: the idea of validity in research; reliability of measures; and ethics.

This class covers a range of different topics that build on top of each other. For example, in the first tutorial, you will learn how to collect data from Twitter, and in subsequent tutorials you will learn how to analyze those data using automated text analysis techniques. For this reason, you may find it difficult to jump towards one of the most advanced issues before covering the basics. Introduction: Strengths and Weaknesses of Text as Data Application Programming Interfaces Screen-Scraping Basic Text Analysis Dictionary-Based Text Analysis Topic Modeling Text Networks Word Embeddings

a measurement instrument for evaluating susceptibility to seven social influence principles, namely social learning, social comparison, social norms, social facilitation, social cooperation, social competition, and social recognition