Determining Quantitative Trustworthiness in Research Results

Determining Quantitative Trustworthiness in Research Results

Dissertation Chapter 3: Research Method

• Introduction (no subheading)
• Research Methodology and Design
• Population and Sample
• Materials or Instrumentation
• Operational Definitions of Variables (quantitative only)
• Study Procedures
• Data Collection and Analysis
• Assumptions
• Limitations
• Delimitations
• Ethical Assurances
• Summary

Introduction • Briefly reintroduces the problem and purpose of the study

• Briefly describes the overall intent and components of the chapter
• Leads into the description of the research methodology and design
• Does not add any new or related information

Research Methodology and Design • Describes the research methodology (i.e., quantitative, qualitative, mixed) and

corresponding specific design

• Elaborates on and defends how the chosen research methodology and design are appropriate to accurately address the study’s problem, purpose, and research questions
• Identifies alternative methodologies and designs indicating why they are less appropriate to accurately address the study’s problem, purpose, and research questions

Research Methodology and Design • A quantitative methodology involves the use of variables to measure an effect or

a relationship

• Employs variables to establish cause-and-effect • Independent/predictor variable is used as an employed effect to be measured
• Dependent/outcome variable is used to measure the employed effect
• Indicates the effect of the independent/predictor variable onto the dependent/outcome variable
• With the use of descriptive and inferential statistics, a statistically significant/non- significant effect or relationship involving two or more variables can be established
• Can be used to determine the size of the effect or to what degree/extent the dependent/outcome variable was changed by the independent/predictor variable
• Can be used to predict future outcomes
• Use what is/are the effect(s) of….. on…..
• Do not use “effect” in qualitative research

Research Methodology and Design • A quantitative methodology involves the use of variables to measure an effect or

a relationship

• Employs variables to establish non-casual relationships • Can be used to establish a statistically significant/non-significant relationship involving

two variables

• Can be used to only determine the strength and direction of the relationship
• Control variables are held constant throughout the experiment to eliminate/reduce their potential effect
• Extraneous/confounding/intervening variables hide/alter the true effect of the intended independent/predictor variable in the experiment, negatively impacting the true results

Research Methodology and Design • Quantitative Methodology – involves descriptive and inferential statistical analyses to test

a set of hypotheses for significance

• Instruments used to derive numerical data
• Correlation – attempts to determine to what extent two variables are related
• Regression – attempts to predict the value of one variable from another variable when a causal relationship exists between the two variables
• Multiple Regression – attempts to predict the value of a one variable based on the value of two or more other variables.
• Group Comparison – attempts to determine differences between two or more groups based on the measured effect of an independent variable on to a dependent variable • Experimental: randomization selection and assignment of participants
• Quasi-Experimental: randomization does not occur
• Non-Experimental: causal comparative/ex post facto (data are obtained from pre-formed groups without independent variable manipulation, already occurred)

Research Methodology and Design • Qualitative Methodology – an in-depth exploration to gain a greater understanding; can

also utilize an analysis of numerical data employing descriptive statistics but does not employ inferential statistical analyses to address a set of hypotheses

• Surveys, individual interviews, focus group interviews, anecdotal records, and observations are used to derive numerical and/or non-numerical descriptive information
• Phenomenology – an in-depth exploration to gain a deeper understanding of individuals’ perceptions and lived experiences regarding some phenomenon from their perspective
• Ethnography – an in-depth exploration to gain a deeper understanding of a particular culture or some facet of it from the perspective of the culture’s members
• Grounded Theory – an in-depth exploration to gain a deeper understanding for the development of a theory to advance, refine, and expand a body of knowledge or ground a theory in the context of the phenomenon under study

Research Methodology and Design • Narrative – an in-depth exploration to gain a deeper understanding of individuals’

meanings they assign to their particular experiences, typically derived from one or a small number of participants involving rich and free-ranging discourse

• Case Study – an in-depth exploration to gain a greater understanding of a single individual, small group, event, place, phenomenon, or other type of circumstance from the participant’s perspective utilizing multiple or triangulated data sources; allows key characteristics, meanings, themes, trends, and implications to be clarified for application and prediction

Population and Sample

• Population – the entirety from the sample is drawn involving people, objects, events, etc.
• Sample – the smaller more manageable subset representative of the larger population
• Describe the population including the estimated size and characteristics and why it is appropriate to be used to address the problem, purpose, and research questions
• Describe the sample including how it was derived and why it is appropriate to be used to address the problem, purpose, and research questions
• Describe the sampling procedure involving how participants will be recruited • Random and convenience sampling are the most common

Population and Sample

• The purpose of a smaller sample is to develop an inference/conclusion about the larger population
• In qualitative research, the smaller sample must be representative or characteristic of the larger population for accuracy in the transferability of the sample results to the population; the sample size is based on the type of qualitative design and when data saturation occurs
• In quantitative research, the smaller sample must be representative or characteristic of the larger population for accuracy in the generalizability of the sample results to the population

Population and Sample

• A sample size is based on a power analysis to determine the smallest sample size suitable to detect an effect at the desired level of significance
• A power analysis yields statistical power, ensuring the probability of a statistical test accurately rejects the null hypothesis
• The higher the statistical power, the lower the probability of making a Type II error (false negative)
• A type I error occurs when the null hypothesis is rejected but it is actually true (false- positive) …the intervention really didn’t work… but concluded it did…
• A type II error occurs when the null hypothesis is failed to be rejected but it is actually false (false negative) …the intervention really did work…but concluded it did not…
• Null Hypothesis: non-significant outcome – failed to be rejected (never accepted) or rejected
• Alternative Hypothesis: significant outcome – accepted

Materials and Instrumentation

• Describe existing materials and/or instruments used to collect data, including evidence of reliability and validity
• Note evidence of permission to use existing materials and/or instruments
• Describe any needed field or pilot testing of materials and/or instruments including evidence of reliability and validity

Operational Definition of Variables

• For the quantitative method only, describe each variable and how it was used in the study (e.g., independent/dependent, predictor/outcome)
• Describe how each instrument was used to measure each variable
• Describe the level of measurement for each variable (e.g., nominal, ordinal, interval, ratio) and potential scores for each variable • Nominal – variables are named or labeled in no specific order, used for classification

with no mathematical value, often used in surveys (e.g., gender)

• Ordinal – variables are used to depict a specific order or ranking, used to determine order/degree of satisfaction or agreement (e.g., strongly disagree to strongly agree)
• Interval – labels, orders/ranks, combined with an equal interval between each value (e.g., temperature)
• Ratio – interval data with a natural zero (e.g., weight)

Study Procedures

• Provide a detailed description (recipe) of how the study was precisely conducted
• Describe the exact steps followed to collect data, addressing what data, how it was collected, when it was collected, where it was collected, and from whom it was collected within sufficient detail for the study to be replicated with as much consistency as humanly possible

Data Collection and Procedures

• Describe the collected data followed by the strategies used to analyze the collected data (e.g., coding, statistical analyses, software)
• For quantitative designs, describe the analysis used to test each hypothesis, provide evidence the statistical test chosen was appropriate to assess the hypotheses and the collected data met the assumptions of the statistical tests (e.g., tests for normality) • Parametric – collected data resembles a normal distribution (bell-shaped curve),

greater statistical power, more likely to reveal a significant effect when one truly exists

• Non-Parametric – collected data do not rely on any distribution shape (distribution- free)
• For qualitative designs, describe how the collected data were processed and analyzed, including any triangulation efforts, and explain the role of the researcher during data collection and analysis

Assumptions, Limitations, Delimitations

• Assumptions are beliefs about the study considered to be true but without supportive evidence • For example: study participants answered all questions honestly without fear of repercussions or what

the researcher may be perceived as desirable

• Assumptions are beyond the researcher’s control
• Limitations are the inherent weaknesses/flaws within the given research design
• For example: insufficient sample size, study time constraints, limited access to data
• Limitations are beyond the researcher’s control
• Delimitations are boundaries established by the researcher to define and limit the scope of the study, often to support the feasibility of conducting the study • For example: utilizing students in grade 10 to evaluate a reading achievement improvement strategy

among high school at-risk students, rather than utilizing a sample of high school students in grades 9 though 12

• Delimitations are within the researcher’s control
• Assumptions, limitations, and delimitations can be threats to the accuracy or trustworthiness in quantitative and qualitative research results

Poll: Assess Your Understanding

Which one of the following portrays the most accurate representation of an assumption, limitation, and delimitation:

1. Data saturation was unable to be achieved due to the time constraints of the student’s dissertation completion course timeline.
2. Second grade student participants responded accurately when asked about how they learn best in school.
3. Participants were only recruited through the use of a Facebook request.

Poll: Assess Your Understanding

1. Data saturation was unable to be achieved due to the time constraints of the student’s dissertation completion course timeline. Limitation
2. Second grade student participants responded accurately when asked about how they learn best in school. Assumption
3. Participants were only recruited through the use of a Facebook request. Delimitation

Poll: Assess Your Understanding

Which one of the following portrays the most accurate representation of an assumption, limitation, and delimitation?

1. The sample located in one part of a region will be representative of the entire population in this region.
2. Instrumentation involved only Likert scale responses.
3. Several participants withdrew prior to study completion.

Poll: Assess Your Understanding

1. The sample located in one part of a region will be representative of the entire population in this region. Assumption
2. Instrumentation involved only Likert scale responses. Delimitation
3. Several participants withdrew prior to study completion. Limitation

Qualitative and Quantitative Trustworthiness

Criteria for Determining Quantitative Trustworthiness in Research Results • Internal Validity – Results accurately

represented what the study intended to address

• External Validity – Results can be generalized/transferred/applied from the representative smaller sample to the larger population
• Reliability – Results are consistently the same/similar if the study was repeated
• Objectivity – Results were objectively realized as much as humanly possible

Criteria for Determining Qualitative Trustworthiness in Research Results

• Credibility – Results are credible/believable from the perspective of the participant
• Transferability – Results can be generalized/transferred/applied to other related contexts/settings
• Dependability – Results are dependable or could be repeated
• Confirmability – Results can be confirmed/corroborated by others

Ethical Assurances

• Confirm the study received approval from NCU’s IRB prior to data collection
• Describe how confidentiality/anonymity was enforced
• Identify how the collected data will be securely stored
• Present strategies used to prevent researcher biases and experiences from influencing the collection and analysis of the data/findings

Summary

• Summarize major points without providing new information
• Logically lead into the next chapter

Study Feasibility Is the study specialization related, is it doable, is it needed and will it

add to the research knowledge base?

STUDY FEASIBILITY – CONSIDERATIONS

Assess whether your study CAN be conducted practicality • Population

• Accessibility
• Time
• Factors for success

Obtainable, accessible, timely, actionable, reasonable, manageable

Determine objectively and rationally (you may be very passionate about your overall topic – remember this is a learning process)

FEASIBILITY

Who? How long? What skills?

• Population Identification
• Vulnerable populations
• Population knowledge & experience
• Access for recruitment
• Recruitment process
• Quantity of participants
• Personal timelines
• Scope & duration
• Scope creep
• Weekly commitment
• Constraining time factors
• Research methods
• Research design
• Data analysis
• Library research
• Tutoring services
• Editing services

IRB Tip! Reminder: Students cannot recruit people they already have a relationship with either personal or professional.

VULNERABILITY

There are two important types of vulnerability:

• “Decisional impairment, whereby potential subjects lack the capacity to make autonomous decisions in their own interest, perhaps as a result of undue influence/inducement
• Situational/positional vulnerability, whereby potential participants may be subjected to coercion”

[Adam Mrdjenovich, Ph.D, 2016]

IRB Tip! If you are considering these populations, red lights should go off and you should dig deeper and talk through the research with IRB. It’s better to discuss early then go through many IRB revisions.

VULNERABILITY

Vulnerable under federal regulations:

• Pregnant women, fetuses, and neonates
• Children
• Prisoners

Other vulnerable populations:

• Racial Minorities
• Institutionalized
• Physically Handicapped

Continued:

• Students
• Employees
• Patients
• Educationally/Economically Disadvantaged
• Impaired decision making (mentally ill/ dementia/traumatic brain injury)
• Illiterate or low fluency in language of study

IRB Tip! If you are considering these populations, red lights should go off and you should dig deeper and talk through the research with IRB. It’s better to discuss early then go through many IRB revisions.

 

PLACE THE ORDER WITH US TODAY AND GET A PERFECT SCORE!!!