all statistics

jimmy

beautifully simplified

statistics

Statistical Significance

What does "statistical significance" really mean? Many researchers get very excited when they have discovered a "statistically significant" finding, without really understanding what it means. When a statistic is significant, it simply means that you are very sure that the statistic is reliable. It doesn't mean the finding is important or that it has any decision-making utility. For example, suppose we give 1,000 people an IQ test, and we ask if there is a significant difference between male and female scores. The mean score for males is 98 and the mean score for females is 100. We use an independent groups t-test and find that the difference is significant at the .001 level. The big question is, "So what?". The difference between 98 and 100 on an IQ test is a very small difference...so small, in fact, that it's not even important. Then why did the t-statistic come out significant? Because there was a large sample size. When you have a large sample size, very small differences will be detected as significant. This means that you are very sure that the difference is real (i.e., it didn't happen by fluke). It doesn't mean that the difference is large or important. If we had only given the IQ test to 25 people instead of 1,000, the two-point difference between males and females would not have been significant. Significance is a statistical term that tells how sure you are that a difference or relationship exists. To say that a significant difference or relationship exists only tells half the story. We might be very sure that a relationship exists, but is it a strong, moderate, or weak relationship? After finding a significant relationship, it is important to evaluate its strength. Significant relationships can be strong or weak. Significant differences can be large or small. It just depends on your sample size. Many researchers use the word "significant" to describe a finding that may have decision-making utility to a client. From a statistician's viewpoint, this is an incorrect use of the word. However, the word "significant" has virtually universal meaning to the public. Thus, many researchers use the word "significant" to describe a difference or relationship that may be strategically important to a client (regardless of any statistical tests). In these situations, the word "significant" is used to advise a client to take note of a particular difference or relationship because it may be relevant to the company's strategic plan. The word "significant" is not the exclusive domain of statisticians and either use is correct in the business world. Thus, for the statistician, it may be wise to adopt a policy of always referring to "statistical significance" rather than simply "significance" when communicating with the public. One-Tailed and Two-Tailed Significance Tests One important concept in significance testing is whether you use a one-tailed or two-tailed test of significance. The answer is that it depends on your hypothesis. When your research hypothesis states the direction of the difference or relationship, then you use a one-tailed probability. For example, a one-tailed test would be used to test these null hypotheses: Females will not score significantly higher than males on an IQ test. Blue collar workers are will not buy significantly more product than white collar workers. Superman is not significantly stronger than the average person. In each case, the null hypothesis (indirectly) predicts the direction of the difference. A two-tailed test would be used to test these null hypotheses: There will be no significant difference in IQ scores between males and females. There will be no significant difference in the amount of product purchased between blue collar and white collar workers. There is no significant difference in strength between Superman and the average person. The one-tailed probability is exactly half the value of the two-tailed probability. There is a raging controversy (for about the last hundred years) on whether or not it is ever appropriate to use a one-tailed test. The rationale is that if you already know the direction of the difference, why bother doing any statistical tests. While it is generally safest to use a two-tailed tests, there are situations where a one-tailed test seems more appropriate. The bottom line is that it is the choice of the researcher whether to use one-tailed or two-tailed research questions. Procedure Used to Test for Significance Whenever we perform a significance test, it involves comparing a test value that we have calculated to some critical value for the statistic. It doesn't matter what type of statistic we are calculating (e.g., a t-statistic, a chi-square statistic, an F-statistic, etc.), the procedure to test for significance is the same. Decide on the critical alpha level you will use (i.e., the error rate you are willing to accept). Conduct the research. Calculate the statistic. Compare the statistic to a critical value obtained from a table. If your statistic is higher than the critical value from the table: Your finding is significant. You reject the null hypothesis. The probability is small that the difference or relationship happened by chance, and p is less than the critical alpha level (p < alpha ). If your statistic is lower than the critical value from the table: Your finding is not significant. You fail to reject the null hypothesis. The probability is high that the difference or relationship happened by chance, and p is greater than the critical alpha level (p > alpha ). Modern computer software can calculate exact probabilities for most test statistics. If you have an exact probability from computer software, simply compare it to your critical alpha level. If the exact probability is less than the critical alpha level, your finding is significant, and if the exact probability is greater than your critical alpha level, your finding is not significant. Using a table is not necessary when you have the exact probability for a statistic. Reference  https://www.statpac.com/surveys/statistical-significance.htm

ANOVA Test: Definition, Types, Examples

Statistics Definitions > ANOVA

Contents:

1. The ANOVA Test
2. One Way ANOVA
3. Two Way ANOVA
4. What is MANOVA?
5. What is Factorial ANOVA?
6. How to run an ANOVA
7. ANOVA vs. T Test
8. Repeated Measures ANOVA
9. Sphericity

The ANOVA Test

An ANOVA test is a way to find out if survey or experiment results are significant. In other words, they help you to figure out if you need to reject the null hypothesis or accept the alternate hypothesis. Basically, you’re testing groups to see if there’s a difference between them. Examples of when you might want to test different groups:

• A group of psychiatric patients are trying three different therapies: counseling, medication and biofeedback. You want to see if one therapy is better than the others.
• A manufacturer has two different processes to make light bulbs. They want to know if one process is better than the other.
• Students from different colleges take the same exam. You want to see if one college outperforms the other.

What Does “One-Way” or “Two-Way Mean?

One-way or two-way refers to the number of independent variables (IVs) in your Analysis of Variance test. One-way has one independent variable (with 2 levels) and two-way has two independent variables (can have multiple levels). For example, a one-way Analysis of Variance could have one IV (brand of cereal) and a two-way Analysis of Variance has two IVs (brand of cereal, calories).

What are “Groups” or “Levels”?

Groups or levels are different groups in the same independent variable. In the above example, your levels for “brand of cereal” might be Lucky Charms, Raisin Bran, Cornflakes — a total of three levels. Your levels for “Calories” might be: sweetened, unsweetened — a total of two levels.

Let’s say you are studying if Alcoholics Anonymous and individual counseling combined is the most effective treatment for lowering alcohol consumption. You might split the study participants into three groups or levels: medication only, medication and counseling, and counseling only. Your dependent variable would be the number of alcoholic beverages consumed per day.

If your groups or levels have a hierarchical structure (each level has unique subgroups), then use a nested ANOVA for the analysis.

What Does “Replication” Mean?

It’s whether you are replicating your test(s) with multiple groups. With a two way ANOVA with replication , you have two groups and individuals within that group are doing more than one thing (i.e. two groups of students from two colleges taking two tests). If you only have one group taking two tests, you would use without replication.

Types of Tests.

There are two main types: one-way and two-way. Two-way tests can be with or without replication.

• One-way ANOVA between groups: used when you want to test two groups to see if there’s a difference between them.
• Two way ANOVA without replication: used when you have one group and you’re double-testing that same group. For example, you’re testing one set of individuals before and after they take a medication to see if it works or not.
• Two way ANOVA with replication: Two groups, and the members of those groups are doing more than one thing. For example, two groups of patients from different hospitals trying two different therapies.

One Way ANOVA

A one way ANOVA is used to compare two means from two independent (unrelated) groups using the F-distribution. The null hypothesis for the test is that the two means are equal. Therefore, a significant result means that the two means are unequal.

When to use a one way ANOVA

Situation 1: You have a group of individuals randomly split into smaller groups and completing different tasks. For example, you might be studying the effects of tea on weight loss and form three groups: green tea, black tea, and no tea.
Situation 2: Similar to situation 1, but in this case the individuals are split into groups based on an attribute they possess. For example, you might be studying leg strength of people according to weight. You could split participants into weight categories (obese, overweight and normal) and measure their leg strength on a weight machine.

Limitations of the One Way ANOVA

A one way ANOVA will tell you that at least two groups were different from each other. But it won’t tell you what groups were different. If your test returns a significant f-statistic, you may need to run an ad hoc test (like the Least Significant Difference test) to tell you exactly which groups had a difference in means.
Back to Top

Two Way ANOVA

A Two Way ANOVA is an extension of the One Way ANOVA. With a One Way, you have one independent variable affecting a dependent variable. With a Two Way ANOVA, there are two independents. Use a two way ANOVA when you have one measurement variable (i.e. a quantitative variable) and two nominal variables. In other words, if your experiment has a quantitative outcome and you have two categorical explanatory variables, a two way ANOVA is appropriate.

For example, you might want to find out if there is an interaction between income and gender for anxiety level at job interviews. The anxiety level is the outcome, or the variable that can be measured. Gender and Income are the two categorical variables. These categorical variables are also the independent variables, which are called factors in a Two Way ANOVA.

The factors can be split into levels. In the above example, income level could be split into three levels: low, middle and high income. Gender could be split into three levels: male, female, and transgender. Treatment groups and all possible combinations of the factors. In this example there would be 3 x 3 = 9 treatment groups.

Main Effect and Interaction Effect

The results from a Two Way ANOVA will calculate a main effect and an interaction effect. The main effect is similar to a One Way ANOVA: each factor’s effect is considered separately. With the interaction effect, all factors are considered at the same time. Interaction effects between factors are easier to test if there is more than one observation in each cell. For the above example, multiple stress scores could be entered into cells. If you do enter multiple observations into cells, the number in each cell must be equal.

Two null hypotheses are tested if you are placing one observation in each cell. For this example, those hypotheses would be:
H01: All the income groups have equal mean stress.
H02: All the gender groups have equal mean stress.

For multiple observations in cells, you would also be testing a third hypothesis:
H03: The factors are independent or the interaction effect does not exist.

An F-statistic is computed for each hypothesis you are testing.

Assumptions for Two Way ANOVA

• The population must be close to a normal distribution.
• Samples must be independent.
• Population variances must be equal.
• Groups must have equal sample sizes.

What is MANOVA?

Analysis of variance (ANOVA) tests for differences between means. MANOVA is just an ANOVA with several dependent variables. It’s similar to many other tests and experiments in that it’s purpose is to find out if the response variable (i.e. your dependent variable) is changed by manipulating the independent variable. The test helps to answer many research questions, including:

• Do changes to the independent variables have statistically significant effects on dependent variables?
• What are the interactions among dependent variables?
• What are the interactions among independent variables?

MANOVA Example

Suppose you wanted to find out if a difference in textbooks affected students’ scores in math and science. Improvements in math and science means that there are two dependent variables, so a MANOVA is appropriate.

An ANOVA will give you a single (“univariate”) f-value while a MANOVA will give you a multivariate F value. MANOVA tests the multiple dependent variables by creating new, artificial, dependent variables that maximize group differences. These new dependent variables are linear combinations of the measured dependent variables.

Interpreting the MANOVA results

If the multivariate F value indicates the test is statistically significant, this means that something is significant. In the above example, you would not know if math scores have improved, science scores have improved (or both). Once you have a significant result, you would then have to look at each individual component (the univariate F tests) to see which dependent variable(s) contributed to the statistically significant result.

Advantages and Disadvantages of MANOVA vs. ANOVA

Advantages

1. MANOVA enables you to test multiple dependent variables.
2. MANOVA can protect against Type I errors.

Disadvantages

1. MANOVA is many times more complicated than ANOVA, making it a challenge to see which independent variables are affecting dependent variables.
2. One degree of freedom is lost with the addition of each new variable.
3. The dependent variables should be uncorrelated as much as possible. If they are correlated, the loss in degrees of freedom means that there isn’t much advantages in including more than one dependent variable on the test.

What is Factorial ANOVA?

A factorial ANOVA is an Analysis of Variance test with more than one independent variable, or “factor“. It can also refer to more than one Level of Independent Variable. For example, an experiment with a treatment group and a control group has one factor (the treatment) but two levels (the treatment and the control). The terms “two-way” and “three-way” refer to the number of factors or the number of levels in your test. Four-way ANOVA and above are rarely used because the results of the test are complex and difficult to interpret.

• A two-way ANOVA has two factors (independent variables) and one dependent variable. For example, time spent studying and prior knowledge are factors that affect how well you do on a test.
• A three-way ANOVA has three factors (independent variables) and one dependent variable. For example, time spent studying, prior knowledge, and hours of sleep are factors that affect how well you do on a test

Factorial ANOVA is an efficient way of conducting a test. Instead of performing a series of experiments where you test one independent variable against one dependent variable, you can test all independent variables at the same time.

Variability

In a one-way ANOVA, variability is due to the differences between groups and the differences within groups. In factorial ANOVA, each level and factor are paired up with each other (“crossed”). This helps you to see what interactions are going on between the levels and factors. If there is an interaction then the differences in one factor depend on the differences in another.

Let’s say you were running a two-way ANOVA to test male/female performance on a final exam. The subjects had either had 4, 6, or 8 hours of sleep.

• IV1: SEX (Male/Female)
• IV2: SLEEP (4/6/8)
• DV: Final Exam Score

A two-way factorial ANOVA would help you answer the following questions:

1. Is sex a main effect? In other words, do men and women differ significantly on their exam performance?
2. Is sleep a main effect? In other words, do people who have had 4,6, or 8 hours of sleep differ significantly in their performance?
3. Is there a significant interaction between factors? In other words, how do hours of sleep and sex interact with regards to exam performance?
4. Can any differences in sex and exam performance be found in the different levels of sleep?

Assumptions of Factorial ANOVA

• Normality: the dependent variable is normally distributed.
• Independence: Observations and groups are independent from each other.
• Equality of Variance: the population variances are equal across factors/levels.

How to run an ANOVA

These tests are very time-consuming by hand. In nearly every case you’ll want to use software. For example, several options are available in Excel:

• Two way ANOVA in Excel with replication and without replication.
• One way ANOVA in Excel 2013.

ANOVA tests in statistics packages are run on parametric data. If you have rank or ordered data, you’ll want to run a non-parametric ANOVA (usually found under a different heading in the software, like “nonparametric tests“).

Steps

It is unlikely you’ll want to do this test by hand, but if you must, these are the steps you’ll want to take:

1. Find the mean for each of the groups.
2. Find the overall mean (the mean of the groups combined).
3. Find the Within Group Variation; the total deviation of each member’s score from the Group Mean.
4. Find the Between Group Variation: the deviation of each Group Mean from the Overall Mean.
5. Find the F statistic: the ratio of Between Group Variation to Within Group Variation.

Reference  http://www.statisticshowto.com/anova/

Type I and type II errors

In statistical hypothesis testing, a type I error is the incorrect rejection of a true null hypothesis (also known as a "false positive" finding), while a type II error is incorrectly retaining a false null hypothesis (also known as a "false negative" finding).^[1] More simply stated, a type I error is to falsely infer the existence of something that is not there, while a type II error is to falsely infer the absence of something that is.

In statistical test theory, the notion of statistical error is an integral part of hypothesis testing. The test requires an unambiguous statement of a null hypothesis, which usually corresponds to a default "state of nature", for example "this person is healthy", "this accused is not guilty" or "this product is not broken". An alternative hypothesis is the negation of null hypothesis, for example, "this person is not healthy", "this accused is guilty" or "this product is broken". The result of the test may be negative, relative to the null hypothesis (not healthy, guilty, broken) or positive (healthy, not guilty, not broken). If the result of the test corresponds with reality, then a correct decision has been made. However, if the result of the test does not correspond with reality, then an error has occurred. Due to the statistical nature of a test, the result is never, except in very rare cases, free of error. Two types of error are distinguished: type I error and type II error.

Type I error

A type I error occurs when the null hypothesis (H₀) is true, but is rejected. It is asserting something that is absent, a false hit. A type I error may be likened to a so-called false positive (a result that indicates that a given condition is present when it actually is not present).

The type I error rate or significance level is the probability of rejecting the null hypothesis given that it is true.^[5][6] It is denoted by the Greek letter α (alpha) and is also called the alpha level. Often, the significance level is set to 0.05 (5%), implying that it is acceptable to have a 5% probability of incorrectly rejecting the null hypothesis.^[5]

Type II error

A type II error occurs when the null hypothesis is false, but erroneously fails to be rejected. It is failing to assert what is present, a miss. A type II error may be compared with a so-called false negative (where an actual 'hit' was disregarded by the test and seen as a 'miss') in a test checking for a single condition with a definitive result of true or false. A Type II error is committed when we fail to believe a true alternative hypothesis.^[7] In terms of folk tales, an investigator may fail to see the wolf ("failing to raise an alarm"). Again, H₀: no wolf.

The rate of the type II error is denoted by the Greek letter β (beta) and related to the power of a test (which equals 1−β).

Table of error types

Tabularised relations between truth/falseness of the null hypothesis and outcomes of the test:^[2]

Reference  https://en.wikipedia.org/wiki/Type_I_and_type_II_errors 

Table of error types		Null hypothesis (H0) is
		True	False
Decision About Null Hypothesis (H0)	Reject	Type I error (False Positive)	Correct inference (True Positive)
	Accept (not rejected)	Correct inference (True Negative)	Type II error (False Negative)

What Is the Difference Between Independent and Dependent Variables?

The two main variables in an experiment are the independent and dependent variable.

An independent variable is the variable that is changed or controlled in a scientific experiment to test the effects on the dependent variable.

A dependent variable is the variable being tested and measured in a scientific experiment.

The dependent variable is 'dependent' on the independent variable. As the experimenter changes the independent variable, the effect on the dependent variableis observed and recorded.

Reference

https://www.thoughtco.com/i-ndpendent-and-dependent-variables-differences-606115

Analysis of variance/Types

Reference https://en.wikiversity.org/wiki/Analysis_of_variance/Types

ANOVA models	Definitions
t-tests	Comparison of means between two groups; if independent groups, then independent samples t-test. If not independent, then paired samples t-test. If comparing one group against a fixed value, then a one-sample t-test.
One-way ANOVA	Comparison of means of three or more independent groups.
One-way repeated measures ANOVA	Comparison of means of three or more within-subject variables.
Factorial ANOVA	Comparison of cell means for two or more between-subject IVs.
Mixed ANOVA (SPANOVA)	Comparison of cells means for one or more between-subjects IV and one or more within-subjects IV.
ANCOVA	Any ANOVA model with a covariate.
MANOVA	Any ANOVA model with multiple DVs. Provides omnibus F and separate Fs.