Power of a Test: Power analysis is basically done to ascertain the accuracy of statistical test conducted on the data. While conducting any kind of test on sample data an analyst can commit two types of error: Type I error and Type II error. Statistical power mainly deals with Type II errors.
Power of a test can be defined as probability of correctly rejecting the null hypothesis or True Positive Rate.
For example, while develop a regression model to predict defaulters based on historical data we would check accuracy of model before deploying it. Most commonly used method is the confusion matrix.
So, the first quadrant where we correctly tagged the defaulters based on our model quantifies the Power of a test. The Power of a test actually is 1 – probability of making type 2 error.
Another measure which can be an indicator of the power of a test is Sensitivity, which is calculated as number of correct positive predictions divided by the total number of actual positives.
Generally, for a given level of significance; the power of a test increases as the size of the dataset increases.