Statistical Terms in Quantitative Research

Statistically significant means a result is unlikely due to chance.
The p-value is the probability of obtaining the difference we saw from a sample (or a larger one) if there really isn’t a difference for all users.
A conventional (and arbitrary) threshold for declaring statistical significance is a p-value of less than 0.05.
Statistical significance doesn’t mean practical significance. Only by considering context, we can determine whether a difference is practically significant; that is, whether it requires action.
The confidence interval around the difference also indicates statistical significance if the interval does not cross zero. It also provides likely boundaries for any improvement to aide in determining if a difference is noteworthy.
With large sample sizes, you’re virtually certain to see statistically significant results, in such situations it’s important to interpret the size of the difference.
Small sample sizes often do not yield statistical significance. When they do, the differences tend to be practically significant; that is meaningful enough to warrant action.