Statistics for Data Analysts: The 10 Concepts That Come Up at Work

Statistics for Data Analysts: The 10 Concepts That Come Up at Work

A stakeholder almost never asks you to calculate a confidence interval. They ask something closer to: revenue dropped, why? Did conversion actually improve, or did it just look that way this week? Is this customer segment genuinely different, or is that a fluke of who happened to be in the sample? Did the new onboarding flow cause retention to go up, or did it just coincide with a good week? Can we trust this number enough to change what we do next?

Every one of those questions has the same shape. Someone observed a number, and now needs to know whether it means anything. That gap, between "here is a number" and "here is what I'm confident enough to recommend," is what statistics actually does for a data analyst. It is not a separate technical skill sitting next to SQL and dashboards. It is the reasoning layer that sits between the two: the discipline of knowing how much uncertainty is hiding inside a number before you let it drive a decision.

This guide is built around a single idea. Statistics for a data analyst is not primarily about calculating numbers. It is about understanding uncertainty well enough to decide what a number actually means, and whether anyone should act on it. Everything below builds toward that.

The shape of the reasoning, before the concepts

Every one of the stakeholder questions above, underneath the surface, moves through the same five stages.

Measure. What actually happened, described honestly.

Estimate. How much can we trust that number, given that it came from a sample rather than the whole truth.

Test. Is the difference we're seeing likely to be real, or could it plausibly be random noise.

Explain. If it's real, what's actually driving it, and can we rule out the obvious alternative explanations.

Decide. Even if it's real and we understand why, is it big enough to justify acting on.

Keep this sequence in mind, because the rest of this guide is organised around it rather than around a list of independent statistics topics. Each stage exists because of a different source of uncertainty in business data: sampling introduces uncertainty about whether your data represents the wider population, natural variability introduces uncertainty about whether one result would repeat, random chance introduces uncertainty about whether a difference is a pattern or noise, confounding introduces uncertainty about what's actually causing what, and experiment design introduces uncertainty about whether the whole setup was even measuring the right thing. Different concepts exist to manage different sources of uncertainty. None of them removes it entirely.

The 10 concepts, at a glance

The 10 concepts are the statistical building blocks. The five-stage framework above explains how those concepts work together when an analyst investigates a real business question. Everything from here on walks through both at once, using one running example.

The question we'll follow

Rather than explaining each concept with a fresh, unrelated example, this guide follows one running business problem the way an analyst actually would.

A product team redesigns the onboarding flow. Two weeks later, the numbers come in:

Old onboarding: 22% seven day retention

New onboarding: 24% seven day retention

The product manager asks the obvious question: did the new onboarding actually improve retention, and should we ship it?

That two point gap looks encouraging. It's also, on its own, close to meaningless, and the rest of this guide exists to explain why, and what to check before you can honestly answer the PM's question.

First: what actually happened?

Before anything else, describe the result accurately. This is where the first quiet mistakes happen.

1. Descriptive Statistics

This is where mean, median and standard deviation live. The mean is sensitive to outliers. A handful of very high or very low values can pull it away from what a typical case looks like. The median, the middle value once sorted, is usually the more honest summary for skewed business data like order value, session length or time between purchases. Standard deviation describes how spread out the values are; a small one means the mean is a fair representation, a large one means it isn't.

The mistake: "Average order value is ₹4,800, so the typical customer spends ₹4,800." A small number of very large orders can pull that mean well above what most customers actually spend, and the median often tells a noticeably different, more representative story.

2. Distributions & Variability

Distribution shape matters here too, not as a formal test you run before you're "allowed" to analyse anything, but as a habit. Look at the skew, notice where the outliers sit, and let that guide whether the mean, the median, or a different summary entirely is the honest one to report. Revenue per customer, ticket volume and time between purchases are usually right-skewed in real business data, with a long tail of high values dragging the mean upward.

In our running example: the 22% and 24% retention figures are themselves just means, a proportion of users who returned within seven days. That's the "what happened" stage. It says nothing yet about whether the gap is trustworthy or whether it's large enough to matter.

Next: how much can we trust the number?

This is where the analysis stops being purely descriptive and becomes inferential. You didn't measure every user who will ever go through onboarding. You measured a sample, and you're using it to infer something about users in general.

3. Sampling

Two things matter enormously here, and they're often confused. A random sample is far more valuable than a large, non-random one, because a biased sampling method distorts the result regardless of how many people are in it. And even a perfectly random sample carries variability: if you ran the same experiment again with a different random set of users, you would not get exactly 22% and 24% again. You'd get numbers close to those, but not identical.

The mistake: "We surveyed 10,000 users, so the result must be reliable." Ten thousand biased responses are still biased. A larger sample can reduce sampling variability; it cannot fix a flawed sampling method.

4. Central Limit Theorem

This is where the Central Limit Theorem quietly does its work. Under common conditions, as sample size increases, the sampling distribution of the sample mean approaches a normal distribution, even when the underlying data isn't normally distributed. In plain terms, that's a large part of the reason the average of a reasonably sized sample from messy, skewed business data still behaves predictably enough to build reliable estimates from. It's a well-established result with real conditions attached, not a rule that rescues every dataset regardless of how it's structured, so it's a reason for confidence, not a licence to skip checking your data.

5. Confidence Intervals

That predictability is what makes a confidence interval possible. A confidence interval is a range of values, generated by a specific statistical procedure, that would contain the true population value at the stated confidence level if you repeated the sampling process many times. The common shorthand, "there's a 95% chance the true value is in this range," isn't quite accurate; a given interval either contains the true value or it doesn't, and the 95% describes how reliable the procedure is across repeated sampling, not the odds on this one result. For everyday reporting, the practical use doesn't change: it gives a far more honest picture of uncertainty than a single point number.

In our running example: with a large enough sample in each group, the 24% figure might carry a confidence interval of roughly 23% to 25%, and the 22% figure a range of roughly 21% to 23%. Those intervals give a clearer picture of the uncertainty around each estimate. To formally determine whether the groups differ, you would still test the difference between them, which is the next stage.

Is the difference real?

This is the stage most people jump to first, and the one most commonly misread.

6. Hypothesis Testing & P-values

Hypothesis testing gives you a structured way to ask a narrow question: how surprising would this observed result be if there were actually no real difference between the two groups? You start with a null hypothesis, typically "there's no real difference," and calculate a p-value: roughly, the probability of seeing a result this extreme, or more extreme, if the null hypothesis were true. A small p-value means the result would be relatively unusual under that assumption.

That's a narrower claim than it usually gets credit for. A small p-value does not tell you the probability that the null hypothesis is true, and it does not tell you the probability that your finding is "real." It only describes how unusual this result would look if nothing were actually going on. The American Statistical Association's own statement on p-values is direct about this: a p-value does not measure the size of an effect or the importance of a result, and by itself does not provide a good measure of evidence for or against a hypothesis.

The mistake: "p is less than 0.05, so the hypothesis is true." A p-value is evidence against the null hypothesis at a conventional threshold. It says nothing about how large the effect is, and it certainly isn't proof of anything.

7. Effect Size: how big is the difference?

A p-value can tell you a result is unlikely to be random. It cannot tell you whether that result is worth acting on, and this is where effect size becomes the concept most articles skip. The same 22% to 24% gap can be described two honest but very different ways: as an absolute change of plus two percentage points, or as a relative change of roughly a 9.1% increase over the baseline. Neither statement is wrong, but reporting only one, especially the relative figure on its own, can make a modest change sound far more dramatic than it is.

This is the bridge between "is it real" and "does it matter." Statistical significance asks whether the evidence is strong enough to take seriously. Effect size asks how large the difference actually is. Practical significance, covered later, asks whether that size is big enough to be worth acting on. All three are separate questions, and answering only one of them is how both false alarms and missed opportunities happen.

Did one thing actually cause the other?

Even a real, well-measured, statistically credible difference doesn't automatically mean the thing you changed is what caused it.

8. Correlation, Causation & Regression

Two variables moving together doesn't mean one causes the other. The classic example is ice cream sales and drowning incidents, which both rise in summer without either one causing the other; a third factor, warmer weather, drives both. In business data, the same trap shows up constantly and less obviously. Users who engage with a new feature might retain better, but that doesn't prove the feature caused the retention. It's equally possible that more engaged users, who were always going to retain better regardless, are simply the ones who try new features first, a case of reverse causation dressed up as a success story.

A handful of specific traps are worth knowing by name, because they recur across almost every analyst investigation: selection bias, where the group you're studying isn't representative of the group you want to conclude something about; confounding, where a third variable drives both sides of an apparent relationship; reverse causality, where the effect you're crediting to your change actually came before it; survivorship bias, where you're only looking at the cases that made it into your dataset because they succeeded, and the failures quietly disappeared from view; and seasonality, where a pattern that looks like cause and effect is really two things that both happen to move with the calendar. In rarer cases, a relationship can even reverse direction once you break the data into subgroups, known as Simpson's paradox, which is a good reminder to check whether an aggregate trend actually holds within each segment before trusting it.

None of these need to be memorised as a checklist. What matters is the habit underneath all of them: before accepting a causal story, ask what else could explain this relationship.

Regression is the tool that lets you investigate that question with some rigour. It models the relationship between a variable you care about and one or more variables that might explain it, and the simplest version fits a line: for every additional rupee spent on marketing, how much does revenue tend to move? What regression is actually estimating is an association, adjusted for whatever other variables you included, not proof of causation on its own. "Marketing spend and revenue are related" is vague and not very useful. "Each additional lakh in spend is associated with roughly 40,000 in incremental revenue, holding season constant" is something a budget decision can actually use, because it's specific about the size of the relationship and about what's been controlled for.

That "holding other things constant" phrase is doing more work than the headline coefficient. A relationship that looks strong in a simple comparison can shrink or disappear entirely once you control for a confounder like seasonality or a concurrent promotion that was quietly driving both sides of the relationship. Regression is built for explanation and for prediction, and those are different jobs: a model can predict well without you understanding why, and a model can help you understand a relationship without necessarily being the best predictor. Regression narrows the causal story and puts a number on it. It does not, by itself, prove that story is true; that generally still requires a controlled experiment.

The mistake: "Marketing spend caused revenue to increase because the regression coefficient is positive." Seasonality, concurrent promotions, broader demand trends, or a variable that was never included in the model can all produce that same positive coefficient without spend being the actual driver.

In our running example: the fact that users saw the new onboarding and retention went up is not, by itself, proof the onboarding caused the improvement. If the redesign launched the same week as a marketing push that brought in a different, more engaged type of user, that alone could explain the gap. This is exactly why the next stage, a properly randomised experiment, matters so much.

Even if it's real, does it matter? (Concept 10: Statistical vs Practical Significance)

A result can be statistically credible, causally sound, and still not worth acting on. This is the stage most tutorials skip, and it's arguably the one that separates an analyst who produces correct arithmetic from one who produces useful recommendations.

A 0.3% lift in conversion from a redesigned checkout button might clear the statistical bar easily in a large enough sample, and still not be worth the engineering effort to build and maintain. A promising fifteen point lift in a small pilot might not be statistically significant yet, meaning you genuinely don't know if it's real, even though it looks exciting in a slide. Practical significance weighs the size of a real effect against the cost of acting on it: the engineering time, the operational complexity, the opportunity cost of not building something else instead. That comparison is a business judgment, not a statistical one, but it depends entirely on the statistical work done earlier to even be possible.

In our running example: if the retention lift holds up as real and roughly two percentage points at the company's actual user volume, the next question is whether the estimated revenue and engagement impact comfortably exceeds the cost of maintaining the new flow. That threshold is a business judgment, not a statistical one, and this guide's job is to hand the business a trustworthy size and a trustworthy causal read, not to invent certainty about the cost side that the data here doesn't provide.

How this comes together in an A/B test (Concept 9: A/B Testing & Experimentation)

A/B testing is where every stage above operates at once, which is why it deserves to be treated as the culmination of this guide rather than one item on a list.

The sequence, in a properly run test, looks like this: a business question becomes a specific metric to track, which becomes a formal hypothesis about what should change. Users are split through randomisation into control and variant groups, with a sample size decided in advance based on the smallest effect worth detecting. The experiment runs for a predetermined period, producing an observed effect, which gets reported as a confidence interval, checked with a hypothesis test, sized with an effect size calculation, and finally weighed for practical significance before it becomes a business decision.

Each stage can fail independently, which is why "the variant won" is never the whole analysis. Was the randomisation actually clean, or did one group end up systematically different for reasons unrelated to the change? Was the sample size decided in advance, or did the test get stopped the moment it looked good, a practice known as peeking that inflates the odds of a false positive? Were multiple metrics checked and only the flattering one reported? Is the confidence interval tight enough to trust, or so wide that "24% versus 22%" could just as easily have been "23% versus 23%" with different random users?

The mistake: "The variant won, so ship it." That sentence skips randomisation, sample size, stopping rules, effect size and practical significance, all in favour of a single number that happened to be higher.

A worked case study: should the new onboarding ship?

Here is the full investigation, the way it would actually unfold in an analyst review.

The question. Product says the new onboarding increased seven day retention from 22% to 24%. Should we ship it?

Measure. Control group: 40,000 users, 22% retention. Variant group: 40,000 users, 24% retention. That's an absolute lift of two percentage points, and a relative lift of roughly 9.1% over the 22% baseline. Both figures are correct. Reporting only the relative number would make a modest change sound far more dramatic than it is.

Estimate. At 40,000 users per group, the sample is large enough that each group's confidence interval is reasonably tight, something like 21.6% to 22.4% for control and 23.6% to 24.4% for variant. The intervals are fairly tight and separated, which is visually encouraging; the formal comparison comes from testing the difference between the groups directly, which is the next step.

Test. A proportions test on this data produces a p-value comfortably below the conventional 0.05 threshold. That means a gap this large would be unusual if there were genuinely no real difference between the two flows, which is a reason to take the result seriously. It still says nothing about how large the effect is or whether it's worth shipping.

Effect size. Two percentage points absolute, roughly 9.1% relative. At this company's user volume, that's not a marginal number, but the size only becomes meaningful once it's translated into an actual business outcome.

Explain. Before crediting the onboarding flow itself, check for confounders: did the redesign launch alongside a marketing campaign, a pricing change, or a seasonal spike that could independently explain higher retention? If the test was properly randomised, with users assigned to control and variant at the same time under the same external conditions, this concern is largely addressed, because both groups would have been equally exposed to any outside factor. That's precisely why the randomisation step matters as much as the statistics that follow it.

Decide. A two point retention lift, sustained over months at this company's user volume, would need to be modelled into an actual revenue and engagement impact before this question can be answered honestly. If that estimated impact comfortably exceeds the cost of maintaining the new flow, the business case for shipping is strong. That comparison, not the p-value on its own, is what should actually drive the recommendation.

The recommendation. Assuming the business model confirms that the estimated impact clears the cost of maintaining the new flow, ship the new onboarding flow. The lift is statistically credible and the randomisation supports a causal read; the remaining question is a business one, not a statistical one, and it deserves its own honest answer rather than an assumed yes. Getting to this point required pulling the underlying cohort data correctly in the first place; the SQL for Data Analysts guide covers the kind of retention query this analysis depends on.

What an analyst actually says

Getting the analysis right is only half the job. The other half is translating it into something a stakeholder can act on without misreading it.

Weak: "The new onboarding performed 9.1% better and p was less than 0.05."

This is technically defensible and communicates almost nothing useful. It leads with the more dramatic-sounding relative number, buries the absolute size, and cites a statistic most stakeholders will either ignore or misinterpret as "proof."

Better: "The new onboarding increased seven day retention by two percentage points. The difference is statistically credible, the test was properly randomised, and at our current user volume the estimated impact is large enough to justify shipping."

This version leads with the absolute number, states what the statistics actually support, and ends on the decision rather than the arithmetic.

A few more translations worth internalising:

Instead of "the p-value was significant," say what that actually licenses you to claim: "the gap is unlikely to be random chance, based on this sample."

Instead of "the correlation is strong," say what you've ruled out: "engaged users retain better, though we haven't yet ruled out that engaged users were always going to retain better regardless."

Instead of "the model shows spend drives revenue," say what was controlled for: "each additional lakh in spend is associated with roughly 40,000 in incremental revenue, after accounting for seasonality."

Instead of "the test isn't significant yet," say what that means for the decision: "we don't yet have enough evidence to tell this apart from no effect, we need a larger sample or more time before recommending either way."

A quick decision tree for the next business question

When a new question lands on your desk, this is roughly the routing logic:

Are you describing what already happened, with no claim about the wider population? That's descriptive statistics: mean, median, distribution shape.

Are you estimating something about a larger population from a sample? That's sampling and confidence intervals.

Are you comparing two or more groups to see if a difference is real? That's hypothesis testing, paired with an effect size.

Are you asking whether one thing caused another? That's causal reasoning, ideally backed by a properly randomised experiment, not just a correlation.

Are you trying to quantify the size of a relationship between variables? That's regression, with the same caution about causation.

Are you deciding whether to act on a result you already trust is real? That's effect size and practical significance, weighed against cost.

Statistics and AI, realistically

AI tools have become genuinely useful for the mechanical side of this work: writing the code for a hypothesis test, explaining what a p-value means in plain language, suggesting which test fits a given dataset shape, checking syntax, and generating a first-draft chart. That's real time saved, and there's no reason to avoid it.

What AI tools should not be trusted to do without a check is the judgment layer this entire guide has been building toward: choosing the correct population to study, spotting a sampling bias baked into how the data was collected, deciding whether a causal claim is actually justified, weighing whether an effect matters commercially, understanding what a business metric actually means inside your company, or deciding whether an experiment is genuinely ready to ship. An assistant has no way to know that your two groups weren't randomised cleanly, or that "retention" is defined slightly differently in this dataset than in the dashboard finance is using.

The useful way to think about it: AI can accelerate statistical mechanics. It cannot outsource statistical judgment. For more on where AI genuinely helps analyst workflows and where it doesn't, the AI and GenAI Tools Changing Analytics Workflows piece covers that split in more depth.

Where to go from here

Statistics is the reasoning layer, but it sits on top of the tools that get you the data in the first place. If you haven't built the underlying querying and aggregation habits yet, that's worth doing alongside this guide rather than after it; a broader Data Analyst roadmap shows where statistics fits relative to SQL, Excel and the rest of the skill set.

If you'd rather build statistical thinking through real experimentation case studies, with practitioners who run these tests for a living, the 20-week Data Analytics program covers statistics and experimentation as a dedicated module rather than a side topic.