Reference: Ragin, C. C. (2006). Set relations in social research: Evaluating their consistency and coverage. Political Analysis, 14(3), 291-310. https://doi.org/10.1093/pan/mpj019
Summary: “If a firm has this condition, it succeeds” and “to succeed, a firm must have this condition” are different claims. The paper treats this difference as a matter of set relations (sufficient and necessary conditions), and shows how to evaluate such a relation: consistency tells you how consistently the cases follow it, and coverage tells you how much of the outcome it covers. I recommend it to anyone who wants to ask what the phrase “success factor” actually means.
Overview
Suppose that many successful firms turn out to be customer-oriented. Can we then say that being thoroughly customer-oriented leads to success? Ragin deals with the logical problem behind this question.
Charles C. Ragin is a sociologist known as the originator of QCA. The starting point of this paper is a question: do the relations stated in a theory match the relations we actually check in an analysis?
Take the claim that “democracies do not fight wars with each other.” This claim states a set relation: the pairs of democracies form a subset of the pairs of countries that do not go to war. On the other hand, there are many pairs of countries that avoid war without both being democracies, so the correlation between “being a pair of democracies” and “not going to war” may not be strong. That does not, however, refute the original claim.
On this basis, Ragin points out that theories stated as set relations tend to be translated, in analysis, into hypotheses about correlation. He then presents simple descriptive measures for evaluating set relations in the form in which they were stated.
Why I chose this paper
In MBA research, we sometimes want to analyze how a certain event comes about, for example the factors behind a company’s success. However, what successful companies have in common cannot always be treated as a law of success.
In business cases, we come across explanations such as “successful firms are thoroughly customer-oriented” or “high-performing stores offer good service.” To judge what can be concluded from them, we need to distinguish the following three questions.
- Do firms that have the condition succeed?
- Is the condition indispensable for success?
- How much of the success we observe can this condition explain?
The value of reading this paper is not limited to learning QCA’s measures. It also helps you clarify what a “success factor” means and look for evidence that fits the claim you are making. Whether or not you use QCA, I recommend it as a paper that conveys a perspective every MBA student should have.
The structure of the paper
The paper discusses two measures for evaluating set relations (consistency and coverage), including how to use them. In this memo, I narrow the discussion to four points.
1. Correlation and set relations answer different questions
Correlation looks at how two variables move together. A set relation looks at whether the set of cases that have a condition is contained in the set of cases that achieve the outcome (or the reverse). Let the condition be \(X\) and the outcome be \(Y\). Sufficient and necessary conditions can then be distinguished as follows.
| Relation | Question | As sets |
|---|---|---|
| \(X\) is sufficient for \(Y\) | If X is present, is Y present? | \(X \subseteq Y\) |
| \(X\) is necessary for \(Y\) | If Y is present, is X present? | \(Y \subseteq X\) |
As an example, consider the claim that “stores that combine a good location with excellent service are high performers.” This is a claim of sufficiency: the set of stores with this combination is contained in the set of high-performing stores. The claim says only whether the stores with the combination are high performers. It does not say the reverse, that every high-performing store has this combination. For instance, some stores may have an ordinary location yet perform well thanks to a distinctive product mix and a strong brand. Such stores simply lie outside the claim; they do not contradict it. (The stores and numbers in this article are a made-up example for illustration. Only the figure 17/20 = 0.85 is chosen to match an example in the paper.)
A case that contradicts the claim is “a store with a good location and excellent service that is not a high performer.” To check a claim of sufficiency, we look at how many such stores there are.
- Suppose 20 stores have the combination, and 17 of them are high performers while 3 are not. These 3 stores are the cases that contradict the claim.
- Suppose there are 50 high-performing stores in total, and 17 of them have the combination. The remaining 33 stores achieve high performance without this combination. They do not contradict the claim.
The first ratio (17/20) leads to the idea of “consistency” explained next. The second (17/50) leads to “coverage” after that.
2. Consistency shows how consistently the cases fit the claim
Consistency shows how well the observed cases follow the set relation we assume. For sufficiency with crisp sets (sets with only two values, in or out), it is the share of cases with the condition that also show the outcome. For example, if 17 of the 20 stores with a certain combination of conditions are high performers, consistency is 17/20 = 0.85.
With fuzzy sets (sets in which membership is expressed as a degree from 0 to 1), let \(X_i\) be the membership of case \(i\) in the condition and \(Y_i\) its membership in the outcome. The consistency of sufficiency is then expressed as follows.
\[\text{Consistency}(X \to Y) = \frac{\sum_i \min(X_i, Y_i)}{\sum_i X_i}\]
If \(X_i \le Y_i\) holds for every case, consistency is 1.
This formula does not simply count consistent and inconsistent cases. A case with a higher membership in the condition carries more weight. In addition, a small deviation, where membership in the condition exceeds membership in the outcome only slightly, is penalized only a little, while a large deviation is penalized heavily. The paper explains that this is the difference from a method that sorts cases into “consistent” and “inconsistent” and counts the proportion (pp.295-297).
Consistency is also affected by how memberships are distributed. If memberships in the condition are low across the board and memberships in the outcome are high across the board, a relation may merely look like sufficiency while actually being produced by the skewed distribution. The paper therefore says that calibration, the assignment of membership scores, should be done carefully, and that the procedure should be made public so that others can reproduce it (pp.298-299).
3. Coverage shows how much the relation covers
The coverage of a sufficient condition shows how much of the set of cases with the outcome is covered by that condition (or combination of conditions). If the 17 stores above are part of 50 high-performing stores in total, coverage is 17/50 = 0.34. The combination is consistently linked to high performance, but it does not broadly cover high-performing stores as a whole.
For reference, with fuzzy sets the formula is as follows.
\[\text{Coverage}(X \to Y) = \frac{\sum_i \min(X_i, Y_i)}{\sum_i Y_i}\]
The paper calls for using the two measures in order: first check consistency, and only then look at coverage. If consistency is not sufficient, a large coverage cannot make the condition empirically important as a sufficient condition. Consistency and coverage can also pull in opposite directions. A combination narrowed down by stacking many conditions tends to have high consistency but very small coverage. The paper also notes that coverage measures empirical importance, not theoretical importance.
4. Multiple paths have overlapping and unique parts
There is not always only one path to the same outcome. When different combinations of conditions lead to the same outcome, this is called equifinality.
When several paths cover the same successful cases, simply adding their coverages counts those cases more than once. The paper therefore separates the coverage of each path into raw coverage, which includes the overlap, and unique coverage, which shows the part not covered by any other path. The paper says this is somewhat similar to the way regression analysis partitions explained variance among variables. It adds, however, that this is not about removing the correlation among conditions; it is about the fact that a single case can fit more than one path.
The paper’s contributions
First, the paper presents simple descriptive measures for evaluating set relations. The paper itself says that consistency and coverage are neither complicated nor entirely new. What is new, it says, is that they are clearly tied to set-theoretic thinking, which shows how this differs from ordinary quantitative analysis built on correlation.
Second, the paper argues that for necessary conditions, too, consistency alone is not enough. A condition that is present in most cases regardless of the outcome can have high consistency as a necessary condition and still contribute little to understanding the outcome. The coverage of a necessary condition is a clue for spotting such trivial conditions. This point builds on Goertz’s discussion.
Third, the paper separates the coverage of multiple paths into overlapping and unique parts. This makes it possible to evaluate the diversity of patterns of success and the weight of each path.
From here on, these are my own views, based on the paper.
What I found noteworthy
What I consider especially important is the order: check consistency first, then interpret coverage. When the consistency of a sufficient condition is weak, we cannot call a pattern a strong pattern of success merely because it covers the outcome broadly.
Another point is the stance emphasized in the conclusion: the measures themselves should not become the goal of research. Ragin treats consistency and coverage as aids to judgment, and gives priority to whether the results help us understand individual cases and the processes and mechanisms at work in them. When studying business cases in an MBA program, I think this stance of moving back and forth between the numbers and the substance of the cases is important.
Thinking about business applications: analyzing success factors with care
For example, when comparing high-performing stores, before settling on a single explanation such as “low prices matter most,” we can examine different combinations, such as “a good location and excellent service” and “a strong brand and a distinctive product mix” (a made-up example). For each combination, we then check how consistently it is linked to high performance, how many of the high-performing stores it covers, and which stores are the exceptions.
However, high consistency does not guarantee a causal effect, in the sense that adopting the combination will raise performance. Large coverage does not directly translate into investment priorities either. To decide on a measure, we also need to consider cost, feasibility, the causal mechanism, and more.
Connection to QCA
When reading the results of a QCA, we check not only the expression of the combinations of conditions but also consistency, coverage, and the specific cases behind them. Beyond “was the value high?”, being able to explain “consistency of what?”, “coverage with what as the denominator?”, and “what are the exceptions?” is what shows real understanding of the analysis. I also introduce the ideas behind QCA in the article Why QCA suits decision-making on this blog.
Summary points for review
Points to revisit
- Distinguish sufficient from necessary conditions. “If the condition is present, the outcome is present” and “if the outcome is present, the condition is present” are different claims.
- Distinguish consistency from coverage. For a sufficient condition, the former is how consistent the relation is, and the latter is how much of the outcome it covers. Check consistency first, then look at coverage.
- Do not judge by the consistency value alone. In its discussion of crisp sets, the paper says that below 0.75 it becomes hard even to claim a rough set relation. This does not mean that 0.75 or above is automatically adequate; you still need to check the number and the substance of the cases.
- For necessary conditions, the denominators switch (see the table below).
- The measures are descriptive evaluations. They do not prove causal effects.
In the table below, let \(M = \sum_i \min(X_i, Y_i)\).
| Relation evaluated | Consistency | Coverage |
|---|---|---|
| \(X\) is sufficient for \(Y\) | \(M / \sum_i X_i\) | \(M / \sum_i Y_i\) |
| \(X\) is necessary for \(Y\) | \(M / \sum_i Y_i\) | \(M / \sum_i X_i\) |
The coverage of a necessary condition reads differently from the coverage of a sufficient condition, which asks how much of the outcome is covered. It is a measure for checking that the set of X, the supposed necessary condition, is not too large compared with the set of the outcome Y, so that X does not turn out to be an empirically trivial condition.
Glossary
- QCA (qualitative comparative analysis): A method that treats each case as a combination of conditions and examines its relation to the outcome through set relations and logical comparison.
- Condition: In QCA, the term for the explanatory side (\(X\) in this article).
- Outcome: In QCA, the term for the result to be explained (\(Y\) in this article). In this article I often write simply “outcome” or “result” for readability.
- Crisp set: A set in which membership is expressed as 0 or 1.
- Fuzzy set: A set in which membership is expressed as a degree from 0 to 1. The values are calibrated according to the meaning of the concept.
- Consistency: How consistently the cases follow the assumed set relation.
- Coverage: The empirical importance of a consistent relation. For a sufficient condition, how much of the outcome it covers.
- Equifinality: The fact that different combinations of conditions lead to the same outcome.
- Raw / unique coverage: The total range covered by a path, and the part of it that no other path covers.
- Trivial necessary condition: A condition that is widespread regardless of the outcome; it may be consistent as a necessary condition but does not help us understand the outcome.
What to check in the original
This article is my own summary of the paper’s main points. When you read the original, the following three places make good entry points.
- Introduction (pp.291-292): What goes wrong when a theory stated as a set relation is translated into a correlation hypothesis?
- The explanation of Figures 1 and 2 (pp.294-297): How do set relations show up in a scatterplot? Why are cases that look like “errors” from a correlation viewpoint not counterexamples to the claim?
- Conclusion (pp.309-310): Why is understanding the cases considered more important than the height of the measures?
References
Ragin, C. C. (2006). Set relations in social research: Evaluating their consistency and coverage. Political Analysis, 14(3), 291-310. https://doi.org/10.1093/pan/mpj019
Back to the paper guides: About this shelf