Fencers frequently switch clubs, either because their current club isn’t meeting their needs, or to train with certain fencers or coaches at another club. But does switching clubs actually lead to better competitive outcomes?
To answer this, I analyzed data from over 600 fencers who switched clubs, using matrix completion—a machine learning technique—to estimate counterfactual outcomes. In other words, I measured how these fencers would have performed if they had stayed at their original clubs.
Surprisingly, I found no evidence that switching clubs, on average, improves or worsens performance compared to similar fencers who didn’t switch. Even among the ~300 fencers who switched to a top 10 club, the performance gains are nominal at best.
The Methodology
Pretend that Joe fences at Gilwich Fencing Club. Having just earned his C rating at the recent NAC, he’s beginning to feel like his club doesn’t have anyone to challenge him. Seeking tougher competition and better coaching, Joe switches to Julfield Fencing Club. After making the switch, Joe continues to improve and eventually earns a B rating.
Meanwhile, Joe’s friend Bob stays at Gilwich and surprises everyone by earning an A rating. Now Joe feels a twinge of jealousy. “Maybe I should have stayed at Gilwich,” he wonders. The problem is, Joe can’t know for sure. Since he can’t see parallel universes, he has no idea what would have happened had he stayed at his old club.
Matrix Completion
Thankfully, data from other fencers, like Bob and those at other clubs, can help us model this parallel universe. We can represent fencers and their performance if they stayed at their original club as a matrix. Each row represents a fencer, and each column represents a competition. The value in each cell is a metric of performance.
| Comp. 1 | Comp. 2 | Comp. 3 | Comp. 4 | Comp. 5 | Comp. 6 | Comp. 7 | |
| Joe | 10 | ????? | 14 | 17 | ????? | ????? | ????? |
| Bob | -7 | 5 | ????? | 12 | 20 | 25 | 26 |
| Sharon | -20 | -10 | ????? | -17 | -9 | -28 | -22 |
| Preston | -20 | -10 | -1 | -25 | -18 | -2 | -3 |
| Sophia | 23 | ????? | 19 | 26 | ????? | 12 | -20 |
| Jamal | 23 | 23 | 12 | 21 | 19 | 15 | 1 |
Let’s say that Joe switches clubs after Competition 4. From that point on, we no longer know how he would have performed had he stayed at his original club, and these unknowns are represented as blue question marks in the matrix. Additionally, there are red question marks scattered throughout the matrix to represent competitions that fencers didn’t attend, regardless of whether they switched clubs or not.
The challenge is to “fill in” these question marks as accurately as possible. By leveraging patterns in the observed data—both from Joe’s past results and from other fencers—we can estimate the missing values. This is called matrix completion [1], and it allows us to estimate what would have happened to Joe had he not switched clubs. By comparing his actual performance at the new club to this estimated performance at his old club, we can quantify the impact of switching clubs on his results.
Data and Question
I used data from over 600 fencers competing in Cadet and Junior events at national fencing tournaments held between 2012 and 2020. To measure performance, I used indicator after pools (touches scored minus touches received in pools). Using V/M instead (victories divided by total matches) delivered similar results. My goal was to answer two questions:
- What is the causal effect of switching clubs on fencers’ performance?
- How does switching to a top 10 club affect fencers’ performance?
I only looked at fencers who remained in the same USA Fencing division when switching clubs. By keeping track of fencers using both their name and division, I was less likely to run into issues with two people having the same name. Additionally, fencers who change club and move divisions at the same time may experience performance changes due to moving that could confound the effect from only changing clubs.
Results
I compared fencers’ actual tournament results to their predicted performance had they stayed at their original clubs using the fect package [2]. The thick vertical gray line in each graph marks when fencers switched clubs. The black line represents the difference between actual and predicted performance. Here’s how to interpret the graphs:
- Before the gray line: This period reflects the difference between fencers’ actual performance and their estimated counterfactual performance had they not switched clubs. No significant effects should appear here, as nobody has switched clubs yet. Therefore, the black line should stay close to the x-axis.
- After the gray line: This is the period where we should expect to see the impact of switching clubs. If there is a significant effect, the black line will diverge substantially from the x-axis.
The shaded areas around the black lines are 95% confidence intervals, which indicate the amount of uncertainty in the estimates.
Switching Clubs in General
Fencers who switched clubs perform slightly worse on average than they would have if they had stayed. However, these differences were not statistically significantstatistically significant a result in statistical testing that provides enough evidence to reject the null hypothesis, suggesting that an observed effect is likely not due to chance alone after accounting for multiple testing, meaning the results could easily be due to chance. Moreover, a one point difference in indicator is so small that it is essentially unnoticeable.
| Foil | Epee | Saber | |
|---|---|---|---|
| Women’s | -0.5169 | -1.437 | -0.08098 |
| Men’s | 0.4407 | -1.245 | -0.9352 |
Switching to Top 10 Club
Fencers who move to a top 10 club show slight performance improvements on average. However, these effects are small and similarly statistically insignificant after adjusting for multiple testing. A 1-point increase in indicator after pools is unlikely to justify the cost and effort of switching clubs anyway.
| Foil | Epee | Saber | |
|---|---|---|---|
| Women’s | -0.193 | -0.977 | 1.048 |
| Men’s | 1.071 | -0.897 | -0.269 |
Conclusion
Fencers often switch clubs to improve their performance, hoping that better training environments and coaches will lead to measurable gains. However, the data challenges this assumption. On average, fencers who switched clubs performed slightly worse than if they had stayed at their original clubs. Those who moved to a top 10 club experienced only marginal improvements. Importantly, these small effects are not statistically significant, meaning they could simply be due to chance.
That said, a lack of statistical significancestatistically significant a result in statistical testing that provides enough evidence to reject the null hypothesis, suggesting that an observed effect is likely not due to chance alone doesn’t rule out the possibility of an effect. It might simply mean the effect is too small to detect with the available data or methodology, or that the effect is only apparent years after switching. This also doesn’t mean that changing clubs is never helpful. Some fencers might do much better—or worse—after switching, even if the overall average shows no significant change in performance. These results do suggest, however, that there is no overwhelming evidence that fencers should switch clubs if they want to improve their performance in competition.
I’d love to hear your thoughts. Have you experienced a performance boost—or decline—after switching clubs?
Robustness checks
Liu et al. [3] recommend using robustness checks to determine if matrix completion is correctly modeling the counterfactual had the fencers stayed at their original club.
One robustness check that they think is especially useful is placebo testing. This test pretends as though each fencer switched clubs a few tournaments before they actually did. Since the fencers haven’t actually switched clubs yet, the predicted counterfactual should be nearly equal to the fencers’ actual performance in competition. There are two ways to test for this equality. One is by simply testing whether the predicted and actual values are different by a statistically significant amount. The second is an equivalence test which is rejected if the values are within a certain range around 0.
The blue parts in the graphs below indicate the periods with the placebo club change. Notice that the p-values are above 0.05 and the equivalence tests are below 0.05 for nearly all weapons/genders (besides women’s epee). This suggests that the model is doing a good job modeling the counterfactual because the predicted and actual values are very close prior to the fencers switching clubs.
References
[1] Athey S, Bayati M, Doudchenko N, Imbens G, Khosravi K. Matrix completion methods for causal panel data models. 2018 Oct; doi:10.3386/w25132
[2] Liu L, Liu Z, Wang Y, Xu Y. FECT: Fixed Effects Counterfactuals. CRAN: Contributed Packages. 2022 Oct 14; doi:10.32614/cran.package.fect
[3] Liu L, Wang Y, Xu Y. A practical guide to counterfactual estimators for Causal inference with time‐series cross‐sectional data. American Journal of Political Science. 2022 Aug 2;68(1):160–76. doi:10.1111/ajps.12723
