No Evidence for a Sex Difference in Immune Response to COVID-19: An Explainer of the GenderSci Lab’s Nature Matters Arising Article
By Annika Gompers
On Wednesday, members of the GenderSci Lab will publish a Matters Arising article in Nature, responding to the paper by Takahashi et al., “Sex differences in immune responses that underlie COVID-19 disease outcomes,” which appeared in the same journal in August of 2020. A “Matters Arising” piece provides a venue for empirical and conceptual critique of a research article in Nature. It is peer-reviewed and accompanied by a reply from the authors of the target article. Matters Arising is the only venue that Nature provides for this kind of critical commentary on research articles.
In our Matters Arising piece, we aren’t just arguing that there are some small issues with the original paper. We argue that its core claim is completely unsubstantiated: that the Takahashi et al. study does not, in fact, demonstrate that biological sex explains differences in COVID-19 patient outcomes.
This blog post serves as an explainer and companion to our Nature Matters Arising article, delving into the problems that we identified in Takahashi et al.’s paper and explaining what we did to elucidate these points. In summary, we found that the study was limited by the following issues:
Myriad analyses conducted, but very few sex differences found
Sex differences swim in and out of view
Limitations of the dataset itself in relation to the claims made
Before describing each of these issues, a brief overview of the study in question is as follows: The study sample included 98 patients with COVID-19 (called Cohort B). A subset of 39 of these patients met additional criteria of not being in the ICU and not receiving immunomodulatory drugs before the first sample collection date (called Cohort A). 64 health care workers (HCW) were included as controls, but were not matched with the patients for age, BMI, or other underlying risk factors. The authors collected biospecimens (blood, nasopharyngeal swabs, saliva, urine, and stool), which they used to analyze the immune response to COVID-19 (though almost all of the analyses presented were done on blood samples).
1. So many analyses, so few differences
Takahashi et al. do not report how many analyses they ran, but it is clear that the number of different comparisons is in the hundreds. At minimum, they looked at measurements of viral RNA, two SARS-CoV-2 antibodies, 71 cytokines and chemokines, 12 markers of peripheral blood mononuclear cell (PBMC) composition, 19 markers of T cell subsets, and 13 intracellular cytokines in T cells. These 100+ markers are compared in at least 21 different strategies (Table 1).
Table 1: A catalogue of the analytical strategies in Takahashi et al.’s paper
Determining the number of comparisons made by Takahashi et al. was concerningly difficult. Several GenderSci Lab members combed through their paper in order to generate the counts of markers and analyses included in our Matters Arising article and this blog post. What is clear is that Takahashi et al. overwhelmingly found similarities between female and male patients, with a few points of difference. Their comparisons yielded over 500 findings, many of which are presented as raw data, unadjusted for possible covariates. We narrowed down this data to 202 comparisons by looking at only adjusted analyses, as claims of sex difference should not be made based on raw comparisons between patients and controls unmatched for age, BMI, and other important factors. This count also focuses on cytokines, chemokines, and markers of immune cell phenotype, which are the measures the authors center in their discussion section. We excluded within-sex comparisons, i.e. female patients versus female healthcare workers, as well as comparing female versus male controls; this is because these analyses should not be used to make claims of sex differences in COVID-19 immune response (though we point out in the below section that the authors do so). Of these 202 comparisons, only 13 (6%) proved to be statistically significant (p<0.05) (Figure 1). Among these, only one finding showed significance at the p<0.01 level. Yet the authors conclude that the “immune landscape in COVID-19 patients is considerably different between the sexes,” and even go on to recommend different vaccines and therapies for men and women. While exploratory studies such as this one do have merit, the authors must be transparent about just how many variables and comparisons they included, and how few of those showed any significant sex differences. Their data actually show a largely similar immune response to COVID-19 between the sexes, and do not justify making sweeping claims about the need for sex-specific treatment strategies.
Figure 1: Statistical significance of Takahashi et al.’s adjusted analyses. We excluded comparisons between female controls and male controls, female patients and controls, and male patients and controls. This figure represents 202 comparisons.
2. Pinning down slippery sex differences
Takahashi et al. speak in broad strokes about sex differences in the immune response to COVID-19 that they claim to find. We tried to pin down their findings of sex differences -- what they are based on and what they show. This was not a straightforward task, as there is considerable mismatch between the claims made in the paper and the findings documented in the data tables. While adjusted analyses are reported in data tables, unadjusted analyses are only reported in figures, in which only significant p-values are displayed. Some of the analyses overlap, such as the two versions of adjusted longitudinal analysis of female and male patients (Extended Data Table 4 and the first column of Extended Data Table 5 -- the authors appear to only use the results of the former in the text of their paper). Additionally, comparisons of male and female healthcare workers are included (Extended Data Tables 3 and 5) but never addressed in the body of the paper.
Focusing on the claims made in the discussion section of the paper, we tracked down (in the body of the paper and all of the tables and figures) which specific markers and analyses were being referenced. We compiled this information into a table published with our Matters Arising to clearly show which analyses of a given marker show sex differences and which do not. Again, this table only addresses those markers which the authors themselves highlight as findings of sex difference -- the vast majority of the myriad other markers measured did not show any sex differences (as discussed above).
Our efforts to nail down this study’s findings illustrate that the reported sex differences are quite slippery, swimming in and out of view as the reader wades through this paper. Most of Takahashi et al.’s reported significant effects are found only in unadjusted analyses -- that is, after adjusting for age, BMI, and other relevant covariates, many of the significant effects disappear. (This is unsurprising, as the patients and controls were not matched, allowing for a patient sample that was much older and had higher BMI than the controls.) This indicates that many of the reported sex differences in immune response to COVID-19 are better explained by factors other than biological sex.
Additionally, some of the reported sex differences are better described as within-sex differences (rather than between-sex differences). By within-sex differences, we mean factors that show significant differences between members of one sex -- for example, female patients as compared to female healthcare workers. This is in distinction to between-sex differences, which are factors that show significant differences when comparing males and females. Takahashi et al. utilize two measures of between-sex differences: a direct male-to-female comparison and a difference-in-differences comparison, in which the female within-sex difference is compared to the male within-sex difference. We contend that the existence of within-sex differences without any significant between-sex differences cannot be interpreted as demonstrating sex difference in immune response to COVID-19 (Figure 2). However, such questionable findings do appear in the paper.
Figure 2: Diagrammatic representation of a within-sex difference that cannot be taken as a properly conceptualized “sex difference” (top), and a between-sex difference that is a properly conceptualized “sex difference (bottom).
For example, Takahashi et al. state in their discussion section that “higher levels of innate immune cytokines were associated with worsening of COVID-19 disease in female patients.” In the body of the paper, they specify that in unadjusted analyses, CCL5, TSNF-10, and IL-15 were higher in deteriorated females than in stable females, while this difference was not found in male patients (these are within-sex differences). In analyses adjusted for age and days from symptom onset, the differences disappeared for TSNF-10 and IL-15, but CCL5 was still increased in female deteriorated patients compared to female stable patients. However, there are no accompanying significant between-sex differences: the difference-in-differences (female deteriorated vs. stable patients compared to male deteriorated vs. stable patients) was not significant for CCL5, and there were no significant differences in direct comparisons of female deteriorated to male deteriorated patients in either unadjusted or adjusted analyses. There is therefore no evidence of a sex difference in CLL5 -- not to mention “innate immune cytokines” in general, as is stated in the conclusion -- associated with disease progression.
In their initial response to our Matters Arising (private communication; we followed protocol of sharing our Matters Arising piece with them before submission and including correspondence in our submission to Nature), Takahashi et al. asserted that they do not misrepresent within-sex differences as between-sex differences. Yet the authors clearly utilize findings of difference within the female patient group to make a claim about factors differentially associated with COVID-19 progression between the sexes, despite a lack of significant findings when comparing both sexes (whether directly or through difference-in-differences).
If these claims are discounted as sex-difference findings, a total of just six between-sex difference findings in five immune markers remain in adjusted analyses. Moreover, for four of these immune markers, a significant difference was found in only one of the statistical comparisons run. That is, if a finding were a true sex difference, we might expect to see differences across more than one comparison within a given analysis strategy. As an example, in longitudinal analyses, CCL5 is higher among male patients than female patients and has a greater difference between male patients vs. controls relative to female patients vs. controls. Yet for IL-8, CXCL-10, ncMono, and CD38 & HLA-DR+ CD8 T cells, a statistically significant difference is only found in either the direct male to female patient comparison or the difference-in-differences comparison at baseline. Takahashi et al. explained their use of the difference-in-differences metric (to account for significant differences in age and BMI between controls and patients), but conveniently ran both comparisons for all immune markers and emphasized only the tests that yielded significant findings for a given marker.
Therefore, these few markers may be interesting candidates for follow-up studies. However, the current findings do not convincingly demonstrate that there are profound sex differences in the immune response to COVID-19, nor that these immune markers underlie sex differences in COVID-19 outcomes.
3. Limitations of the data itself
Finally, it is important to note several limitations of the data Takahashi et al. used to make broad claims of sex differences. The sample size of this study is quite small, and varied subgroups of the sample are employed for many of the analyses. A total of 98 patients were included, but all of the baseline analyses are carried out in a subset of 39 patients (Cohort A). Under the categorization of Cohort A used for the disease progression analysis, only 6 male and 6 female patients deteriorated. 64 healthcare workers were included as controls, but Takahashi et al. only had flow cytometry data for 51 of them and cytokine/chemokine measurements for 43 -- only 6 of whom were male.
There is also a lack of longitudinal samples collected; only half of the 98 patients in the so-called “longitudinal sample” had more than one sample collected, and less than 20% had more than two samples. Only nine healthcare workers had more than one sample collected, eight female and one male, and the single male did not have cytokine/chemokine measurements collected. While collecting multiple samples from hospitalized patients is understandably a challenge, the authors do not discuss the incompleteness of their longitudinal data collection. Furthermore, the authors do not make clear the timing of their longitudinal data collection in the narrative of the paper. This data is available as a Supplemental Table; however, a brief investigation reveals that the time between collection of the first and second samples spanned from 2 to 16 days. This is a huge range, and if samples collected from one sex were on average collected over a longer period of time and/or at later disease stages than the other sex, this could - erroneously - make it seem like there are sex differences in immune response to COVID-19 over time. Takahashi et al. control for days from symptom onset in one of their longitudinal data tables (Extended Data Table 4) but not the other (Extended Data Table 5), potentially invalidating the results presented in the latter. Overall, these gaps and inconsistencies in the data seriously weaken the authors’ ability to make claims about sex differences in longitudinal analyses of COVID-19 disease course.
There is nothing wrong with an exploratory study. But here the authors incautiously overinterpret findings of difference among this small study sample. It is especially unwarranted to make causal claims regarding sex differences based on this data -- i.e. that observed differences are due to biological sex, and that these differences in a few immune markers underlie differential COVID-19 disease outcomes. But the authors do just this, concluding with a call for sex-based vaccines and therapies. This is a big claim, suggesting that sex-specific treatment targeting one of the immune molecules identified in their analysis would improve outcomes for men and women. This, of course, would only work if that molecule truly differed among men and women and also gave rise to differing COVID-19 outcomes. The present data, however, is far too limited to make such a sweeping claim.
Conclusion: A simple but devastating case of conclusions unwarranted by the evidence provided
Takahashi et al. boldly reported findings of sex differences in immune response to COVID-19, but a careful interrogation of their paper reveals serious limitations of the study dataset and design, which call the validity of their conclusions into question. Their data shows that immune responses to COVID-19 are overwhelmingly similar between men and women. They identified a handful of molecules that showed significant differences between the sexes, which can serve as a fruitful starting point for additional studies characterizing COVID-19. However, these modest sex differences were overemphasized in the presentation of this paper, overshadowing other factors that likely have a larger effect on COVID-19 outcomes. This data could just have easily yielded a paper on the importance of age and BMI in immune response to COVID-19, given that almost all of the significant findings in unadjusted analyses disappear after controlling for these factors. And this is to say nothing of other demographic and social factors that we know impact COVID-19 outcomes to varying degrees, such as comorbidities, race/ethnicity, socioeconomic status, and occupation.
It is imperative that scientists communicate clearly and honestly about the analyses conducted, the limitations, and the realistic implications of their research -- especially when conducted on the fraught and assumption-laden area of biological sex differences (see work by Fine, Epstein, Jordan-Young, and Richardson). Yet Takahashi et al. signal no awareness of these issues, instead touting the importance of sex differences in COVID-19 patients without consideration of possible sources of bias in their analysis or the role of alternative explanations for putative sex difference findings. They communicated this irresponsible and inaccurate message to the media, as well. The sensationalized communication and reception of these flawed findings, including the potentially dangerous recommendation of sex-specific vaccines and treatments, is the subject of our next blog post, here.
SUGGESTED CITATION
Gompers, Annika. “No Evidence for a Sex Difference in Immune Response to COVID-19: An Explainer of the GenderSci Lab’s Nature Matters Arising Article.” 2021 Sept. 20. GenderSci Blog. genderscilab.org/blog/nature-matters-arising-explainer
STATEMENT OF INTELLECTUAL LABOR
Annika Gompers wrote the blog post based on the GenderSci Lab’s Matters Arising publication authored by Heather Shattuck-Heidorn, Ann Caroline Danielsen, Joseph Bruch, Helen Zhao, Marion Boulicault, Jamie Marsella, and Sarah Richardson. Lab members, especially Sarah Richardson and Kelsey Ichikawa, edited the post.