We outline a network method to synthesize a literature overview from search results obtained by multiple team members. Several network statistics are used to create a single representativeness ranking. We illustrate the method with the dispersed literature on a common misinterpretation of analysis of covariance (ANCOVA). The network method yields a top ten list of the most relevant articles that students and researchers can take as a point of departure for a more detailed study on this topic. The proposed methodology is implemented in Shiny, an open-source R package.

Analysis of covariance (ANCOVA) remains a widely misunderstood approach for dealing with group differences on potential covariates (

Dora wants to assess whether, in her own university, men earn more than women. She has access to the salaries of a subset of researchers, and, as expected, men earn significantly more than women (

As explained in Miller and Chapman (

As detailed below, 44 students selected articles from the pertinent literature (i.e., work on the misunderstanding of ANCOVA), and rated their relevance (i.e., the importance and informativeness of the article with respect to the problem under consideration). One can organize and synthesize the resulting data in different ways. For instance, one could focus on the frequency with which the students reported an article. The disadvantage is that hidden treasures—valuable articles found by only a few students—are overlooked. Another method is to consider the average relevance rating that an article receives; the disadvantage here is that this method ignores the wisdom of the group, as an article that has been found by a single user and rated “10” may falsely appear as highly important. In addition, neither of these two methods quantifies the possible association between different articles.

As an alternative way to analyze and visualize the outcomes of the literature search, we outline two network models. A network model yields a flexible representation of the importance of objects and their relationships. Its distinguishing feature is that the schema can be viewed as a graph in which object types are nodes and relationships between the objects are edges. For instance, in a network of depression, the nodes correspond to the symptoms (e.g., sleep loss, fatigue, loss of appetite) and edges between the symptoms quantify the strength of association, such that a pronounced association between sleep loss and fatigue will be reflected by a prominent edge between the nodes that correspond to those symptoms (e.g.,

We first used the

The literature search was conducted by 44 students in the 2016 Research Master’s course “Good Research Practices” at the Department of Psychology of the University of Amsterdam. Each student collected 40 articles published on the ANCOVA problem: 20 articles published prior to the seed article by Miller and Chapman (

Note that the literature search departed from knowledge of the Miller and Chapman (

For concreteness, consider the hypothetical networks shown in Figure

Two different networks. On the left, a co-occurrence network based on the mock data in Table

Mock data from five raters who each reported two or three relevant articles in random order. The year uniquely identifies an article. (e.g., the article “1982” is reported by raters 1, 2, and 5).

Rater | Article 1 | Article 2 | Article 3 |
---|---|---|---|

1 | 1982 | 2008 | |

2 | 2001 | 2008 | 1982 |

3 | 1967 | 2008 | 2001 |

4 | 1967 | 2001 | |

5 | 1982 | 2008 |

To quantify how central an article is, the co-occurrence network method uses several centrality measures. These centrality measures can be seen as indicators of the importance of that node (

Centrality measures for the co-occurrence network on the mock data.

Article | Betweenness | Closeness | Strength |
---|---|---|---|

1967 | –.072 | –1.08 | –1.16 |

1982 | –0.72 | –0.61 | –0.38 |

2001 | 0.06 | 0.84 | 0.38 |

2008 | 1.39 | 0.84 | 1.16 |

The node from 2008 has the highest betweenness, because it connects every article with each other and has paths that are stronger. The

For a directed network, the centrality measures consist of

Centrality measures for the citation network on the mock data.

Article | Betweenness | In-degree | Out-degree |
---|---|---|---|

1967 | –0.5 | 0 | –1.22 |

1982 | 1.5 | 1.22 | 0 |

2001 | –0.5 | 0 | 0 |

2008 | –0.5 | –1.22 | 1.22 |

The

A citation network is created as follows. First, we create an

We wrote an R script and an R Shiny web application to create adjacency matrices for the networks, plot the networks, apply the Bayesian rank-based method, and implement this method for one’s own literature synthesis. These can be found in the online appendix (

In both networks, we can rank articles according to their corresponding network-specific importance measures (i.e., the centrality indices). In order to arrive at a single importance ranking, the importance measures need to be aggregated across the different centrality indices. This aggregation can be accomplished by introducing a latent, normally distributed variable of importance whose values are constrained by the observed ordinal information. In statistics, this technique is known as data augmentation, and the latent variable can be estimated through Gibbs sampling. Thus, for each ordinal data point (i.e., rank), its corresponding latent value of importance is estimated by means of a posterior distribution, that is, a representation of uncertainty across the different importance values. In the past, this technique has been used to estimate the polychoric correlation coefficient (

For illustration, we apply the network method to the subset of 233 articles in the literature before 2001. Results for the literature after 2001 are presented in online Appendix A (

To illustrate how one could create a top-ten list of relevant articles without the benefit of a network model, we show the results of a descriptive method to rank articles on relevance. Based on ad-hoc cut-offs, we included articles only when reported by a minimum of 10 raters and with a mean relevance grade of at least 8. The result is shown in Table

Articles found by at least 10 raters and with a minimum mean relevance grade of 8.

Article | Found by | Mean relevance grade (SD) |
---|---|---|

Evans & Anastasio ( |
42 | 8.8 (1.2) |

Lord ( |
39 | 8.6 (1.3) |

Elashoff ( |
36 | 8.4 (1.2) |

Overall & Woodward (1997) | 32 | 8.3 (0.9) |

Lord (1969) | 26 | 8.4 (1.3) |

Adams et al. ( |
21 | 8.1 (1.2) |

Glass et al. ( |
13 | 8.1 (1.2) |

Storandt & Hudson ( |
13 | 8.2 (1.1) |

We estimated the co-occurrence network using all articles found by the 44 raters before 2001. The network is displayed in Figure

The co-occurrence network applied to the search process for the ANCOVA-pitfall literature before 2001. Each node represents a reported article. Undirected edges represent the number of times the two articles were reported by the same rater. Thicker and saturated edges represent stronger connections.

Centrality plot containing all centrality indices for the single nodes of the co-occurrence network. For readability, only 44 of the 244 articles are shown.

We then computed Kendall’s tau, a rank-based correlation coefficient, for the correlations between the article’s relevance ratings and the various centrality measures (

Kendall correlations and 95% credible intervals for the correlations between the mean relevance grades and centrality measures of the articles used in the co-occurrence network.

τ | 95% credible interval | |
---|---|---|

Relevance grade — Betweenness | 0.255 | [0.164, 0.337] |

Relevance grade — Closeness | 0.224 | [0.134, 0.307] |

Relevance grade — Strength | 0.258 | [0.166, 0.341] |

The aggregated importance ranks of the co-occurrence network were then used to construct a list of the top ten most relevant articles. This list is shown in Table

Top ten articles in the entire search before 2001. The ranks are the final network ranks obtained from the co-occurrence network.

Article | Title | Rank |
---|---|---|

Evans & Anastasio ( |
Misuse of analysis of covariance when treatment effect and covariate are confounded. | 1 |

Lord ( |
A paradox in the interpretation of group comparisons. | 2 |

Overall & Woodward ( |
Nonrandom assignment and the analysis of covariance. | 3 |

Cochran ( |
Analysis of covariance: Its nature and uses. | 4 |

Elashoff ( |
Analysis of covariance: A delicate instrument. | 5 |

Porter & Raudenbush ( |
Analysis of covariance: Its model and use in psychological research. | 6 |

Adams et al. ( |
Analysis of covariance as a remedy for demographic mismatch of research subject groups: Some sobering simulations. | 7 |

Lord (1969) | Statistical adjustments when comparing preexisting groups. | 8 |

Wainer ( |
Adjusting for differential base rates: Lord’s paradox again. | 9 |

Keselman et al. ( |
Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analysis. | 10 |

As shown in Table

Kendall correlations and 95% credible intervals computed between the final rank based on the co-occurrence network and ranks based on mean relevance grade and number of raters.

τ | 95% credible interval | |
---|---|---|

A: Rank by number of raters — Final rank | 0.789 | [0.687, 0.858] |

B: Rank by relevance grade — Final rank | 0.241 | [0.150, 0.325] |

Rank by average of A and B — Final rank | 0.554 | [0.459, 0.631] |

For the construction of the citation network we selected a subset of 20 key articles from all raters’ nominations using the following criteria: (a) select 15 articles with the highest average relevance grade based on a minimum of five nominations; (b) select five articles with the highest average relevance grade based on a maximum of four nominations; and (c) in case of ties, include all articles with the same relevance grade.

As there were several ties in the relevance grades, the selection criteria yielded 26 articles in total. For the 15 best-graded articles with a minimum of five nominations, we observed a cut-off inclusion grade of 7.86. Articles with tied scores were added to the set, resulting in the selection of 17 articles. For the 5 best-graded articles with a maximum of four nominations we observed a cut-off inclusion grade of 9.00. Articles with tied scores were also added to the set, resulting in the selection of nine articles. The resulting network can be seen in Figure

The citation network applied to the search process for the ANCOVA-pitfall literature before 2001. Each node represents a reported article. Directed edges represent whether or not an article cited another article. Connected nodes represent a source-target relation. The source is the paper that cites the target, for example the arrow from Cox (

The citation network is not fully connected, because some articles do not cite any of the other articles, and do not receive any citations from those articles. Articles (i.e., nodes) with many incoming arrows are cited relatively often. The citation network can be analyzed using the same technique we applied to the co-occurrence network. Figure

Centrality plot containing all centrality indices for the single nodes. Note that closeness requires a fully connected network and is therefore omitted.

As before, we computed correlations between the centrality indices and the mean relevance grades. As can be seen in Table

Kendall correlations and 95% credible intervals computed between the mean relevance grades and centrality measures of the articles used in the citation network.

τ | 95% credible interval | |
---|---|---|

Relevance grade — Betweenness | –0.090 | [–0.333, 0.168] |

Relevance grade — In-degree | 0.012 | [–0.240, 0.261] |

Relevance grade — Out-degree | –0.192 | [–0.423, 0.078] |

The aggregated importance ranks of the citation network were then used to construct a top ten list of relevant articles. This list is shown in Table

Top ten articles in the entire search before 2001. The ranks are the final network ranks obtained from the citation network.

Article | Title | Rank |
---|---|---|

Elashoff ( |
Analysis of covariance: A delicate instrument. | 1 |

Huitema ( |
The analysis of covariance and alternatives. | 2 |

Evans & Anastasio ( |
Misuse of analysis of covariance when treatment effect and covariate are confounded. | 3 |

Loftin ( |
The extreme dangers of covariance corrections. | 4 |

Wainer ( |
Adjusting for differential base rates: Lord’s paradox again. | 5 |

Cochran ( |
Analysis of covariance: Its nature and uses. | 6 |

Glass et al. ( |
Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. | 7 |

Porter & Raudenbush ( |
Analysis of covariance: Its model and use in psychological research. | 8 |

Cox & McCullagh ( |
Some aspects of analysis of covariance. | 9 |

Adams et al. ( |
Analysis of covariance as a remedy for demographic mismatch of research subject groups: Some sobering simulations. | 10 |

As shown in Table

Kendall correlations and 95% credible intervals computed between the final rank based on the citation network and ranks based on mean relevance grade and number of raters.

τ | 95% credible interval | |
---|---|---|

A: Rank by number of raters — Final rank | –0.032 | [–0.281, 0.220] |

B: Rank by relevance grade — Final rank | 0.126 | [–0.138, 0.363] |

Rank by average of A and B — Final rank | 0.077 | [–0.182, 0.319] |

We used two networks to organize a dispersed literature. Compared to the descriptive method (i.e., a list of the average relevance grade and the number of times an article is reported), the networks allow one to discover relationships between articles, and discover hidden treasures. Nevertheless, the value of the network approach over the descriptive method warrants future empirical scrutiny.

For this particular project we started with one particular seed paper (i.e.,

To facilitate the use of the network method we recommend an additional resource over and above the

Screenshot of the Literature Networks web application showing a co-occurrence network with a corresponding top 10 list of most relevant articles. See online Appendix C (

As suggested to us in the review process, literature network models may find application in meta-analysis, where a key concern is that the contributions to the literature are based on one or two influential groups. A citation network can help visualize the interdependencies between individual contributions that would otherwise remain hidden. Specifically, the citation network can help identify different clusters of research groups, and illuminate the extent to which they consistently find the same or different results.

Finally, it should be noted that the two network models can operate sequentially instead of in parallel; specifically, citation networks could be created automatically, and the results from these networks could be used to create highly informative items for a subsequent assessment of relevance.

In sum, we have demonstrated how different network models can be applied to a dispersed scientific literature, making it easy to inspect the relationships between various articles and gauge their relative importance.

The authors have no competing interests to declare.