Skip to main content

Multilayer networks: aspects, implementations, and application in biomedicine

Abstract

Modeling and analyses of complex systems using network theory have been an object of study for a long time. They have caught attention in many disciplines such as sociology, epidemiology, ecology, psychology, biology, biomedicine, and other fields. Network theory is especially an efficient tool to model biological networks such as gene co-expression networks, protein-protein interaction networks, or pathways. Considering the enhanced resolutions of complex real-world systems, the interest has been directed to multilayered networks. However, despite this surge of recent attention, the use of the multilayer framework in the biological field is still in its youth. In this paper, we review the different aspects and terminologies of multilayered networks. We also briefly discuss the variant applications of the multilayer framework, and finally, we give an overview of various existing applications of the multilayer model in network biology.

Background

In this review, we address the lack of a terminology convention in the field of multilayered networks, by distinguishing their different abstract formulation and aspects. For each aspect, we distinguish the appropriate applications, based on its structure and the state-of-the-art articles. We also describe and illustrate the different implementations of these aspects, with their assets and usage in previous research. In addition, we enumerate the various published state-of-the-art articles, in which the authors used a multilayer framework to model networks in Biomedicine, and solve different biomedical problems. We discuss the convenience of this usage and the reasons to choose specific implementations to solve specific problems.

Introduction

Network theory has been used for many years in the modeling and analysis of complex systems. As the data evolves and becomes more heterogeneous and complex, monoplex networks become an oversimplification of the corresponding systems [1, 2]. In the case of biological networks, the represented systems are complex as well as the problems studied, and exploiting information from a new perspective might improve the understanding of biological structure and functionality [3]. In fact, the classical network representation used for years seems to fail against the heterogeneity of the objects and the relations. This imposes a need to go beyond traditional networks into a richer framework capable of hosting objects and relations of different scales [4], called Multilayer Network.

The term Multilayer Networks has been used by many scientists over the years. These networks have many definitions and implementations. One of these definitions is presented by Allard et al. [5], who define a Multilayer or a Multitype Network by a network with a set of N Nodes each assigned a type from a set of M types. However, the definition of a Multilayer Network is much broader. The simplest definition of a multilayer network is a set of nodes, edges, and layers, where the interpretation of the layers depends on the implementation of the model. One of the main problems faced when studying these graphs is the absence of a terminology and a nomenclature convention. In their review of Multilayer Networks, Kivelä et al. [4] summarize the different naming of these networks in a table of 26 different names. We have come to learn that some of the names are representing the same structure and framework, but others depend on the type of the graph.

Multilayer Networks provide better modeling for complex networks, like biological networks. These networks are heterogeneous in different manners since any biological function is rarely depending on just one element or type of interaction between the elements. The importance of their usage in the domain of biomedicine was briefly discussed by some researchers [1], but to our knowledge, it was never assimilated and reviewed. With this review, we aim at summarizing the different state-of-the-art articles that used a multilayer framework in biomedicine. This article is divided into 3 sections. In the first one, we present an abstract formulation of a multilayer graph and the different criteria for choosing layers. We explain the transformation from a monolayer to a multilayer framework. The second section explores the implementation of the multilayer framework, as well as usage examples. In the third section, we summarize some of the state-of-the-art articles that have been published in different fields of biomedicine and are all using one of the different implementations of multilayer networks.

Multilayered networks: aspects and terminologies

The multilayer nature of graphs has been a subject of study for multiple years. Considering that this framework has many applications in various fields, its interpretation and implementation depend on the subject it is serving. The main difference between the types is the criteria to link a node to a layer. It is indeed crucial to determine how the layers should be assigned when converting from a monoplex to a multilayer model [4]. In this context, two types of networks can be distinguished.

Node-colored graphs

Node-colored graphs (Fig. 1) are representations of systems with heterogeneous nodes. Nodes have different aspects or types, defined by colors. The graph GN (Eq. 1) is a tuple defined by a set of nodes V, a set of edges E, and a set of layers C, where layers represent the color of the nodes [4]. Two nodes on layers α and β are connected with an edge Eαβ.

$$ {\mathrm{G}}_{\mathrm{N}}=\left(V,E,C\right),\kern0.5em {\mathrm{V}}_{\alpha}\in V\times C,\kern0.5em {\mathrm{E}}_{\alpha \beta}\in {V}_{\alpha}\times {V}_{\beta },\kern0.5em where\kern0.5em \alpha, \beta \in C $$
(1)
Fig. 1
figure 1

Node-colored Graphs. Nodes having similar colors are grouped in the same layer. The edges are the same as in the original network

In such graphs, the focus is exploiting the topology of the system from a node-based perspective. In most of the cases, the graph is layer-disjoint, i.e. the node can only belong to one layer, having only one color.

Edge-colored graphs

Edge-colored graphs (Fig. 2) are representations of systems with heterogeneous connections between the nodes. The graph GE (Eq. 2) is a tuple defined by a set of nodes V, a set of colors C with length b, and a set of edges E, each assigned a type or aspect defined by the color. Edges of the same color belong to the same layer. Each layer can contain a subset or the complete set of nodes.

$$ {\displaystyle \begin{array}{c}{G}_E=\left(V,E,C\right)\\ {}{\left\{{G}_{\alpha}\right\}}_{\alpha =1}^b={\left\{\left({V}_{\alpha },{E}_{\alpha}\right)\right\}}_{\alpha =1}^b,\forall \alpha, \beta \in C\kern1em {\mathrm{V}}_{\alpha =}{V}_{\beta },\\ {}{\mathrm{E}}_{\alpha}\in {V}_{\alpha}\times {V}_{\alpha}\times C\\ {}{\left\{{G}_{\alpha}\right\}}_{\alpha =1}^b={\left\{\left({V}_{\alpha },{E}_{\alpha}\right)\right\}}_{\alpha =1}^b\end{array}} $$
(2)
Fig. 2
figure 2

Edge-colored Graphs. The nodes of the original networks are replicated over the different layers. The layers represent the different aspects of relationships in the network. Edges are divided among the layers based on their colors

The graph is node-aligned, i.e. the same node can belong to different layers simultaneously. In such graphs, the focus is on the dynamics of the network and the interactions between its components. It is very efficient to use when studying different aspects of the same system. Two nodes can only connect using one edge of each color. In case the same type of connection should be established between two nodes, this framework can be combined with another one to reach this purpose [4].

Implementations of multilayered networks

In this section, we present to you the main types and implementations of the multilayer framework. For each implementation, we enumerate some of their applications in different fields.

Multiplex networks

Multiplex Networks (Fig. 3) are edge-colored graphs, where the nodes connect with edges belonging to M different aspects (Eq. 3). The edges are embedded based on their type in different layers all containing the complete set of nodes X of length n [4, 6,7,8]. Multiplex networks explore and incorporate the interconnectivity of the same system in multiple channels represented by the layers [9].

$$ Mlayers,\kern3.25em \left\{{G}_{\alpha };\alpha \in \left\{1,\cdots, M\right\}\right\},\kern3.25em {G}_{\alpha }=\left({X}_{\alpha },{E}_{\alpha}\right),\kern0.5em with{X}_{\alpha }=\left\{{x}_1,\cdots, {x}_N\right\} $$
(3)
Fig. 3
figure 3

A Multiplex Graph. The nodes of the network are replicated over layers, and each layer represents a particular aspect of connection between the nodes

A perfect example of a multiplex network is a social network, where different social interactions between individuals are studied, for example, friendship, vicinity, kinship, membership of the same cultural society, partnership or co-workership, etc. [8, 10]. The nodes, in this case, are the individuals, and the edges are the interactions between them. These networks were studied thoroughly over the past years since they are of great importance in affecting our daily lives. The study of these networks is particularly important for social media companies like Facebook, which helps them establish connections between their users and improve their services, including family connections, friendships, social interests, political views, etc., based on the information provided by the users, like backgrounds, demographics, etc. [11].

Another application presented by Cardillo et al. [12] introduces The European Air Transportation Multiplex Network, consisting of 37 layers representing the European airlines, N number of nodes representing the airports in Europe, and links between the nodes representing the direct flights.

Multilevel networks

Multilevel networks (Fig. 4) are edge-colored graphs. They are similar to multiplex networks, with a difference being that layers can contain not only a subset of edges but a subset of nodes as well. The graph (Eq. 4) is a tuple defined by the sets of nodes X, edges E, and layers S with length p.

$$ {\displaystyle \begin{array}{c}M=\left(X,E,S\right)\\ {}S=\left\{{S}_1,\dots, {S}_p\right\}\kern0.5em sub- graphs\\ {} With\kern0.5em {S}_j=\left({X}_j,{E}_j\right),j=1,\dots, p\\ {}X=\underset{j=1}{\overset{p}{\cup }}{X}_j\kern0.5em E=\underset{j=1}{\overset{p}{\cup }}{E}_j\end{array}} $$
(4)
Fig. 4
figure 4

Multilevel Graph. Each layer of the graph is a subset of the original network (nodes and intra-edges). The inter-edges only connect the same nodes on different layers

An example of the application of this framework is the Urban Transportation Network modeled by Aleta et al. [13]. In their paper, the authors present a multilevel model, representing various transportation methods in nine cities in Europe. The nodes are the different stops, while the layers are the different modes of transportation. This network is multilevel because the nodes or the stops do not exist on all the layers.

Multi-hypernetworks

Multi-Hypernetworks or hypergraphs (Fig. 5) are node-based graphs [1, 4], formed by different intersecting communities or sub-networks. A Multi-hypergraph (Eq. 5) is defined by a pair (X, H), X being the set of nodes, and H a multi-set of subsets of X being the edges [14]. These graphs focus on the nodes belonging to the same group rather than the connections between them. Nodes in the same subset are mapped to a layer with fully connected nodes. The notion of edges here differs from normal networks; edges in these networks are called hyperedges and can connect multiple nodes at the same time [4, 15,16,17]. Each hyperedge is then mapped to a single layer. Multi-Hypernetworks are not layer-disjoint, but node-aligned since one node can be assigned to multiple layers, based on the intersection of the subsets.

$$ {\displaystyle \begin{array}{c}\varphi =\left(X,H\right)\\ {}H=\left\{{H}_1,\dots {H}_p\right\} sub- sets\\ {}{E}_{\alpha \beta}=\left\{\left(x,x\right);x\in {X}_{\alpha}\cap {X}_{\beta}\right\},\mathrm{where}\kern0.5em \alpha, \beta \in \mathrm{H}\end{array}} $$
(5)
Fig. 5
figure 5

Hypernetwork. Adapted from [1]. a The network in a monoplex view; b The network in a multilayer view. Nodes belonging to the same ensemble are assigned the same layer. All the nodes on one layer are intra-connected. The same nodes on different layers are interconnected (dotted edges)

Hypernetworks are optimal representations of networks with n-ary relationships since they allow hyperedges connecting more than 2 nodes at the same time [18]. They have been mainly used in Folksonomy, where a semantic structure is created by a collaborative annotation. An example of this application is Flickr, used to share multimedia between users, who are allowed to tag them. The network here can be viewed as a tripartite Hypernetwork consisting of three types of nodes being the users, the resources and the tags, and hyperedges connecting those three, meaning a user annotating a resource with a tag [1, 4, 16, 19].

Another application is presented by Chan and Hsu [20], where they discuss how the usage of hypernetworks in the service science can reveal hidden social structures in human networks, and therefore provide a better understanding of them.

Hypernetworks are also used in team sport networks. As presented by Ramos et al. [18], these networks can be used to study multiple aspects of interactions between the players, like spatiotemporal interactions, which inspects the dynamics of the game at different times and spots.

Interacting/interconnected/interdependent networks and networks of networks

This framework (Fig. 6) is a set of monoplex networks interacting with each other. Nodes can have intra-edges as well as inter-edges connecting them. The inter-edges E define the interactions between the L different monoplex networks G (Eq. 6), but a dependency in interdependent networks, i.e. some of the nodes in each network are dependent on nodes from others [1, 4, 21, 22]. The multilayer model of these networks is node-colored, layer-disjoint formed by embedding the different monoplex networks each into a single layer, then connecting them with their inter-edges. These networks help achieve global synchronization between different sub-systems [1].

$$ {\displaystyle \begin{array}{c}\left\{{G}_1,{G}_2,\dots {G}_L\right\}\\ {}{E}_{\alpha \beta}= interact ion\kern0.5em between\kern0.5em {G}_{\alpha}\kern0.5em and\kern0.5em {G}_{\beta}\end{array}} $$
(6)
Fig. 6
figure 6

Interacting/Interconnected Networks. Adapted from [1]. a The network in a monoplex view; b The network in a multilayer view. Nodes belonging to the same system are assigned the same layer. Inter- and intra-edges are kept the same

Examples of such networks are transportation, telecommunication, electrical grids [23], and social networks [24]. The infrastructure network, in particular, is a perfectly fitting example, since it relies on electricity to function. This dependency between the networks makes this framework vulnerable since a failure in one part can cascade to affect the whole network. This problem has caught a lot of attention and many scientists are still studying it [22, 23, 25,26,27,28].

Multilayered networks and biology

Biological networks are difficult to explore because they contain a mass number of nodes connected in myriad ways. The best way to obtain information from these networks is to summarize the information about the nodes and links conveniently [29]. This summarization follows the criteria which we discussed in our first section but depends on the biological domain. Some domains require edge-colored methodologies to explore the different interactions between the molecules, like gene co-expression networks, or protein-protein interaction networks, or different behavior in cross-species networks. Others require node-colored frameworks, like cell networks to integrate interconnected and interdependent networks.

This usage is still in its infancy in some domains, like gene-expression networks, but very thriving in others, like epidemiology.

In this section, we summarize the usage of the multilayer framework in Biology with state-of-the-art work on this topic.

Protein-protein interaction networks

Protein-protein interaction (PPI) networks are collections of interactions between two or more proteins binding to carry a certain function [30], where nodes are the proteins and edges are physical or functional interactions between them [31]. These networks are heterogeneous in regard to the different types of interactions between them. The network could be seen as an edge-colored Multiplex or Multilevel network inspecting diverse aspects of interactions.

OhmNet [32] is an algorithm that transforms monoplex networks into a multilayer model. This algorithm aims at studying proteins in different tissues and learning their features. In this model, the tissues act as layers, and the proteins act as nodes. The model implemented by OhmNet was used in a recent publication by Kapadia et al. [33] to predict features of a multi-layer blood cell PPI network.

Multilayer PPI networks have also been used to study the life stages in species. In [34], the life stages in Caenorhabditis elegans are modeled in a node-colored multilayer network, where protein interactions are collected from various bioinformatics repositories, and then proteins occurring in different life cycles are distributed over six layers representing the different life stages.

Another Multilayer model introduced by Zhao et al. [35] aims at integrating PPI network, protein domain information, and protein complexes. The authors present a Multiplex edge-colored model consisting of three layers: the physical interaction layer (PIL), containing the protein-protein interactions, sharing domain layer (SDL), with proteins connected in case they share the same domain, and sharing complex layer (SCL), in case they are contained in the same complex. The nodes are the proteins replicated over the three layers (Fig. 7).

Fig. 7
figure 7

Multilayer Protein Network. Adapted from [35]. The figure shows the edge-colored multiplex graph implemented by Zhao et al. [35], with three layers representing different interactions between the proteins

In addition, the High-Throughput data integration method implemented by Liang et al. [36] used an edge-colored Multiplex framework. The base of the graph is PPI networks from yeast and human interactomes which form the two layers, weighted with transcriptional data, and combined with biomedical knowledge.

Cell networks

Cell networks or pathways represent complex functions and reactions that occur in the cell. They describe interactions between genes, proteins, and metabolites [37]. To understand the cell structure, one needs to investigate not only these molecules and their intra-connectivity in isolation, but also their inter-connectivity. Gligorijević and Pržulj in [38] illustrate the cell network as an interdependent network of five layers: phenotypes, genome, proteome, metabolome, and transcriptome. The study of these different substrates results in a heterogeneous network formed of different types of nodes and interactions.

An example of a cell network is implemented by Liu et al. in [37], where they use a node-colored multilayer consisting of a gene-regulatory layer, PPI layer, and a metabolite layer. The genes are connected to their gene products on the PPI layer directly, and proteins are connected to metabolites in a many-to-many relation when they associate or participate in the same chemical reaction.

In [39], Rai et al. demonstrate how the usage of a multilayer framework helps to understand the behavior of cancer cells. In their paper, the authors study seven different cancer types (breast, oral, ovarian, cervical, lung, colon, and prostate), using a multilevel network model. Their model consists of seven layers for two networks, the normal and the disease networks, in which each layer reflects a cancer type. The nodes are the proteins in PPI networks, chosen based on their expression in normal and cancer cells. The common proteins are then extracted in three different sets: common in all normal cells, common in all cancer cells, and common in normal and disease cells (Fig. 8).

Fig. 8
figure 8

Multilayer Cancer Cells’ Network. Adapted from [39]. The Figure shows two graphs, normal and disease. The nodes in these graphs are the proteins, assigned to seven layers reflecting seven cancer types

A similar work by Yu et al. [40] employs a multiplex network of three layers, for PPI networks in three different tissues: breast, prostate, and blood. They study the overlapping between the genes under the action of the drug trichostatin A (TSA) in three diseases it treated, leukemia, breast cancer, and prostate cancer. They identified two drug-target modules of TSA (M17, M18) as the potential treatment patterns of TSA.

Another paper by Li et al. [41] aims at identifying frequent coupled transcription-splicing modules. They build a two-layer interconnected model of gene co-expression networks and exon co-splicing networks. The layers contain respectively genes and exons, inter-connected if there is a relationship between them, and intra-connected if they are respectively co-expressed and co-spliced.

A tool also addressing data integration in Biology using a multilayer approach is Synergy Landscapes by Kuzmin et al. [42]. The tool constructs a multilayer model to connect researchers with resources through molecular interactions. The model integrates three networks: bio-medical co-authorship collaboration networks, molecules interacting in bioprocesses and papers describing them, and networks of bioprocesses involved in various diseases.

As an initiative for pathway data integration, we presented in a previous work a tool called mully [43]. The tool is an R package to create, modify, and visualize multilayer graphs. We aim at collecting pathway knowledge from different databases and embed the different elements into a node-colored multilayer model (Fig. 9).

Fig. 9
figure 9

A mully multilayer model. Each layer reflects a single type of the pathways’ elements. The graph is node-colored

One of the pathway databases is Reactome, which is manually curated, open-source, and fully downloadable [44]. Reactome contains BioPAX-encoded pathways that could be parsed using the R package rBiopaxParser implemented by Kramer et al. [45]. Another pathway resource is ndex, which is an online commons where scientists are able to upload and share networks [46]. The ndex servers can be queried using the R package ndexr implemented by Auer et al. [47], in order to download, modify, and upload networks.

Gene expression networks

The complexity in genetic networks lies in the diversity of interactions between genes. A gene network is a collection of gene-gene regulatory connections [48]. It contains information on genes affecting each other with their expression levels. These networks can be inspected from a multiplex or a multilevel framework, with layers representing co-expression profiles in different settings including different time points, diseases, cells, or organisms, as well as from a Hypernetwork framework, with edges connecting more than two genes [48, 49].

In [50], Li et al. use a multiplex framework to illustrate co-expression networks to identify heavy recurrent subgraphs. The networks are extracted from microarray datasets, and each network is assigned to a layer.

In the paper describing the software muxViz [3], the authors also use genetic networks as an example to demonstrate their tool, used to visualize multiplex networks.

In another publication [51], Klosik et al. use an interdependent framework to study interactions between metabolites and genes. Their model consists of three layers: the gene regulatory layer, the protein-interface layer, and the metabolic layer.

Human neural networks

Neural networks are one of the most complex networks in the human body. A brain network is a structural network where nodes represent neural elements, i.e. the neurons or neuronal regions, and edges represent the physical connection between them, i.e. synapses or axonal projections. The connections in these networks are also affected by spatial factors, which means that the location and distance between the elements play a role in the probability of them being connected [52]. The modeling of brain networks on multiple levels is an evocative application, since they are continuously evolving, and different factors should be considered, such as temporal and spatial factors or different frequency bands. This model has been reviewed and adapted by many scientists to study the human brain in recent years [53,54,55,56,57,58,59].

Jérémy Guillon presents in his Ph.D. thesis [60] a multiplex network of the human brain with Alzheimer’s disease containing information from the brain at different frequency bands [61]. This model combines structural and functional connectivity in the brain and is used to detect core-periphery organizations [62].

Similarly, Dang et al. [63] present in their paper the Multivariate Time-Frequency Multilayer (MTFM) network, which is an edge-colored model to study the dynamics of two complex systems, namely oil-water flow, and driving fatigue in the brain. The model explores these dynamics by dividing the key frequency over L layers, each of which contains nodes representing the channel time-series, and edges representing mutual information connections. The driving fatigue in the brain MTFM network shows the changing of state in the brain during driving from alert to mental fatigue.

A similar model was also introduced by Brookes et al. [64] to study magnetoencephalography (MEG) data connectivity.

Spreading processes and disease behavior networks

Epidemiology is one of the biological fields in which the multilayer framework is used extensively. It is used to study the transmission and spreading of diseases between individuals, which is based on heterogeneous factors, like age, geographical locations, social status, and sexual behavior. These factors are used to assign the individuals of the population classes or types [65].

Examples of spreading networks are the epidemic models, SIR and SIS. In SIR, the individuals are assigned one of three traits: susceptible, meaning not yet infected but susceptible to infection, infected, meaning already carry the disease and can transmit it, and removed, meaning previously infected and do not transmit anymore. The infection starts with some individuals called seeds, and get transferred to adjacent or neighbor individuals at a certain time. So the nodes can go from susceptible to infected, or when cured to removed. In SIS, the individual only goes from infected to susceptible [65,66,67,68,69]. Although these models can be seen as node-colored graphs at different timestamps [70], they are mainly implemented as edge-colored. In [71] the authors use a two-layer interconnected model to study the interaction between two SIR-epidemics. Kivelä et al. [4] list a significant number of references on these implementations. An immunization strategy is also presented in [72] using the multiplex two-layer SIR model.

An adaptation of the SIR model is used by Riad et al. [73] to assess the risk of spreading of Ebola in Uganda. The authors propose a combination of SIR concept with a social network. Their model consists of two layers representing the interactions between the people, a permanent layer being family relations, and a temporal layer being the other general connections. The nodes, which are the individuals, can have either of two types (active and inactive) depending on if they carry the disease or not, and they are distributed in 23 districts.

In their publication [5], Allard et al. discuss some of the difficulties and problems faced in epidemiology when using simple models, like the iid hypothesis, and state the advantages of transforming them into multitype networks, using HIV as an example.

Mao and Yang [74] present another approach using multilayer networks, in which they use a two-layer multiplex model to demonstrate that infections and preventive behavior are transmitted simultaneously and affect each other. The individuals are replicated over both layers. The first layer contains the transmission relationships, and the second contains the interactions of influencing preventive behavior. Later the same author presents a similar model with an extra layer called Information, signifying whether the person was informed or uninformed of the disease [75] (Fig. 10). A similar approach is later used by Carnell et al. [76].

Fig. 10
figure 10

A Multilayer Triple-diffusion Network. Adapted from [75]. The layers illustrate three diffusion processes. The nodes have different colors based on the state of the individuals in each network. The dotted lines connect the same nodes on the three layers. The arrows between the layers denote a positive (green) and negative (red) effect between the processes

dbNEI is a web-based knowledge resource that collects information related to the neuro-endocrine-immune systems (NEI) [77]. The database was later updated to dbNEI2.0, by adding new molecules and information. dbNEI2.0 builds a multilayer network for drug-NEI-disease. The model is node-colored, consisting of three layers, diseases, drugs, and NEI genes layers [78].

In other work, scientists were interested in disease-disease interactions and connections. Halu et al. present in [79] a two-layer model of diseases. On the genotype layer, the diseases are linked in case they share a disease gene, while on the phenotype layer, they are linked if they share a symptom (Fig. 11).

Fig. 11
figure 11

Multiplex disease-disease interaction network. Adapted from [79]. The graph consists of two layers. The nodes are the diseases, connected on the first layer if they share a phenotypic interaction, and on the second if they share a genotypic one

Moreover, in [80], Yu et al. construct a four-layer similarity model of four disease interconnected networks: Human Disease Network Based on Protein Interaction Network (PIDN), Human Disease Similarity Network Based on Symptoms (DSDN), Gene Ontology- and Disease Ontology-Based Disease Similarity Networks (GODN and DODN). They aim at getting more accurate conserved disease modules and find potential disease-disease relationships.

In Table 1, the references are listed for each aspect, implementation and biomedical application of multilayered graphs.

Table 1 Classification of the references based on the used implementation

Conclusions

The usage of Multilayer Graphs has proven to be very promising and prosperous in different fields of application, especially in Biology. Multilayer Graphs allow better inspection and representation of the topology and dynamics of heterogeneous systems compared to monoplex models [1]. In the case of biological networks, the heterogeneity lies in the diversity of interacting substrates, as well as in the various channels of interactions between them. Many scientists have adapted the multilayer model to solve complex biomedical problems in recent years, by using particular implementations for different situations. For instance, muxViz can be used to visualize edge-colored multilayer networks, such as multiplex or multilevel networks for gene expression data. Our tool mully can be used to create, modify, and visualize multi-omics data, like pathways, drugs, and diseases in a node-colored multilayer model [43].

The multilayer framework has many other applications yet to be discovered. Another approach is to mix different implementations in one model. For instance, a node-colored multilayer model built to integrate different omics data can contain five layers: protein, genes, metabolites, diseases, and drugs. However, it can be extended by dividing the protein and the gene layers to exploit gene co-expressions in multiple tissues or environment, and different protein-protein interactions from different experimental protocols [4]. This indeed helps reduce the loss of data and obtaining misleading results [3], detect patterns and correlations, and avoid hairball effects, resulting from the high number of interactions between the nodes.

Availability of data and materials

Not Applicable.

Abbreviations

PPI:

Protein-protein Interaction

TSA:

Trichostatin A

Ph.D.:

Philosophiae doctor

MTFM:

Multivariate Time-Frequency Multilayer

MEG:

Magnetoencephalography

SIR:

Susceptible-Infected-Removed

SIS:

Susceptible-Infected-Susceptible

HIV:

Human immunodeficiency virus

NEI:

Neuro-endocrine-immune

References

  1. Boccaletti S, Bianconi G, Criado R, del Genio CI, Gómez-Gardeñes J, Romance M, et al. The structure and dynamics of multilayer networks. Phys Rep. 2014;544(1):1–122. https://doi.org/10.1016/j.physrep.2014.07.001.

    Article  MathSciNet  Google Scholar 

  2. Traxl D, Boers N, Kurths J. Deep graphs - a general framework to represent and analyze heterogeneous complex systems across scales. Chaos. 2016;26(6):065303. https://doi.org/10.1063/1.4952963.

    Article  MathSciNet  MATH  Google Scholar 

  3. De Domenico M, Porter MA, Arenas A. MuxViz: a tool for multilayer analysis and visualization of networks. J Complex Netw. 2015;3(2):159–76. https://doi.org/10.1093/comnet/cnu038.

    Article  Google Scholar 

  4. Kivelä M, Arenas A, Barthelemy M, Gleeson JP, Moreno Y, Porter MA. Multilayer networks. J Complex Netw. 2014;2(3):203–71. https://doi.org/10.1093/comnet/cnu016.

    Article  Google Scholar 

  5. Allard A, Noël P-A, Dubé LJ, Pourbohloul B. Heterogeneous bond percolation on multitype networks with an application to epidemic dynamics. Phys Rev E. 2009;79(3):036113. https://doi.org/10.1103/PhysRevE.79.036113.

    Article  Google Scholar 

  6. Nicosia V, Bianconi G, Latora V, Barthelemy M. Growing multiplex networks. Phys Rev Lett. 2013;111(5):058701. https://doi.org/10.1103/PhysRevLett.111.058701.

    Article  Google Scholar 

  7. Battiston F, Nicosia V, Latora V. Structural measures for multiplex networks. Phys Rev E. 2014;89(3):032804. https://doi.org/10.1103/PhysRevE.89.032804.

    Article  Google Scholar 

  8. Bianconi G. Statistical mechanics of multiplex networks: entropy and overlap. Phys Rev E. 2013;87(6):062806. https://doi.org/10.1103/PhysRevE.87.062806.

    Article  Google Scholar 

  9. De Domenico M, Solè-Ribalta A, Cozzo E, Kivelä M, Moreno Y, Porter MA, et al. Mathematical formulation of multi-layer networks. Phys Rev X. 2013;3(4). https://doi.org/10.1103/PhysRevX.3.041022.

  10. Solá L, Romance M, Criado R, Flores J, García del Amo A, Boccaletti S. Eigenvector centrality of nodes in multiplex networks. Chaos. 2013;23(3):033131. https://doi.org/10.1063/1.4818544.

    Article  MATH  Google Scholar 

  11. Lewis K, Kaufman J, Gonzalez M, Wimmer A, Christakis N. Tastes, ties, and time: a new social network dataset using Facebook.com. Soc Networks. 2008;30(4):330–42. https://doi.org/10.1016/j.socnet.2008.07.002.

    Article  Google Scholar 

  12. Cardillo A, Gómez-Gardeñes J, Zanin M, Romance M, Papo D, del Pozo F, et al. Emergence of network features from multiplexity. Sci Rep. 2013;3:1344. https://doi.org/10.1038/srep01344.

    Article  Google Scholar 

  13. Aleta A, Meloni S, Moreno Y. A Multilayer perspective for the analysis of urban transportation systems. Sci Rep. 2017;7(1):1–9. https://doi.org/10.1038/srep44359.

    Article  Google Scholar 

  14. Pearson KJ, Zhang T. On spectral hypergraph theory of the adjacency tensor. Graphs Combinations. 2014;30(5):1233–48. https://doi.org/10.1007/s00373-013-1340-x.

    Article  MathSciNet  MATH  Google Scholar 

  15. Zlatić V, Ghoshal G, Caldarelli G. Hypergraph topological quantities for tagged social networks. Phys Rev E Stat Nonlinear Soft Matter Phys. 2009;80(3 Pt 2):036118. https://doi.org/10.1103/PhysRevE.80.036118.

    Article  Google Scholar 

  16. Cromar GL, Zhao A, Yang A, Parkinson J. Hyperscape: visualization for complex biological networks. Bioinformatics. 2015;31(20):3390–1. https://doi.org/10.1093/bioinformatics/btv385.

    Article  Google Scholar 

  17. Thai MT, Wu W, Xiong H. Big Data in Complex and Social Networks (1st. ed.). Chapman & Hall/CRC. 2016;ISBN:978-1-4987-2684-9.

  18. Ramos J, Lopes RJ, Marques P, Araújo D. Hypernetworks reveal compound variables that capture cooperative and competitive interactions in a soccer match. Front Psychol. 2017;8:1379. https://doi.org/10.3389/fpsyg.2017.01379.

    Article  Google Scholar 

  19. Ghoshal G, Zlatić V, Caldarelli G, Newman MEJ. Random hypergraphs and their applications. Phys Rev E. 2009;79(6):066118. https://doi.org/10.1103/PhysRevE.79.066118.

    Article  MathSciNet  Google Scholar 

  20. Chan WKV, Hsu C. How hyper-network analysis helps understand human networks? Serv Sci. 2010;2(4):270–80. https://doi.org/10.1287/serv.2.4.270.

    Article  Google Scholar 

  21. Donges JF, Schultz HCH, Marwan N, Zou Y, Kurths J. Investigating the topology of interacting networks: theory and application to coupled climate subnetworks. Eur Phys J B. 2011;84(4):635–51. https://doi.org/10.1140/epjb/e2011-10795-8.

    Article  Google Scholar 

  22. Baxter GJ, Dorogovtsev SN, Goltsev AV, Mendes JFF. Avalanche collapse of interdependent networks. Phys Rev Lett. 2012;109(24):248701. https://doi.org/10.1103/PhysRevLett.109.248701.

    Article  Google Scholar 

  23. Almoghathawi Y, Barker K, Albert LA. Resilience-driven restoration model for interdependent infrastructure networks. Reliab Eng Syst Safe. 2019;185:12–23. https://doi.org/10.1016/j.ress.2018.12.006.

    Article  Google Scholar 

  24. Wang B, Chen X, Wang L. Probabilistic interconnection between interdependent networks promotes cooperation in the public goods game. J Stat Mech. 2012;2012(11):P11017. https://doi.org/10.1088/1742-5468/2012/11/P11017.

    Article  MathSciNet  Google Scholar 

  25. Buldyrev SV, Parshani R, Paul G, Stanley HE, Havlin S. Catastrophic cascade of failures in interdependent networks. Nature. 2010;464(7291):1025–8. https://doi.org/10.1038/nature08932.

    Article  Google Scholar 

  26. Jin Q, Wang L, Xia C-Y, Wang Z. Spontaneous symmetry breaking in interdependent networked game. Sci Rep. 2014;4:4095. https://doi.org/10.1038/srep04095.

    Article  Google Scholar 

  27. Wang Z, Szolnoki A, Perc M. Optimal interdependence between networks for the evolution of cooperation. Sci Rep. 2013;3:2470. https://doi.org/10.1038/srep02470.

    Article  Google Scholar 

  28. Vespignani A. The fragility of interdependency. Nature. 2010;464(7291):984–5. https://doi.org/10.1038/464984a.

    Article  Google Scholar 

  29. Guimerà R, Nunes Amaral LA. Functional cartography of complex metabolic networks. Nature. 2005;433(7028):895–900. https://doi.org/10.1038/nature03288.

    Article  MATH  Google Scholar 

  30. Andreopoulos W, Labudde D. Protein-protein interaction networks. 2013. http://www.bioforscher.de/bigM/ippb9076rp8sityx/manager/documents/general/pdf/books/chapters/protein_protein_interaction_networks.pdf. Accessed 2 Dec 2019.

    Google Scholar 

  31. Safari-Alighiarloo N, Taghizadeh M, Rezaei-Tavirani M, Goliaei B, Peyvandi AA. Protein-protein interaction networks (PPI) and complex diseases. Gastroenterol Hepatol Bed Bench. 2014;7(1):17–31.

    Google Scholar 

  32. Zitnik M, Leskovec J. Predicting multicellular function through multi-layer tissue networks. Bioinformatics. 2017;33(14):i190–8. https://doi.org/10.1093/bioinformatics/btx252.

    Article  Google Scholar 

  33. Kapadia P, Khare S, Priyadarshini P, Das B. Predicting protein-protein interaction in multi-layer blood cell PPI networks. In: Luhach AK, Jat DS, Hawari KBG, Gao X-Z, Lingras P, editors. Advanced informatics for computing research. ICAICR 2019. Communications in computer and information science, vol. 1076. Singapore: Springer; 2019. p. 240–51. https://doi.org/10.1007/978-981-15-0111-1_22.

    Chapter  Google Scholar 

  34. Shinde P, Jalan S. A multilayer protein-protein interaction network analysis of different life stages in Caenorhabditis elegans. EPL. 2015;112(5):58001. https://doi.org/10.1209/0295-5075/112/58001.

    Article  Google Scholar 

  35. Zhao B, Hu S, Li X, Zhang F, Tian Q, Ni W. An efficient method for protein function annotation based on multilayer protein networks. Hum Genomics. 2016;10(1):33. https://doi.org/10.1186/s40246-016-0087-x.

    Article  Google Scholar 

  36. Liang L, Chen V, Zhu K, Fan X, Lu X, Lu S. Integrating data and knowledge to identify functional modules of genes: a multilayer approach. BMC Bioinformatics. 2019;20(225). https://doi.org/10.1186/s12859-019-2800-y.

  37. Liu X, Maiorino E, Halu A, Loscalzo J, Gao J, Sharma A. Robustness and lethality in multilayer biological molecular networks. bioRxiv. 2019;818963. https://doi.org/10.1101/818963.

  38. Gligorijević V, Pržulj N. Methods for biological data integration: perspectives and challenges. J R Soc Interface. 2015;12(112). https://doi.org/10.1098/rsif.2015.0571.

  39. Rai A, Pradhan P, Nagraj J, Lohitesh K, Chowdhury R, Jalan S. Understanding cancer complexome using networks, spectral graph theory and multilayer framework. Sci Rep. 2017;7(1):1–16. https://doi.org/10.1038/srep41676.

    Article  Google Scholar 

  40. Yu L, Shi Y, Zou Q, Gao L. Studying the drug treatment pattern based on the action of drug and multi-layer network model. bioRxiv. 2019;780858. https://doi.org/10.1101/780858.

  41. Li W, Dai C, Liu C-C, Zhou XJ. Algorithm to identify frequent coupled modules from two-layered network series: application to study transcription and splicing coupling. J Comput Biol. 2012;19(6):710–30. https://doi.org/10.1089/cmb.2012.0025.

    Article  MathSciNet  Google Scholar 

  42. Kuzmin K, Gaiteri C, Szymanski BK. Synergy landscapes: a multilayer network for collaboration in biological research. In: Wierzbicki A, Brandes U, Schweitzer F, Pedreschi D, editors. Advances in network science. NetSci-X 2016. Lecture notes in computer science, vol. 9564. Cham: Springer; 2016. p. 205–12. https://doi.org/10.1007/978-3-319-28361-6_18.

    Chapter  Google Scholar 

  43. Hammoud Z, Kramer F. mully: an R package to create, modify and visualize multilayered graphs. Genes (Basel). 2018;9(11):519. https://doi.org/10.3390/genes9110519.

    Article  Google Scholar 

  44. Croft D, Mundo AF, Haw R, Milacic M, Weiser J, Wu G, et al. The Reactome pathway knowledgebase. Nucleic Acids Res. 2014;42(D1):D472–7. https://doi.org/10.1093/nar/gkt1102.

    Article  Google Scholar 

  45. Kramer F, Bayerlová M, Klemm F, Bleckmann A, Beissbarth T. rBiopaxParser--an R package to parse, modify and visualize BioPAX data. Bioinformatics. 2013;29(4):520–2. https://doi.org/10.1093/bioinformatics/bts710.

    Article  Google Scholar 

  46. Pratt D, Chen J, Welker D, Rivas R, Pillich R, Rynkov V, et al. NDEx, the network data exchange. Cell Syst. 2015;1(4):302–5. https://doi.org/10.1016/j.cels.2015.10.001.

    Article  Google Scholar 

  47. Auer F, Hammoud Z, Ishkin A, Pratt D, Ideker T, Kramer F. ndexr-an R package to interface with the network data exchange. Bioinformatics. 2018;34(4):716–7. https://doi.org/10.1093/bioinformatics/btx683.

    Article  Google Scholar 

  48. Brazhnik P, de la Fuente A, Mendes P. Gene networks: how to put the function in genomics. Trends Biotechnol. 2002;20(11):467–72. https://doi.org/10.1016/S0167-7799(02)02053-X.

    Article  Google Scholar 

  49. Penfold CA, Wild DL. How to infer gene networks from expression profiles, revisited. Interface Focus. 2011;1(6):857–70. https://doi.org/10.1098/rsfs.2011.0053.

    Article  Google Scholar 

  50. Li W, Liu C-C, Zhang T, Li H, Waterman MS, Zhou XJ. Integrative analysis of many weighted co-expression networks using tensor computation. PLoS Comput Biol. 2011;7(6):e1001106. https://doi.org/10.1371/journal.pcbi.1001106.

    Article  MathSciNet  Google Scholar 

  51. Klosik DF, Grimbs A, Bornholdt S, Hütt M-T. The interdependent network of gene regulation and metabolism is robust where it needs to be. Nat Commun. 2017;8(1):1–9. https://doi.org/10.1038/s41467-017-00587-4.

    Article  Google Scholar 

  52. Bullmore E, Sporns O. Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci. 2009;10(3):186–98. https://doi.org/10.1038/nrn2575.

    Article  Google Scholar 

  53. Muldoon SF, Bassett DS. Network and multilayer network approaches to understanding human brain dynamics. Philos Sci. 2016;83(5):710–20. https://doi.org/10.1086/687857.

    Article  MathSciNet  Google Scholar 

  54. Mandke K, Meier J, Brookes MJ, O’Dea RD, Van Mieghem P, Stam CJ, et al. Comparing multilayer brain networks between groups: introducing graph metrics and recommendations. NeuroImage. 2018;166:371–84. https://doi.org/10.1016/j.neuroimage.2017.11.016.

    Article  Google Scholar 

  55. Puxeddu MG, Petti M, Mattia D, Astolfi L. The optimal setting for multilayer modularity optimization in multilayer brain networks*. In: 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC), Berlin, Germany; 2019. p. 624–7. https://doi.org/10.1109/EMBC.2019.8856674.

    Chapter  Google Scholar 

  56. Pedersen M, Zalesky A, Omidvarnia A, Jackson GD. Reply to Yang et al.: multilayer network switching and behavior. PNAS. 2019;116(34):16673. https://doi.org/10.1073/pnas.1910493116.

    Article  Google Scholar 

  57. Pedersen M, Zalesky A, Omidvarnia A, Jackson GD. Brain connectivity dynamics: multilayer network switching rate predicts brain performance. bioRxiv. 2018;403105. https://doi.org/10.1101/403105.

  58. Vaiana M, Muldoon S. Multilayer Brain Networks. J Nonlinear Sci. 2018. https://doi.org/10.1007/s00332-017-9436-8.

  59. De Domenico M. Multilayer modeling and analysis of human brain networks. Gigascience. 2017;6(5):1–8. https://doi.org/10.1093/gigascience/gix004.

    Article  Google Scholar 

  60. Guillon J. Multilayer approach to brain connectivity in Alzheimer’s disease. Neuroscience. Pierre and Marie Curie University, 2018. English. tel-01985286. https://tel.archives-ouvertes.fr/tel-01985286/file/PhD_Thesis_v2.0.pdf.

  61. Guillon J, Attal Y, Colliot O, La Corte V, Dubois B, Schwartz D, et al. Loss of brain inter-frequency hubs in Alzheimer’s disease. Sci Rep. 2017;7(1):10879. https://doi.org/10.1038/s41598-017-07846-w.

    Article  Google Scholar 

  62. Battiston F, Guillon J, Chavez M, Latora V, De Vico Fallani F. Multiplex core-periphery organization of the human connectome. J R Soc Interface. 2018;15(146):20180514. https://doi.org/10.1098/rsif.2018.0514.

    Article  Google Scholar 

  63. Dang W, Gao Z, Lv D, Liu M, Cai Q, Hong X. A novel time-frequency multilayer network for multivariate time series analysis. New J Phys. 2018;20(12):125005. https://doi.org/10.1088/1367-2630/aaf51c.

    Article  Google Scholar 

  64. Brookes MJ, Tewarie PK, Hunt BAE, Robson SE, Gascoyne LE, Liddle EB, et al. A multi-layer network approach to MEG connectivity analysis. NeuroImage. 2016;132:425–38. https://doi.org/10.1016/j.neuroimage.2016.02.045.

    Article  Google Scholar 

  65. Vazquez A. Spreading of infectious diseases on heterogeneous populations: multi-type network approach. Phys Rev E. 2006;74(6):066114. https://doi.org/10.1103/PhysRevE.74.066114.

    Article  MathSciNet  Google Scholar 

  66. Salehi M, Sharma R, Marzolla M, Magnani M, Siyari P, Montesi D. Spreading processes in multilayer networks. IEEE Trans Netw Sci Eng. 2015;2(2):65–83. https://doi.org/10.1109/TNSE.2015.242596.

    Article  Google Scholar 

  67. Saumell-Mendiola A, Serrano MÁ, Boguñá M. Epidemic spreading on interconnected networks. Phys Rev E. 2012;86(2):026106. https://doi.org/10.1103/PhysRevE.86.026106.

    Article  Google Scholar 

  68. Sahneh FD, Scoglio C, Chowdhury FN. Effect of coupling on the epidemic threshold in interconnected complex networks: a spectral analysis. In: 2013 American Control Conference, Washington, DC; 2013. p. 2307–12. https://doi.org/10.1109/ACC.2013.6580178.

    Chapter  Google Scholar 

  69. de Arruda GF, Cozzo E, Peixoto TP, Rodrigues FA, Moreno Y. Disease localization in multilayer networks. Phys Rev X. 2017;7(1):011014. https://doi.org/10.1103/PhysRevX.7.011014.

    Article  Google Scholar 

  70. Hindes J, Singh S, Myers CR, Schneider DJ. Epidemic fronts in complex networks with metapopulation structure. Phys Rev E. 2013;88(1):012809. https://doi.org/10.1103/PhysRevE.88.012809.

    Article  Google Scholar 

  71. Zhou S, Xu S, Wang L, Liu Z, Chen G, Wang X. Propagation of interacting diseases on multilayer networks. Phys Rev E. 2018;98(1–1):012303. https://doi.org/10.1103/PhysRevE.98.012303.

    Article  Google Scholar 

  72. Singh V, Verma P, Muthukumaar V, Kumar V, Tewari M, Lai K-K, et al. Immunization strategy for epidemic spreading based on membership (m) over a multilayer network. Bus Strategy Dev. 2019. https://doi.org/10.1002/bsd2.87.

  73. Riad MH, Sekamatte M, Ocom F, Makumbi I, Scoglio CM. Risk assessment of Ebola virus disease spreading in Uganda using a multilayer temporal network. bioRxiv. 2019;645598. https://doi.org/10.1101/645598.

  74. Mao L, Yang Y. Coupling infectious diseases, human preventive behavior, and networks – a conceptual framework for epidemic modeling. Soc Sci Med. 2012;74(2):167–75. https://doi.org/10.1016/j.socscimed.2011.10.012.

    Article  Google Scholar 

  75. Mao L. Modeling triple-diffusions of infectious diseases, information, and preventive behaviors through a metropolitan social network—an agent-based simulation. Appl Geogr. 2014;50:31–9. https://doi.org/10.1016/j.apgeog.2014.02.005.

    Article  Google Scholar 

  76. Granell C, Gómez S, Arenas A. Dynamical interplay between awareness and epidemic spreading in multiplex networks. Phys Rev Lett. 2013;111(12):128701. https://doi.org/10.1103/physrevlett.111.128701.

    Article  Google Scholar 

  77. Zhuang Y, Li S, Li Y. dbNEI: a specific database for neuro-endocrine-immune interactions. Neuro Endocrinol Lett. 2006;27(1–2):53–9.

    Google Scholar 

  78. Zhang J, Ma T, Li Y, Li S. dbNEI2.0: building multilayer network for drug–NEI–disease. Bioinformatics. 2008;24(20):2409–11. https://doi.org/10.1093/bioinformatics/btn388.

    Article  Google Scholar 

  79. Halu A, De Domenico M, Arenas A, Sharma A. The multiplex network of human diseases. NPJ Syst Biol Appl. 2019;5(1):1–12. https://doi.org/10.1038/s41540-019-0092-5.

    Article  Google Scholar 

  80. Yu L, Yao S, Gao L, Zha Y. Conserved disease modules extracted from multilayer heterogeneous disease and gene networks for understanding disease mechanisms and predicting disease treatments. Front Genet. 2019. https://doi.org/10.3389/fgene.2018.00745.

Download references

Acknowledgments

I would like to acknowledge the help of my friend Farah Diab, and of my colleagues Florian Auer, Dominik Müller, Peter Parys, Iñaki Soto Rey, Johann Frei, Samantha Cramer, Sophia Reichenzeller, and Alicia D’Angelo.

Funding

This work is a part of the Multipath and coNfirm Projects funded by the GERMAN MINISTRY OF EDUCATION AND RESEARCH (Bundesministerium für Bildung und Forschung, BMBF) grant respectively FKZ01ZX1508 and FK01ZX1708D.

Author information

Authors and Affiliations

Authors

Contributions

ZH contributed to Conceptualization, Investigation, and Writing – Original Draft Preparation. FK contributed to Funding Acquisition, Supervision, and Writing – Review & Editing. Both authors approved the submitted version of this paper.

Corresponding author

Correspondence to Zaynab Hammoud.

Ethics declarations

Ethics approval and consent to participate

Not Applicable.

Consent for publication

Not Applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hammoud, Z., Kramer, F. Multilayer networks: aspects, implementations, and application in biomedicine. Big Data Anal 5, 2 (2020). https://doi.org/10.1186/s41044-020-00046-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41044-020-00046-0

Keywords