 Research
 Open Access
A hybrid model for short term realtime electricity price forecasting in smart grid
 Xing Luo^{1, 2},
 Xu Zhu^{1, 3} and
 Eng Gee Lim^{2}Email author
 Received: 20 June 2018
 Accepted: 26 September 2018
 Published: 17 October 2018
Abstract
Background
With the prominent growth of power market, realtime electricity price has become a trend in smart grid as it enables moderation of power consumption of customers. Accurate forecast of realtime price (RTP) has much influence on customers’ behaviors, such as better scheduling operating time of domestic appliances in order to maximize benefit. In this paper, an innovative hybrid RTP forecasting model considering linear and nonlinear behaviors within input data, is proposed to forecast the shortterm electricity prices in smart grid.
Results
The effectiveness of the proposed hybrid forecasting model is verified by numerical results in terms of forecasting performance evaluations. The results clearly demonstrate that our approach is effective in RTP forecasting with a high accuracy. The mean absolute percentage error (MAPE) is approximate to 3.5% and it also significantly outperforms the existing models.
Conclusion
Based on the achieved results, we can conclude that the proposed hybrid model is an accurate and efficient tool in shortterm RTP forecasting and it is potentially effective to a variety of forecasting tasks.
Keywords
 Power market
 RTP forecasting
 Hybrid model
 Shortterm electricity prices
 Smart grid
Background
Realtime price (RTP), also referred to as dynamic tariff or spot price which was first introduced in the 1980s [1], nowadays is tentatively applied to the power system in many countries including the US, Australia, etc. The realtime price tariff is an inexorable trend in next generation of power system reforming [2, 3]. Unlike regulated markets which the companies determine prices independently, electricity prices are significantly dependent on a supply–demand relationship in a deregulated market. Generally, RTP offers higher prices during peak load demand periods and provides lower prices during offpeak load demand periods [4, 5]. In consideration of the manufacturing cost in different load levels, the dynamic tariff is a potential load management method for properly allocating incremental prices of electricity consumption to the time delivery, thus ensuring the overall economic rationality [6].
In addition, RTP tariff is broadly utilized as a basic control signal to support the demand response management (DRM) which is an excellent longterm solution to improving energy efficiency and reducing wastage [7, 8]. On the one hand, RTP tariff is benefit to power grid as it offers specific price instructions for participants to average the power usage at different time so that alleviates the load burden of power grid especially in peak demand time. On the other hand, such an electricity tariff encourages consumption by price reduction during periods of abundance and allows customers to have multiple choices to determine the time of electricity consumption. The participants in electricity market can regulate the operating time of electrical devices automatically or manually during highprice periods and gain the benefits from lowprice periods via DRM, thus achieving the aims of reducing energy usage and saving electric bills for themselves [5, 9–11]. Therefore, the research on RTP tariff is of interest to researchers, production companies, investors, independent market operators and large industrial consumers in recent years [12, 13].
Moreover, the realtime price is normally provided with the instantaneous property. Thus, it is a necessity to forecast RTP in advance in this competitive electricity market for electricity consumers and power suppliers in scheduling their operations and controlling the price risks. Over last two decades, much research has been conducted on RTP forecasting. In summary, the existing methods can be classified into two main categories: machine learning based methods like SVM (Support Vector Machine) and ANN (artificial neural network) [14–16], and statistical time series based methods like ARIMA (auto regressive integrated moving average) model and GARCH (generalized auto regressive conditional heteroskedasticity) model [17, 18].
Specifically, in [15], the authors proposed methods including hybrid networks of selforganized map (SOM) and supportvector machine (SVM) to predict shortterm electricity price. With the trained network, one can predict the future hourly electricity prices in one day ahead. To confirm its feasibility, the proposed model had been trained and tested on the data of historical energy prices from the New England electricity market. In addition, in [16], a sensitivity analysis of similar days (SD) parameters to rise the accuracy of ANN model and SDbased shortterm price forecasting model were presented. In order to train the network, a large sum of data were used. The model had been tested in PennsylvaniaNew JerseyMaryland (PJM) electricity market. The results showed that the mean absolute percentage error (MAPE) was around 11%. Furthermore, in [17], the authors introduced a method to predict nextday electricity prices based on the ARIMA methodology which was used to analyze the time series problem. The ARIMA model was tested in California electricity market. More than 30day historical data samples were required to train the model.
However, the shared limitation of the mentioned studies above is that a large number of historical RTP data is required for training the model. The insufficient training data causes considerable estimation errors. Hence, our research in the paper mainly concentrates on building an effective estimation model for electricity price forecasting in smart grid with high accuracy by using limited sets of historical data. In order to evaluate the performance of methods, numerical error measures such as mean absolute error (MAE), means square error (MSE), rootmean square error (RMSE) and mean absolute percentage error (MAPE) are also used in this work.
 (1)
A hybrid RTP forecasting model which is a consolidation of leastsquare (LS) fitting model, grey prediction (GP) model and artificial neural network (ANN), is proposed. The LS fitting model considers the linear behavior of the time series data and the GP model considers the nonlinear behavior. However, the ANN model is an optional forecasting procedure and used in the error optimization.
 (2)
Less historical RTP data is required, thereby improving the practicability. Since both LS and GP models can be established on the basis of a small number of data sets, the proposed hybrid forecasting model is easy to install and more practical compared with the previous methods.
 (3)
The accuracy of time series RTP forecasting increases by using the hybrid model. The effectiveness of the hybrid forecasting model is verified by numerical results in terms of MAE, MSE, RMSE and MAPE evaluations. The results indicate that our method is an accurate and efficient tool to forecast the dayahead RTP and it also significantly outperforms the previous methods.
To the best of our knowledge, this is the first work of combining above pieces together in RTP forecasting.
Method
This section introduces the methodology which includes the architecture of the proposed forecast strategy and the specific description of the proposed hybrid forecasting model in this work.
Architecture of the proposed forecasting strategy
Considering time scales, the RTP forecasting is classified into ultrashort term, short term, medium term and long term [19]. The Ultrashort term is from several minutes to 1 h ahead forecasting. The short term means the forecasting values from 1 h to several hours. From a few hours to 1 week ahead forecasting is defined as the medium term and beyond that it is the long term forecasting. However, we focus on the day (24 h) ahead RTP forecasting with a resolution of 0.5 h in this work, which belongs to the short term forecasting.
where P_{t} is the forecasting RTP at time t. L_{t} and N_{t} represent the estimations of linear behavior and nonlinear behavior, respectively, of the input data. Additionally, \(E_{t}^{\ast }\) which is an optional forecasting component, denotes the error optimization procedure.
In next subsections, the specific descriptions of the relevant forecasting components in the hybrid model are introduced in details.
Least squares fitting model for linear behavior forecasting
On the one side, a higher value of the fitting degree d leads to a better performance of the estimation when J is in a reasonable range. On the other hand, it results more complexity of the calculation and more CPU wastage. Therefore, selecting an appropriate fitting degree in the fitting model is significant and may lead to a better linear behavior estimation performance.
Total square errors with different values of fitting degree; d=1 to 7 are evaluated
d  1  2  3  4  5  6  7 

\(J^{\times 10^{2}}\)  5.601  4.152  3.223  2.988  2.995  2.981  2.950 
Values of the parameters in L_{t}. Fourier format is selected and d=4
Parameters  a _{0}  a _{1}  a _{2}  a _{3}  a _{4}  b _{1}  b _{2}  b _{3}  b _{4}  ω 

Values  32.640  0.570  2.153  0.022  0.831  2.950  4.158  1.738  1.185  0.289 
Grey prediction model for nonlinear behavior forecasting
The GP model or GM(1,1) was first proposed to deal with the data in grey system. It is able to analyze system that includes insufficient information and unapparent relationship [25–27]. Hence, the GP model is often used in predicting data in nonlinear system based on limited information. It transforms the forms of the irregular discrete sequences and displays the potential regularities within the sequences. Transforming the forms of the sequences can make the properties of stochastic and randomness get weaker thereby turning irregular sequences to regular ones [28–30]. Since only a few nonlinear data proceeded from LS fitting model are used, it is quite appropriate to employ the GP model to estimate the nonlinear behavior within the input data on this stage.
Artificial neural network for error optimization
The artificial neural network (ANN) is also a nonlinear modeling where any prior knowledge of relationship between input and output is needed [31]. It gives great results for forecasting problems [16]. To establish the model, only sufficient data is required to assimilate the connection between inputs and outputs. The main parameters of ANN model are the number of the input vectors, the number of layers and the number of neurons in each layer [32–34]. However, the large and sudden spikes in the input data will lead to less accuracy in the output using ANN. In this study, the back propagation (BP) algorithm is utilized to train the ANN model.
 (1)
Initialize the variables W_{ir},T_{r},V_{rj} and θ_{j} with small random values.
 (2)For each model pair (A^{(k)},C^{(k)}) (k=1,2,...,p), take the following steps.

Input the values of A^{(k)} at layer LA, then calculate b_{r} and c_{j} by Eqs. (19) and (20).

Calculate the bias d_{j} of the desired value and calculate the value c_{j} of the layer LC nodes and let$$ d_{j}=c_{j} \cdot (1c_{j}) \cdot \left(c_{j}^{(k)}c_{j}\right) $$(22)

Back propagate the errors to the layer LB nodes and let$$ e_{r}=b_{r} \cdot (1b_{r}) \cdot \left(\sum\limits_{j=1}^{n} V_{rj} \cdot d_{j} \right) $$(23)

Adjust the connection weights V_{rj} and the bias of the layer LC nodes θ_{j}:$$ V_{rj}=V_{rj}+\alpha \cdot d_{j}+\beta \cdot \Delta V_{rj}^{\prime} $$(24)$$ \theta_{j}=\theta_{j}+\alpha \cdot d_{j}+\beta \cdot \Delta \theta_{j}^{\prime} $$(25)
where \(\Delta V_{rj}^{\prime }\) and \(\theta _{j}^{\prime }\) are the adjusting values of the previous learning loop. α is the learning ratio and 0<α<1. β is the momentum factor.

Adjust the connection weights W_{ir} and the bias of the layer LB nodes T_{r}:$$ W_{ir}=W_{ir}+\alpha \cdot e_{r}+\beta \cdot \Delta W_{ir}^{\prime} $$(26)$$ T_{r}=T_{r}+\alpha \cdot e_{r}+\beta \cdot \Delta T_{r}^{\prime} $$(27)
where \(\Delta W_{ir}^{\prime }\) and \(\Delta T_{r}^{\prime }\) are the adjusting values of the previous learning loop.

 (3)
Repeat step (2), until d_{j} becomes adequately small.
In accordance with the analysis in previous sections, the ANN model is used on this stage to improve the accuracy of the RTP forecasting further in particular time slots, such as between 6:30  8:30 as shown in Fig. 4. In this case, 2 hidden layers with 20 and 40 neurons are designed and 10day historical data is adopted. In next section, a number of simulations are carried out to prove the effectiveness of the proposed hybrid model and the forecasting quality is also evaluated in terms of some evaluation criteria.
Results
This section demonstrates the realtime electricity prices forecasting results by using the proposed hybrid forecasting model. Limited data sets (5 days) of the historical RTP with a time interval of 0.5 h in Australia is adopted. The achieved results are also compared with the previous methods (e.g., ARIMA model, independent BPANN model, etc.) in this work.
RTP Forecasting quality evaluation comparison between models
Models  Evaluation criteria  

MAE ($/MWh)  MSE ($/MWh)  RMSE ($/MWh\()^{\frac {1}{2}}\)  MAPE (%)  
ARIMA model  2.61  9.25  3.04  8.29 
LS model  2.52  9.41  3.07  8.51 
GP model  1.35  3.01  1.74  4.29 
LS+GP models  1.53  5.33  2.31  4.65 
BPANN model  1.49  3.50  1.87  4.69 
Hybrid model  1.06  1.72  1.31  3.38 
Discussion
The hybrid model analyzes the input data in views of linear behavior, nonlinear behavior and errors optimization within the data. The advantage of our approach is that the hybrid model is more robust in dealing with forecasting tasks based on insufficient data compared with the traditional models such as ARIMA which needs a large number of historical data for training. The RTP forecasting quality evaluation results in Table 3 also indicate that the ARIMA model is not effective in the case with limited input data. In addition, the individual LS model did not perform well in this case, as the LS model only extracts the main stream within the input data. Therefore, using a LS model independently to forecast the RTP will lead to considerable errors as expected. It is much more interesting to see that the LS model cooperated with the GP model performs a bit worse than the independent GP model in overall evaluation. This is because the combined model (LS model + GP model) perform not well in a specific time period, i.e., 6:30 to 8:30 in this case, so that the errors improved significantly in overall, although it has higher forecasting accuracies in other time periods compared with the GP model.
However, given a group of historical data, there are several forecasting models can be used and each model may be able to complete the task of forecasting with different accuracies. After a great number of tests, we realize that the forecasting performance is crucially dependent on both selecting an appropriate model and the data correlations. A forecasting model works well to one group of data, but can be not effective for another group of data. Therefore, a hybrid forecasting model is generally more efficient than an independent model.
Conclusion
In this paper, a hybrid model consisting of LS model, GP model and BPANN model, is proposed to forecast the dayahead realtime prices based on limited historical data. The achieved forecasting performance evaluation results clearly demonstrate that our approach is an accurate and efficient tool in RTP forecasting and it also significantly outperforms the previous forecasting models. As RTP tariff is a trend for smart grid in next decade, the theories in this paper have bright prospects not only in RTP forecasting, but also in applications in other industrial fields, such load forecasting, wind forecasting, GDP forecasting, etc.
Notes
Declarations
Acknowledgement
This work is partially supported by the XJTLU Key Programme Special Fund (KSFP02).
Availability of data and materials
All data generated or analyzed during this study are included in this paper and reference list. The data is available from the corresponding author on reasonable request.
Authors’ contributions
XL proposed this topic, carried out numerical experiments, and drafted the manuscript. XZ and EGL checked and clarified the manuscript carefully. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
 Garcia EV, Runnels JE. The utility perspective of spot pricing. IEEE Power Eng Rev. 1985; PER5(6):45. https://doi.org/10.1109/MPER.1985.5526640.View ArticleGoogle Scholar
 Saele H, Grande OS. Demand response from household customers: Experiences from a pilot study in norway. IEEE Trans Smart Grid. 2011; 2(1):102–9. https://doi.org/10.1109/TSG.2010.2104165.View ArticleGoogle Scholar
 Vlachos AG, Biskas PN. Simultaneous clearing of energy and reserves in multiarea markets under mixed pricing rules. IEEE Trans Power Syst. 2011; 26(4):2460–71. https://doi.org/10.1109/TPWRS.2011.2126025.View ArticleGoogle Scholar
 Yu R, Yang W, Rahardja S. A statistical demandprice model with its application in optimal realtime price. IEEE Trans Smart Grid. 2012; 3(4):1734–42. https://doi.org/10.1109/TSG.2012.2217400.View ArticleGoogle Scholar
 Luo X, Zhu X, Lim EG. Load scheduling based on an advanced realtime price forecasting model. In: Proc. 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM). 2015. p. 1252–7. https://doi.org/10.1109/CIT/IUCC/DASC/PICOM.2015.186.
 Chang CS, Yi M. Realtime pricing related shortterm load forecasting. In: Proc. 1998 International Conference on Energy Management and Power Delivery. 1998. p. 411–4162. https://doi.org/10.1109/EMPD.1998.702588.
 Luo X, Zhu X, Lim EG. Dynamic pricing based and electric vehicle assisted demand response strategy. In: 2017 IEEE International Conference on Smart Grid Communications (SmartGridComm). 2017. p. 357–62. https://doi.org/10.1109/SmartGridComm.2017.8340691.
 Qian K, Zhou C, Allan M, Yuan Y. Modeling of load demand due to ev battery charging in distribution systems. IEEE Trans Power Syst. 2011; 26(2):802–10. https://doi.org/10.1109/TPWRS.2010.2057456.View ArticleGoogle Scholar
 O’Dwyer C, Duignan R, O’Malley M. Modeling demand response in the residential sector for the provision of reserves. In: 2012 IEEE Power and Energy Society General Meeting.2012. p. 1–8. https://doi.org/10.1109/PESGM.2012.6344757.
 Yao L, Lim WH, Tsai TS. A realtime charging scheme for demand response in electric vehicle parking station. IEEE Trans Smart Grid. 2017; 8(1):52–62. https://doi.org/10.1109/TSG.2016.2582749.View ArticleGoogle Scholar
 Tsui KM, Chan SC. Demand response optimization for smart home scheduling under realtime pricing. IEEE Trans Smart Grid. 2012; 3(4):1812–21. https://doi.org/10.1109/TSG.2012.2218835.View ArticleGoogle Scholar
 Mazengia DH, Tuan LA. Forecasting spot electricity market prices using time series models. In: 2008 IEEE International Conference on Sustainable Energy Technologies. 2008. p. 1256–61. https://doi.org/10.1109/ICSET.2008.4747199.
 Tahmasebifar R, SheikhElEslami MK, Kheirollahi R. Point and interval forecasting of realtime and dayahead electricity prices by a novel hybrid approach. IET Gener Transm Distrib. 2017; 11(9):2173–83. https://doi.org/10.1049/ietgtd.2016.1396.View ArticleGoogle Scholar
 Amjady N.Dayahead price forecasting of electricity markets by a new fuzzy neural network. IEEE Trans Power Syst. 2006; 21(2):887–96. https://doi.org/10.1109/TPWRS.2006.873409.MathSciNetView ArticleGoogle Scholar
 Fan S, Mao C, Chen L. Nextday electricityprice forecasting using a hybrid network. IET Gener Transm Distrib. 2007; 1(1):176–82. https://doi.org/10.1049/ietgtd:20060006.View ArticleGoogle Scholar
 Mandal P, Srivastava AK, Park JW. An effort to optimize similar days parameters for annbased electricity price forecasting. IEEE Trans Ind Appl. 2009; 45(5):1888–96. https://doi.org/10.1109/TIA.2009.2027542.View ArticleGoogle Scholar
 Contreras J, Espinola R, Nogales FJ, Conejo AJ. Arima models to predict nextday electricity prices. IEEE Trans Power Syst. 2003; 18(3):1014–20. https://doi.org/10.1109/TPWRS.2002.804943.View ArticleGoogle Scholar
 Garcia RC, Contreras J, van Akkeren M, Garcia JBC. A garch forecasting model to predict dayahead electricity prices. IEEE Trans Power Syst. 2005; 20(2):867–74. https://doi.org/10.1109/TPWRS.2005.846044.View ArticleGoogle Scholar
 Nair KR, Vanitha V, Jisma M. Forecasting of wind speed using ann, arima and hybrid models. In: 2017 International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT). 2017. p. 170–5. https://doi.org/10.1109/ICICICT1.2017.8342555.
 In: AEMOAustrilian Energy Market Operator. Available: http://www.aemo.org.au/. Accessed 25 Dec 2017.
 Miao Z, Fan L, Aghamolki HG, Zeng B. Least squares estimation based sdp cuts for socp relaxation of ac opf. IEEE Trans Autom Control. 2018; 63(1):241–8. https://doi.org/10.1109/TAC.2017.2719607.MathSciNetView ArticleGoogle Scholar
 Kesaniemi M, Virtanen K. Direct least square fitting of hyperellipsoids. IEEE Trans Pattern Anal Mach Intell. 2018; 40(1):63–76. https://doi.org/10.1109/TPAMI.2017.2658574.View ArticleGoogle Scholar
 Renczes B, Kollar I, Daboczi T. Efficient implementation of least squares sine fitting algorithms. IEEE Trans Instrum Meas. 2016; 65(12):2717–24. https://doi.org/10.1109/TIM.2016.2600998.View ArticleGoogle Scholar
 Guilin T, Yunming Q. Improved least square method apply in ship performance analysis. In: Proc. 2010 3rd International Conference on Advanced Computer Theory and Engineering (ICACTE). 2010. p. 5–5945596. https://doi.org/10.1109/ICACTE.2010.5579420.
 Haiping H, Shichao H, Jiutian C, Ruchuan W. Image hiding algorithm in discrete cosine transform domain based on grey prediction and grey relational analysis. China Commun. 2013; 10(7):57–70. https://doi.org/10.1109/CC.2013.6571289.View ArticleGoogle Scholar
 Yang SH, Huang WJ, Tsai JF, Chen YP. Symbiotic structure learning algorithm for feedforward neuralnetworkaided grey model and prediction applications. IEEE Access. 2017; 5:9378–88. https://doi.org/10.1109/ACCESS.2017.2702340.View ArticleGoogle Scholar
 Wang Y, Liu Q, Tang J, Cao W, Li X. Optimization approach of background value and initial item for improving prediction precision of gm(1,1) model. J Syst Eng Electron. 2014; 25(1):77–82. https://doi.org/10.1109/JSEE.2014.00009.View ArticleGoogle Scholar
 Lee JS, Lee YC. An application of grey prediction to transmission power control in mobile sensor networks. IEEE Internet Things J. 2018; 5(3):2154–62. https://doi.org/10.1109/JIOT.2018.2826008.View ArticleGoogle Scholar
 Hui P, Qu W, Tang J, Chen J. Traffic indexes prediction based on grey prediction model. In: Proc. 2013 Sixth International Symposium on Computational Intelligence and Design (ISCID). 2013. p. 244–7. https://doi.org/10.1109/ISCID.2013.68.
 Xu N, Zhang XR. Traffic volume prediction based on improved grey selfadaptable prediction formula. In: Proc. 2010 International Conference on Machine Learning and Cybernetics (ICMLC). 2010. p. 1027–30. https://doi.org/10.1109/ICMLC.2010.5580624.
 Khashei M, Bijari M. An artificial neural network (p,d,q) model for timeseries forecasting. Expert Syst Appl. 2010; 37(1):479–89. https://doi.org/10.1016/j.eswa.2009.05.044.View ArticleGoogle Scholar
 Zhang LD, Jia L, Zhu WX. Overview of traffic flow hybrid ann forecasting algorithm study. In: Proc. 2010 International Conference on Computer Application and System Modeling (ICCASM 2010). 2010. p. 1–6151619. https://doi.org/10.1109/ICCASM.2010.5620414.
 Zhao H, Peng L, Takahashi T, Hayashi T, Shimizu K, Yamamoto T. Ann based data integration for multipath ultrasonic flowmeter. IEEE Sensors J. 2014; 14(2):362–70. https://doi.org/10.1109/JSEN.2013.2282466.View ArticleGoogle Scholar
 Lian T, Xie M, Xu J, Chen L, Gao H. Modified bp neural network model is used for oddeven discrimination of integer number. In: Optoelectronics and Microelectronics (ICOM), 2013 International Conference On. 2013. p. 67–70. https://doi.org/10.1109/ICoOM.2013.6626492.
 Zhu D, Zhang T, Mao G. Backpropagation artificial neural networks for water supply pipeline model. Tsinghua Sci Technol. 2002; 7(5):527–31.Google Scholar
 Methaprayoon K, Yingvivatanapong C, Lee WJ, Liao JR. An integration of ann wind power estimation into unit commitment considering the forecasting uncertainty. IEEE Trans Ind Appl. 2007; 43(6):1441–8. https://doi.org/10.1109/TIA.2007.908203.View ArticleGoogle Scholar
 Zhao X, Wang S, Li T. Review of evaluation criteria and main methods of wind power forecasting. Energy Procedia. 2011; 12:761–9. https://doi.org/10.1016/j.egypro.2011.10.102. The Proceedings of International Conference on Smart Grid and Clean Energy Technologies.View ArticleGoogle Scholar
 Mehdiyev N, Enke D, Fettke P, Loos P. Evaluating forecasting methods by considering different accuracy measures. Procedia Comput Sci. 2016; 95:264–71. https://doi.org/10.1016/j.procs.2016.09.332. Complex Adaptive Systems Los Angeles, CA November 24, 2016.View ArticleGoogle Scholar
 Vihinen M. How to evaluate performance of prediction methods? measures and their interpretation in variation effect analysis. BMC Genomics. 2012; 13(4):2. https://doi.org/10.1186/1471216413S4S2.View ArticleGoogle Scholar