Forecasting Stock Market Volatility: A Wavelet-Enhanced Hybrid GARCH-Deep Learning Approach

Document Type : Original Article

Authors

1 School of Computer Engineering, Iran University of Science and Technology, Iran, Tehran.

2 School of Computer Engineering, Iran University of Science and Technology, Iran, Tehran

10.22091/jemsc.2026.13545.1291

Abstract

Accurate volatility forecasting is vital for effective decision-making in financial markets, yet remains a complex challenge due to the noisy, non-stationary nature of financial time series and the diversity of volatility definitions. This study proposes a novel hybrid framework that combines statistical, signal processing, and machine learning techniques to enhance volatility prediction. The approach begins with wavelet transformations to extract multi-scale features from raw financial data, effectively addressing non-stationarity. These features are then evaluated using multiple volatility estimators to determine their predictive relevance. The framework integrates GARCH models, wavelet-derived inputs, and deep learning architectures, with Particle Swarm Optimization (PSO) employed for optimal parameter tuning. Leveraging S&P 500 data from 2000 to 2024 and incorporating multi-source inputs, the model achieves a more holistic representation of market dynamics. Empirical results demonstrate that the hybrid method significantly reduces prediction errors and consistently outperforms baseline models and established benchmarks. To validate its practical utility, we developed trading strategies based on the predicted volatility. Backtesting results show substantial performance gains.

Keywords

Main Subjects

References

Amirshahi, B., & Lahmiri, S. (2023). Hybrid deep learning and GARCH-family models for forecasting volatility of cryptocurrencies. Machine Learning with Applications, 12(1), 100465. https://doi.org/10.1016/j.mlwa.2023.100465
Andrés García-Medina, & Aguayo-Moreno, E. (2023). LSTM–GARCH hybrid model for the prediction of volatility in cryptocurrency portfolios. Computational Economics, 1(1). https://doi.org/10.1007/s10614-023-10373-8
Baillie, R. T., Bollerslev, T., & Mikkelsen, H. O. (1996). Fractionally integrated generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 74(1), 3-30. https://doi.org/10.1016/S0304-4076(95)01749-6
Bate, A., Lindquist, M., Edwards, I. R., Olsson, S., Orre, R., Lansner, A., & De Freitas, R. M. (1998). A Bayesian neural network method for adverse drug reaction signal generation. European Journal of Clinical Pharmacology, 54, 315-321. https://doi.org/10.1007/s002280050466
Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307-327. https://doi.org/10.1016/0304-4076(86)90063-1
Bollerslev, T., Chou, R. Y., & Kroner, K. F. (1992). ARCH modeling in finance: A review of the theory and empirical evidence. Journal of Econometrics, 52(1-2), 5-59. https://doi.org/10.1016/0304-4076(92)90064-X
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. https://doi.org/10.48550/arXiv.1412.3555
Ciner, C. (2025). Forecasting the aggregate market volatility by boosted neural networks. Finance Research Letters, 72(1), 106505. https://doi.org/10.1016/j.frl.2024.106505
Conrad, C., & Engle, R. F. (2025). Modelling volatility cycles: The MF2‐GARCH model. Journal of Applied Econometrics, 40(4), 438–454. https://doi.org/10.1002/jae.3118
Dessie, E., Birhane, T., Mohammed, A., & Walelign, A. (2025). A hybrid GARCH, convolutional neural network and long short term memory methods for volatility prediction in stock market. Journal of Combinatorial Mathematics and Combinatorial Computing, 124(461), 476. https://doi.org/10.61091/jcmcc124-30
Di Persio, L., Garbelli, M., Mottaghi, F., & Wallbaum, K. (2023). Volatility forecasting with hybrid neural networks methods for risk parity investment strategies. Expert Systems with Applications, 229(1), 120418. https://doi.org/10.1016/j.eswa.2023.120418
Di Persio, L., Garbelli, M., Mottaghi, F., & Wallbaum, K. (2023). Volatility forecasting with hybrid neural networks methods for Risk Parity investment strategies. Expert Systems with Applications, 229(1), 120418. https://doi.org/10.1016/j.eswa.2023.120418
Engle, R. F. (1982). Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica: Journal of the Econometric Society, 987-1007. https://doi.org/10.2307/1912773
Fałdziński, M., & Osińska, M. (2016). Volatility estimators in econometric analysis of risk transfer on capital markets. Dynamic Econometric Models, 16, 21-35. https://doi.org/10.12775/DEM.2016.002
Garman, M. B., & Klass, M. J. (1980). On the estimation of security price volatilities from historical data. Journal of Business, 67-78. https://doi.org/10.1086/296072
Gong, X., & Lin, B. (2018). Modeling stock market volatility using new HAR-type models. Physica a Statistical Mechanics and Its Applications, 516(1), 194–211. https://doi.org/10.1016/j.physa.2018.10.013
Graves, A., Fernández, S., & Schmidhuber, J. (2005, September). Bidirectional LSTM networks for improved phoneme classification and recognition. In International Conference on Artificial Neural Networks (pp. 799-804). Berlin, Heidelberg. https://doi.org/10.1007/11550907_126
Hajizadeh, E., Seifi, A., Fazel Zarandi, M. H., & Turksen, I. B. (2012). A hybrid modeling approach for forecasting the volatility of S&P 500 index return. Expert Systems with Applications, 39(1), 431–436. https://doi.org/10.1016/j.eswa.2011.07.033
Hamid, S. A., & Iqbal, Z. (2004). Using neural networks for forecasting volatility of S&P 500 Index futures prices. Journal of Business Research, 57(10), 1116–1125. https://doi.org/10.1016/s0148-2963(03)00043-2
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hu, Y., Ni, J., & Wen, L. (2020). A hybrid deep learning approach by integrating LSTM-ANN networks with GARCH model for copper price volatility prediction. Physica A: Statistical Mechanics and Its Applications, 557(1), 124907. https://doi.org/10.1016/j.physa.2020.124907
Jaeger, H. (2007). Echo state network. Scholarpedia, 2(9), 2330. https://doi.org/10.4249/scholarpedia.2330
Kim, J. M., Jun, C., & Lee, J. (2021). Forecasting the volatility of the cryptocurrency market by GARCH and Stochastic Volatility. Mathematics, 9(14), 1614. https://doi.org/10.3390/math9141614
Kim, H. Y., & Won, C. H. (2018). Forecasting the volatility of stock price index: A hybrid model integrating LSTM with multiple GARCH-type models. Expert Systems with Applications, 103(1), 25–37. https://doi.org/10.1016/j.eswa.2018.03.002
Koo, E., & Kim, G. (2022). A hybrid prediction model integrating garch models with a distribution manipulation strategy based on lstm networks for stock market volatility. IEEE Access, 10(1), 34743-34754. https://doi.org/10.1109/ACCESS.2022.3163723
Kristjanpoller R, W., & Hernández P, E. (2017). Volatility of main metals forecasted by a hybrid ANN-GARCH model with regressors. Expert Systems with Applications, 84(1), 290–300. https://doi.org/10.1016/j.eswa.2017.05.024
Kristjanpoller, W., & Minutolo, M. C. (2015). Gold price volatility: A forecasting approach using the Artificial Neural Network–GARCH model. Expert Systems with Applications, 42(20), 7245–7251. https://doi.org/10.1016/j.eswa.2015.04.058
Kristjanpoller, W., & Minutolo, M. C. (2016). Forecasting volatility of oil price using an artificial neural network-GARCH model. Expert Systems with Applications, 65(1), 233–241. https://doi.org/10.1016/j.eswa.2016.08.045
Kristjanpoller, W., Fadic, A., & Minutolo, M. C. (2014). Volatility forecast using hybrid Neural Network models. Expert Systems with Applications, 41(5), 2437–2442. https://doi.org/10.1016/j.eswa.2013.09.043
Kumar, S., Rao, A., & Dhochak, M. (2025). Hybrid ML models for volatility prediction in financial risk management. International Review of Economics & Finance, 98, 103915. https://doi.org/10.1016/j.iref.2025.103915
Laurent, S. (2004). Analytical derivates of the APARCH model. Computational Economics, 24(1), 51-57. https://doi.org/10.1023/B:CSEM.0000038851.72226.76
Li, Y., Liu, G., & Zhang, Z. (2022). Volatility of volatility: Estimation and tests based on noisy high frequency data with jumps. Journal of Econometrics, 229(2), 422–451. https://doi.org/10.1016/j.jeconom.2021.02.007
Liu, Y. (2019). Novel volatility forecasting using deep learning - Long short term memory recurrent neural networks. Expert Systems with Applications, 1(1). https://doi.org/10.1016/j.eswa.2019.04.038
Lu, X., Que, D., & Cao, G. (2016). Volatility forecast based on the hybrid artificial Neural Network and GARCH-type models. Procedia Computer Science, 91(1), 1044–1049. https://doi.org/10.1016/j.procs.2016.07.145
Mademlis, D. K., & Dritsakis, N. (2021). Volatility forecasting using hybrid GARCH neural network models: The case of the Italian stock market. International Journal of Economics and Financial Issues, 11(1), 49. https://doi.org/10.32479/ijefi.10842
Mademlis, D. K., & Dritsakis, N. (2021). Volatility forecasting using hybrid GARCH neural network models: The case of the Italian stock market. International Journal of Economics and Financial Issues, 11(1), 49–60. https://doi.org/10.32479/ijefi.10842
Mahajan, V., Thakan, S., & Malik, A. (2022). Modeling and forecasting the volatility of NIFTY 50 using GARCH and RNN models. Economies, 10(5), 102. https://doi.org/10.3390/economies10050102
Manogna, R. L., Dharmaji, V., & Sarang, S. (2025). A novel hybrid neural network-based volatility forecasting of agricultural commodity prices: Empirical evidence from India. Journal of Big Data, 12(1), 85. https://doi.org/10.1186/s40537-025-01131-8
Miura, R., Lukáš Pichl, & Taisei Kaizoji. (2019). Artificial neural networks for realized volatility prediction in cryptocurrency time series. Lecture Notes in Computer Science, 1(1), 165–172. https://doi.org/10.1007/978-3-030-22796-8_18
Namdari Birgani, S., Sedighi, A. H., & Molaalizadeh Zavardehi, S. (2024). Portfolio optimization using deep reinforcement learning. Engineering Management and Software Computing, 10(2), 1–22. https://doi.org/10.22091/jemsc.2025.11158.1192
Nelson, D. B. (1991). Conditional heteroskedasticity in asset returns: A new approach. Econometrica: Journal of the Econometric Society, 347-370. https://doi.org/10.2307/2938260
Olubusola, O., Mhlongo, N. Z., Daraojimba, D. O., Ajayi-Nifise, A. O., & Falaiye, T. (2024). Machine learning in financial forecasting: A US review: Exploring the advancements, challenges, and implications of AI-driven predictions in financial markets. World Journal of Advanced Research and Reviews, 21(2), 1969-1984. https://doi.org/10.30574/wjarr.2024.21.2.0444
Parkinson, M. (1980). The extreme value method for estimating the variance of the rate of return. Journal of Business, 61-65. https://www.jstor.org/stable/2352357
Pierdzioch, C., Risse, M., & Rohloff, S. (2016). A boosting approach to forecasting the volatility of gold-price fluctuations under flexible loss. Resources Policy, 47(1), 95–107. https://doi.org/10.1016/j.resourpol.2016.01.003
Poon, S. H., & Granger, C. W. J. (2003). Forecasting volatility in financial markets: A review. Journal of Economic Literature, 41(2), 478-539. https://doi.org/10.1257/002205103765762743
Ramos-Pérez, E., Alonso-González, P. J., & Núñez-Velázquez, J. J. (2019). Forecasting volatility with a stacked model based on a hybridized Artificial Neural Network. Expert Systems with Applications, 129(1), 1–9. https://doi.org/10.1016/j.eswa.2019.03.046
Rogers, L. C. G., & Satchell, S. E. (1991). Estimating variance from high, low and closing prices. The Annals of Applied Probability, 504-512. https://doi.org/10.1214/aoap/1177005835
Rubio, L., Adriana Palacio P., Adriana Mejía C., & Ramos, F. (2023). Forecasting volatility by using wavelet transform, ARIMA and GARCH models. Eurasian Economic Review, 13(3-4), 803–830. https://doi.org/10.1007/s40822-023-00243-x
Rubio, L., Palacio Pinedo, A., Mejía Castaño, A., & Ramos, F. (2023). Forecasting volatility by using wavelet transform, ARIMA and GARCH models. Eurasian Economic Review, 13(3), 803-830. https://doi.org/10.1007/s40822-023-00243-x
Sardelich, M., & Manandhar, S. (2018, December 25). Multimodal deep learning for short-term stock volatility prediction. ArXiv.org. https://doi.org/10.48550/arXiv.1812.10479
Schwert, G. W. (1990). Stock market volatility. Financial Analysts Journal, 46(3), 23-34. https://doi.org/10.2469/faj.v46.n3.23
Seo, M., & Kim, G. (2020). Hybrid forecasting models based on the Neural Networks for the volatility of bitcoin. Applied Sciences, 10(14), 4768. https://doi.org/10.3390/app10144768
Shen, Z., Wan, Q., & Leatham, D. J. (2021). Bitcoin return volatility forecasting: A comparative study between GARCH and RNN. Journal of Risk and Financial Management, 14(7), 337. https://doi.org/10.3390/jrfm14070337
Siddiraju, N., & Hasan, R. (2025). A Hybrid LSTM-GARCH Network for stock market volatility prediction. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC), 1(1), 00427–00433. https://doi.org/10.1109/ccwc62904.2025.10903807
Stoll, H. R., & Whaley, R. E. (1990). Stock market structure and volatility. The Review of Financial Studies, 3(1), 37-71. https://doi.org/10.1093/rfs/3.1.37
Trierweiler Ribeiro, G., Alves Portela Santos, A., Cocco Mariani, V., & dos Santos Coelho, L. (2021). Novel hybrid model based on echo state neural network applied to the prediction of stock price return volatility. Expert Systems with Applications, 184(1), 115490. https://doi.org/10.1016/j.eswa.2021.115490
Ulu, Y. (2025). Forecasting stock volatility via hybrid deep learning and GARCH family models: A case study from BIST30. Journal of Applied Mathematics and Computation, 9(4). http://dx.doi.org/10.26855/jamc.2024.12.001
Yang, D., & Zhang, Q. (2000). Drift‐independent volatility estimation based on high, low, open, and close prices. The Journal of Business, 73(3), 477-492. https://doi.org/10.1086/209650
Yu, S., & Li, Z. (2018). Forecasting stock price index volatility with LSTM deep Neural Network. Recent Developments in Data Science and Business Analytics, 1(1), 265–272. https://doi.org/10.1007/978-3-319-72745-5_29
Zhang, C., Zhang, Y., Cucuringu, M., & Qian, Z. (2023). Volatility forecasting with machine learning and intraday commonality. Journal of Financial Econometrics, 1(1). https://doi.org/10.1093/jjfinec/nbad005
Zhang, G., Patuwo, B. E., & Hu, M. Y. (1998). Forecasting with artificial neural networks: The state of the art. International Journal of Forecasting, 14(1), 35-62. https://doi.org/10.1016/S0169-2070(97)00044-7
Zhao, P., Zhu, H., Siu, W., & Lee, D. L. (2024). From GARCH to neural network for volatility forecast. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15), 16998–17006. https://doi.org/10.1609/aaai.v38i15.29643
Send comment about this article
CAPTCHA Image

Files

Share

How to cite

Statistics