Machine learning methods to improve spatial predictions of coastal wind speed profiles and low-level jets using single-level ERA5 data

Observations of the wind speed at heights relevant for wind power are sparse, especially offshore, but with emerging aid from advanced statistical methods, it may be possible to derive information regarding wind profiles using surface observations. In this study, two machine learning (ML) methods are developed for predictions of (1) coastal wind speed profiles and (2) low-level jets (LLJs) at three locations of high relevance to offshore wind energy deployment: the US Northeastern Atlantic Coastal Zone, the North Sea, and the Baltic Sea. The ML models are trained on multiple years of lidar profiles and utilize single-level ERA5 variables as input. The models output spatial predictions of coastal wind speed profiles and LLJ occurrence. A suite of nine ERA5 variables are considered for use in the study due to their physics-based relevance in coastal wind speed profile genesis and the possibility to observe these variables in real-time via measurements. The wind speed at 10 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$ and the surface sensible heat flux are shown to have the highest importance for both wind speed profile and LLJ predictions. Wind speed profile predictions output by the ML models exhibit similar root mean squared error (RMSE) with respect to observations as is found for ERA5 output. At typical hub heights, the ML models show lower RMSE than ERA5 indicating approximately 5 % RMSE reduction. LLJ identification scores are evaluated using the symmetric extremal dependence index (SEDI). LLJ predictions from the ML models outperform predictions from ERA5, demonstrating markedly higher SEDIs. However, optimization utilizing the SEDI results in a higher number of false alarms when compared to ERA5.

1 Introduction

Wind energy is one of the fastest growing sources of renewable energy worldwide, representing 23 % of the electricity generation from renewable sources in 2021, and each year there is an increase in the installed capacity of wind power by approximately 10⁹ kWh (EIA, 2023). Motivated by the occurrence of higher and less variable wind speeds over large bodies of water, the offshore wind energy industry is a rapidly developing and growing industry. As of 2021, 17 GW of offshore wind energy capacity was under construction globally, with the United States and Europe implementing ambitious offshore wind energy goals (WFO, 2022; IRENA, 2022). The United States and Europe plan to install 30 and 60 GW of offshore wind energy by 2030, respectively (American Clean Power Association, 2020; European Commission, 2020; Barthelmie et al., 2021). Along the US East Coast, 16 lease areas for offshore wind energy projects are in various stages of development. Recent research (Pryor et al., 2021) has suggested that the 15 northernmost lease areas could provide nearly 3 % of the national electricity demand by deploying 1922 15 MW wind turbines with a 1.85 km spacing and that offshore installments in the area are highly competitive in terms of estimated levelized cost of energy (LCoE, Foody et al., 2024). In Europe, the North Sea and the Baltic Sea are key sites for future and current offshore wind development. In 2019, Europe's installed offshore wind energy capacity was nearly 22 GW, and the North Sea held approximately 77 % of this (Ramirez et al., 2020). In the Baltic Sea region, approximately 2.2 GW of offshore wind power capacity was installed as of 2020, but the region is projected to produce 93 GW of offshore wind power by 2050 (COWI, 2019; Wind Europe, 2021).

Accurate measurements and predictions of the offshore wind resource are complex, especially in the coastal zone, with its sharp transition between surfaces with different properties, i.e., roughness length and specific heat capacity (see, e.g., Sempreviva et al., 2008; Meyer and Gottschall, 2022). For example, during spring and early summer, when air heated over the land surface is advected over open water that is still cold after the winter, turbulence is suppressed and a stable marine boundary layer can be formed. As a result of the decrease in turbulent resistance to the air flow, the pressure gradient force becomes unbalanced, resulting in an increase of the wind speed. This process, known as frictional decoupling, can create a local maximum in the wind speed profile in the lowest hundreds of meters in the atmosphere (Smedman et al., 1993; Debnath et al., 2021; Hallgren et al., 2022; also compare Luiz and Fiedler, 2023). Also, differences in the specific heat capacity of land and sea surfaces can create diurnal wind patterns known as the sea–land breeze (e.g., Miller et al., 2003; Hallgren et al., 2023b), affecting both the vertical profiles of wind speed and wind direction. Further, coastline complexity such as capes or inlets and near-shore topography (Burk and Thompson, 1996; Barthelmie et al., 1996; Talbot et al., 2007) affects the wind profile locally, and on the larger scale, synoptic baroclinicity and (cold) front passages with strong vertical temperature gradients regularly cause strong low-level flows (Kotroni and Lagouvardos, 1993; Amador, 2008). Finally, swell conditions can result in a positive momentum flux, increasing the wind speed in the lower part of the profile and resulting in wind profiles with negative shear at heights relevant for wind power (Högström et al., 2009; Semedo et al., 2009; Smedman et al., 2009; Hallgren et al., 2022). Adding to these multi-scale variabilities, atmospheric stability conditions offshore may frequently invalidate Monin–Obukhov similarity theory assumptions (Mahrt, 1998; Newman and Klein, 2014), particularly at heights relevant for wind energy deployment, increasing the complexity and difficulty of predicting wind speed profiles offshore.

Low-level jets (LLJs) are special cases of complex, non-ideal (i.e., non-logarithmic) wind speed profiles and are of high relevance to the wind energy industry due to their effect on structural and aerodynamic loading, power production, and wake recovery (Gutierrez et al., 2019; Gadde and Stevens, 2021; Gadde et al., 2021). An LLJ is generally described as a wind speed maximum that forms within the atmospheric boundary layer and is associated with altered rotor plane distributions of turbulence and shear (Pichugina et al., 2017; Aird et al., 2021). LLJs can occur offshore due to a variety of factors that manifest from local to synoptic scales and have been investigated offshore in numerous studies using both measurements and models (Ranjha et al., 2013; Nunalee and Basu, 2014; Soares et al., 2020). Offshore, the frequency of LLJs varies seasonally and LLJs are most common when a stable marine boundary layer forms under steep land–sea temperature gradients causing frictional decoupling (Angevine et al., 2006). Further, the low-level wind maximum occurs with strong gradients, having positive wind shear below the core and negative shear above the core, that possibly could prevent larger atmospheric eddies from propagating downward toward the surface – a phenomenon known as shear sheltering (Smedman et al., 2004; Prabha et al., 2008; Hallgren et al., 2022). This introduces increases in the variance in the horizontal velocity field as vertical variance is diminished and could possibly result in further reductions in the turbulent flux below the core.

Studies of offshore LLJs have generally indicated a peak in frequency during spring or summer, likely attributed to the advection of warm air over comparably cooler coastal waters. For example, a study of LLJs using 2 years of output from the Weather Research and Forecasting (WRF) model for the US NE Atlantic coast found a pronounced peak in LLJ seasonality in June (Aird et al., 2022). Further, the study showed a significant association with low boundary-layer heights, pronounced spring– summer horizontal land–sea temperature gradients, and LLJ occurrence. A second study of an LLJ event in the coastal New York Bight region (Colle and Novak, 2010) utilized high-resolution light detection and ranging (lidar) observations and simulations from the WRF model to conclude that LLJ occurrence in the region is dependent on diurnal heating. A study of LLJs over the North Sea utilizing a combination of reanalysis data and observations from met masts and lidar also found a pronounced peak in LLJ frequency in the spring to early summer months (Kalverla et al., 2019). Similarly, a study utilizing four reanalyses and lidar observations over the Baltic Sea found further evidence of peak LLJ occurrence in the early spring and summer months (Hallgren et al., 2020). These findings present compelling initial evidence of unique weather conditions associated with LLJ occurrence.

As offshore wind turbine dimensions increase markedly each year – with most recent offshore wind turbine models such as the 12 MW GE Haliade-X (, ) with a hub height of 150 m and a rotor diameter of 218 m and the International Energy Agency (IEA) 15 MW offshore reference turbine (Gaertner et al., 2020) with a hub height of 150 m and a rotor diameter of 240 m – more frequent interaction between LLJs and the rotor plane is likely and LLJs have been found to occur with jet core heights and speeds relevant to wind energy over the North Sea, Baltic Sea, and US Northeastern Atlantic Coastal Zone (see, e.g., Smedman et al., 1996; Kalverla et al., 2019; Hallgren et al., 2020; Aird et al., 2022). Further, a study of LLJs in the North Sea (Duncan, 2018) concluded that LLJs occur with a high degree of spatial coherence – e.g., if conditions are favorable for LLJs at one location within a wind farm, the probability of LLJ occurrence throughout the entire farm may be high – which results in implications for predicting LLJ conditions across entire wind farms in real time.

Although LLJs have been observed and simulated frequently offshore at heights relevant to wind energy, numerical weather prediction (NWP) models exhibit difficulty in resolving LLJ characteristics with high accuracy, i.e., in terms of timing and morphology (jet core height and speed) of LLJs. Kalverla et al. (2020), Hallgren et al. (2020), and Sheridan et al. (2023) all showed that regional models, optimized for a specific region and with higher horizontal resolution than the global models, are better in resolving coastal LLJs in the North Sea, the Baltic Sea, and off the California coast, respectively. However, not only the horizontal resolution is crucial, but also how calculations in the boundary layer are treated by the models (i.e., the planetary boundary layer scheme). When it comes to the average wind conditions in the profile, it seems to be less of a difference between state-of-the-art models, as long as the horizontal and vertical resolutions are good enough (Hallgren et al., 2020), even if the model performance varies with atmospheric stability (Kalverla et al., 2020).

NWP models offer higher spatial resolution than observational data but are computationally expensive to run and often only cover limited geographic regions (Holt, 1996; Gevorgyan, 2018; Jiménez-Sánchez et al., 2019). Global reanalyses can provide wind speed data at sufficiently high vertical resolutions for wind power, combining past observations with modern weather forecasting models to produce meteorological variables at consistent temporal and spatial resolutions (Kaiser-Weiss et al., 2019; Hersbach et al., 2020). However, due to limitations in computing resources or model structure, reanalysis models may not output wind speed profiles at heights relevant to wind energy or with sufficient vertical resolution for wind energy applications. While measurements in the field, i.e., from lidars, may be useful to capture wind speeds at higher vertical resolutions than those from NWP or reanalysis models, deployment can be costly and requires frequent monitoring to ensure the lidars are operating properly and gathering the desired data. Usually lidars are only available during a limited period of time and at a few selected sites. A solution to this could be to utilize continuous surface observations, at proposed or existing wind farm sites, to predict wind speed profiles.

Motivated by the previous considerations, this paper investigates the development of machine learning (ML) models to predict coastal wind speed profiles and LLJ occurrence from single-level meteorological variables. Recent studies (Vassallo et al., 2020; Bodini et al., 2023) have demonstrated promise in utilizing ML methods to vertically extrapolate wind speed profiles from lower-level measurements, as compared to traditional methods of extrapolation such as Monin–Obukhov similarity theory or the power law. Vassallo et al. (2020) applied artificial neural networks (ANNs) on a set of 11 meteorological variables and extrapolated the wind profile at three sites in different type of terrain, including a coastal site, where lidar wind profile data were available from measurement campaigns. The results were compared to standard extrapolation methods, the logarithmic wind law and the power law, and showed promising results with 65 % versus 52 % increase in extrapolation accuracy, respectively. Bodini et al. (2023) utilized surface-layer floating buoy observations and lidar measurements and used ML methods to extrapolate coastal wind speed profiles and quantify the long-term uncertainty of offshore model output from the WRF model on the US Atlantic coast.

Figure 1 Overview of the workflow for the ML WSPP regression and LLJ classification tasks.

Table 1 Lidar campaign information.

^a 7 October 2016–30 September 2021. ^b 1 October 2021–31 December 2021. ^c Before downsampling. ^d As utilized in this study. ^e Hours with wind speed profiles. ^f Hours with an LLJ, crit. 1

In this study, we aim to explore the feasibility and accuracy of using surface-layer measurements to spatially predict wind speed profiles, with a focus on non-ideal wind speed profiles that may not be adequately described by Monin–Obukhov similarity theory, i.e., LLJs. The paper utilizes a combination of multiple years of the European Centre for Medium-Range Weather Forecasts (ECMWF) fifth-generation reanalysis (ERA5) and lidar observations at three offshore/coastal sites in the Baltic Sea, the North Sea, and the US Northeastern Atlantic Coastal Zone. Two ML models, the random forest (RF) and the neural network (NN), are trained and developed to predict (1) wind speed profiles and (2) LLJ occurrence utilizing single-level variables output by ERA5. The two ML models are trained independently to verify consistency in results. The use of single-level meteorological variables for wind speed profile prediction (WSPP) and LLJ prediction is motivated by the potential utilization of observational meteorological data, e.g., wind speed or temperature measurements from meteorological masts deployed in wind farms, for real-time wind forecasting at wind farms.

The paper is structured as follows. A description of utilized data sets, the general workflow, and the developed ML models are presented in Sect. 2 along with the definition used to identify LLJs and the metrics to analyze the performance of the models. In Sect. 3, the results are presented, followed by a discussion in Sect. 4. Finally, a summary of the study and some concluding remarks are provided in Sect. 5.

2 Materials and methods

2.1 Overview of workflow

Two types of supervised ML methods were investigated herein: random forest (RF) and neural networks (NNs). The models were trained for each site individually, using single-level meteorological variables from ERA5 as the predictors and data from the lidar profiles as the predictand (see workflow in Fig. 1). Both ML methods were evaluated for their fidelity in two tasks: (1) predicting the coastal wind speed profile (regression task) and (2) predicting the LLJ (classification task) at each of the three sites investigated in this study. Note that the predictions were performed in space, using ERA5 single-level data to predict the speed and shape of the wind profile up to approximately 300 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$ – not in time.

The RF and the NN were evaluated separately and their performance for each task compared. Although the two models differ in complexity – with the NN being more statistically complex and requiring more computational time to train and output predictions than the RF model – the two models were primarily developed separately to verify that results for the predictors with the highest importance are replicable and not just attributed to random chance or the probabilistic learning sequence of ML models.

Figure 2 Maps of each of the three offshore sites investigated in this study – ASIT (US Northeastern Coastal Zone), MMIJ (North Sea), and Utö (Baltic Sea). Lidar locations are marked with circles.

2.2 Offshore lidar measurements

The ML models were trained and tested using multiple years of lidar profiles collected at three sites in areas of high relevance to offshore wind energy in the United States and Europe (Fig. 2, Table 1): the Woods Hole Oceanographic Institute (WHOI) Air–Sea Interaction Tower (ASIT) in the US Northeastern Coastal Zone south of Martha's Vineyard, eastern Massachusetts, at the Meteorological Mast IJmuiden (MMIJ) in the Dutch region of the North Sea, and in the Baltic Sea, on the small island of Utö in the outer part of the Finnish archipelago. The sites were selected due to the availability of high-quality multi-year lidar observations and to analyze whether the results hold across different offshore sites across the globe.

Each lidar data set was down-sampled to hourly resolution to maintain consistency between the frequency of ERA5 output and the lidar data. Note however that data representative of a grid box from ERA5 are compared to site measurements in the study. Each site demonstrated a moderate frequency of LLJ events (defined as in Sect. 2.6, crit. 1), ranging from 2 % to 10 %, throughout the data collection period. In the analysis, the lidar wind speed profiles were taken as the ground truth of the actual conditions at the site, and the uncertainty inherent in the measurements was not taken into account.

2.2.1 ASIT

The WHOI ASIT platform was installed in 2002. It is a fixed platform located 3 km south of Martha's Vineyard (Fig. 2) at a site where the water depth is 17 m (Kirincich, 2020). The ASIT is exposed to open wind and wave conditions and is located approximately 10 nmi from US offshore wind energy lease areas near Rhode Island and Massachusetts. The wind speed measurements used herein were performed on the main platform at 13 m above sea level ($\mathrm{a}.\mathrm{s}.\mathrm{l}.$) and were collected from a pulsed Leosphere WindCube vertically profiling lidar that was in operation 7 October 2016–30 September 2021 and then replaced by a Zephir ZX300 continuous-wave vertical scanning lidar, measuring at the same heights (51, 60, 80, 90, 100, 110, 120, 140, 160, 180, and 200 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$). The two lidars were run in parallel 1 September 2021–7 October 2021 and were shown to be in good agreement, assuring consistency in the data set. Previous analyses of these have indicated a high prevalence of low-turbulence conditions (Bodini et al., 2020).

2.2.2 MMIJ

The Meteomast IJmuiden (MMIJ, Maureira Poveda and Wouters, 2015) is an offshore measurement platform located approximately 85 km off of the Dutch coast, where the water depth is approximately 28 m. Wind speed measurements utilized herein were collected from a Zephir ZX300 continuous-wave vertically profiling lidar that was deployed November 2011–March 2016. Post-processing was conducted by the Energy Centre of the Netherlands, and wind speed data are provided as 10 min averages (Werkhoven and Verhoef, 2012). Lidar measurements have been validated, having a mean bias of 1 %, using data from the cup anemometers located on the mast at 90 m. Data from the MMIJ wind lidar have been used extensively in the analysis of the North Sea LLJ, and for an in-depth analysis of temporal occurrence and LLJ morphology we refer to Kalverla et al. (2017, 2019, 2020). Measurement heights of the MMIJ lidar are 90, 115, 140, 165, 190, 215, 240, 265, 290, and 315 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$

2.2.3 Utö

The island of Utö is located at the southern edge of Finnish archipelago in the Baltic Sea, approximately 60 km off of the Finnish mainland. The mean depth of the Archipelago Sea north of Utö is 19 m, and to the south the Baltic Proper it is deeper. The Utö Atmospheric and Marine Research Station (Hirsikko et al., 2014; Laakso et al., 2018) hosts a number of measurements including a Halo Photonics Stream Line scanning Doppler lidar. The Halo lidar laser and amplifier were upgraded to the XR version in October 2017. Here, we utilized horizontal winds retrieved from 15° elevation angle conical, i.e., vertical azimuth display (VAD), scans at every 15 min. A threshold of the signal-to-noise ratio of 0.005 was applied to radial measurements post-processed according to Vakkari et al. (2019) before wind retrieval following Browning and Wexler (1968); the retrieved wind speed agrees well with local anemometer (Tuononen et al., 2017). Range resolution of the lidar is 30 m, and the three lowest range gates were discarded due to effects by the outgoing pulse. With the lidar located at 8 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$, this results in 7.8 m vertical resolution from 35 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$ up.

2.2.4 Quality control

Quality control was performed by all data providers. In addition to that, inspection of lidar profiles was done to exclude those with more than 75 % of data points missing within a vertical profile and sizable gaps in the profile data (only wind speed profiles with at least five consecutive data points in a row are included in the analysis). A few profiles were removed after manual inspection, identifying malfunction of the device.

2.3 ERA5

The ECMWF ERA5 reanalysis is a source for global atmosphere, land surface, and ocean wave conditions from 1940 onward (Hersbach et al., 2020). ERA5 reanalysis uses the Integrated Forecasting System (IFS) Cy41r2 and replaces the former reanalysis from ECMWF, the ERA-Interim. The ERA5 horizontal resolution is 0.25° × 025°, and meteorological variables are output hourly. In this study, only data from the grid point closest to the location of the lidar were used, resulting in distances of 10 km for ASIT, 12 km for MMIJ, and 7.7 km for Utö. The values of the ERA5 land–sea mask (ranging from 0 for open sea to 1 for land) for the selected grid points were 0.04 for ASIT, 0 for MMIJ, and 0.01 for Utö. While only single-level variables from ERA5 are used for the ML models, the ERA5 wind profile was also assessed using the wind components on terrain-following hybrid-sigma model level. To calculate the height of the these levels, the surface pressure and temperature and humidity data from the model levels were downloaded.

ERA5 is utilized widely for wind energy applications due to its consistency across the globe and has been extensively validated for use in wind energy contexts (e.g., Olauson, 2018; Hallgren et al., 2020; Soares et al., 2020; Hayes et al., 2021; Pryor and Barthelmie, 2021; Gualtieri, 2022). ERA5 temperature, wind, and humidity profiles have been validated through comparison with radiosonde data, and ocean wave height has been validated through comparison with buoy wave data (Soares et al., 2020).

Based on the long list of single-level variables provided by ERA5, a selection of nine variables, all with Pearson correlation coefficients in the range −0.5 to +0.5, was performed. The selection was based on the plausible relevance of the variable for LLJ occurrence or coastal wind speed profile forecasting (thus, the variables are physics-based) and the potential for measurability in the field, i.e., the potential to use the predictors in dynamic real-time wind speed profile prediction. The low cross-correlation assures a high degree of independency among variables which in turn simplifies the analysis of how important the individual variables are in terms predicting the wind speed profile. The selected variables are the 10 m wind speed (ws10), the 10 m wind direction (wdir10), the sea surface temperature (SST), the mean sea level pressure (MSLP), the total precipitation (precip.), the convective available potential energy (CAPE), the surface sensible heat flux (SHF), the surface net radiation (Rn), and the low cloud cover (LCC). Although difficult to measure using only surface observations, CAPE was one of the variables included, to allow for an analysis of whether the available energy in the atmosphere aloft has a strong impact on the wind speed profile. The variables and the physics-based motivation for possible importance for wind profile prediction are presented in Table S1 in the Supplement.

To allow for a comparison of the ERA5 wind speed profile to the lidar wind profile, the ERA5 wind speed profile was interpolated to the same heights as in the lidar profiles. The interpolation was performed by fitting a piecewise cubic Hermite interpolating polynomial (PCHIP) on a logarithmic height scale (Fritsch and Carlson, 1980; Brodlie and Butt, 1991 ; see also Hallgren et al., 2020).

Figure 3 Overview of the data split for the Utö time series, February 2015–December 2022. The data were split into training (light gray), validation (blue), and testing (yellow) subsets and to minimize temporal auto-correlation one day of data before and after blocks of validation and testing are excluded (dark gray). Hours where there was an LLJ in the lidar profile are marked with red circles.

2.4 Splitting the data

For each site, the time series of lidar observations and isochronal ERA5 data were split into training (approximately 60 % of the data), validation (approximately 15 %), and testing (approximately 15 %). Temporal auto-correlation between both training and validation as well as training and testing time periods was minimized through grouping the testing and validation data sets into 1-week blocks that are randomly placed in the time series and excluding one day before and after the blocks (resulting in 90 % coverage). In Fig. 3, the split of the data set is shown for Utö (see Supplement for ASIT and MMIJ). LLJ seasonality was evaluated at each site for the training, validation, and testing subsets and was well represented across each data set; i.e., both validation and testing subsets indicate the same seasonal pattern as in the training set. Although similarities between different height levels and across sites in which are the important predictors are to be expected, all models were trained independent of each other; i.e., no information was exchanged between the models.

2.5 ML methods

As described previously, two separate ML tasks are evaluated herein: (1) wind speed predictions at each vertical level in the observational data and (2) binary predictions of LLJs. The wind speed profile prediction (WSPP) task is a regression problem and was optimized through minimization of the root mean squared error (RMSE) with respect to the wind speed as measured by the lidars, whereas the task for binary LLJ prediction is a classification problem and was optimized through minimization of the symmetric extremal dependence index (SEDI), with the lidar observations being used as ground truth for LLJ occurrence.

In general, for both ML methods and for both tasks, the training–validation process consists of finding the optimal set of predictors. This is done by comparing the results from the forward selection and backward elimination feature selection processes (e.g., Mao, 2004). After finding the optimal set of predictors, the contribution of each predictor in order to get the best score was calculated. The predictor importance, PI, was calculated for each predictor by evaluating which score, $\widehat{s}$, the ML model would result in for the validation period, higher RMSE or lower SEDI, if omitting the predictor from the set of optimal predictors. The PI was then calculated as

$$\begin{array}{}\text{(1)}& \text{PI}=\left|\mathrm{1}-{\displaystyle \frac{\widehat{s}}{S}}\right|,\end{array}$$

where S is the score for the validation period if all predictors in the optimal set are included. Important predictors result in high values of PI, while predictors of low importance yield low values of PI. As the final step, the prediction for the test period was calculated, using the optimal set of predictors.

2.5.1 Random forest (RF)

The RF (Breiman, 2001) ML method builds on the output of a large number of individual decision trees. Each tree was fed with a random subset of the training data which minimizes the problem of overfitting, and – since each split in the tree is associated with some randomness – an individual prediction of the wind speed or the shape of the profile, LLJ yes or no, is output by each tree. These individual predictions by the trees are then merged to create a total prediction for the forest as a whole.

The number of trees in the RF is associated with both the performance and the computational time, with more trees generally resulting in higher performance and longer computational time. In this study, 100 trees were used in forward selection and backward elimination but increased to 500 trees for calculation of PI and for the prediction for the test period. The minimum leaf size, a RF hyperparameter which is inversely proportional to the number of splits in a tree, referring to the maximum amount of data points on the final nodes in a tree, was set to the recommended values: 5 for the WSPP regression task and 1 for the LLJ classification task.

For the classification task, a cost matrix was included in the training of the RF. The cost matrix attributes a higher cost to failing to predict an LLJ compared to when falsely predicting an LLJ when there was actually no LLJ according to the lidar profile. As the cost matrix was seen to have a major impact on the performance of the RF, it is crucial to find a good cost matrix prior to the feature selection. Thus, 15 pre-defined cost matrices were tested with cost factors ranging from 1 to 30 000 and evaluated for the validation period using all nine ERA5 variables as predictors. The cost factor resulting in the highest SEDI was then kept for feature selection and in the prediction for the test period.

2.5.2 Neural networks (NNs)

Two fully connected neural network architectures are implemented for the WSPP regression (using fitrnet as included in MATLAB 2021b) and LLJ classification (Tshitoyan, 2023) tasks. We refer to references therein for detailed information regarding the network architecture.

Both NN models were optimized separately for each task due to variations in their network architecture. In the case of the classification task, the NNs were optimized for each site by finding the combination of parameters that maximize the SEDI within the optimization data subset: number and configuration of hidden layers, training iterations, and the regularization parameter λ. For the WSPP task, the default activation function – the rectified linear unit function (for further details on rectifiers, see He et al., 2015) – was implemented during training, and the predictors were standardized using the subset mean and standard deviation (z-score normalization). The model was optimized via finding the number of training iterations and relative gradient tolerance for the gradient of the loss function that minimizes the RMSE. Similarly to the RF model training for the regression task, the NN was optimized using the optimization subset for each vertical level in the observational data, and both NNs (regression and classification) were optimized for each site independently.

2.6 Definition of the LLJ and design of preliminary sensitivity study

The LLJ definition describes which wind speed profiles are classified as LLJs and which are not. Generally, there is a high dispersion of LLJ definitions across studies, and although a shear-based definition is recommended for wind energy purposes (Hallgren et al., 2023a), an absolute and a relative criterion relating to the falloff (the decrease in wind speed) above and below the local maximum in the wind speed profile, i.e., the jet core, was applied in this study to simplify comparisons with other work performed in the three areas (see, e.g., Hallgren et al., 2020; Kalverla et al., 2020; Aird et al., 2022). As was shown by Aird et al. (2020), stronger LLJs may be linked to different causal mechanisms than weaker LLJs, and the LLJ definitions consequentially affect the core speed magnitude of wind speed profiles identified as LLJs.

A sensitivity study was performed to evaluate the significance of the ERA5 single-level variables during LLJ formation. The sensitivity study employed three LLJ criteria with increasing levels of strictness:

• crit. 1 is a 1 m s⁻¹ and 10 % decrease in wind speed above and below the jet core.

• crit. 2 is a 2 m s⁻¹ and 20 % decrease in wind speed above and below the jet core.

• crit. 3 is a 3 m s⁻¹ and 30 % decrease in wind speed above and below the jet core.

The study evaluates whether the changes in strictness significantly affect the distributions of the ERA5 variables during LLJ and non-LLJ hours. More specifically, the sensitivity study evaluated (1) whether monthly median meteorological conditions during LLJ formation are significantly different than hours without LLJs and (2) whether monthly median meteorological conditions during LLJ occurrence are significantly different for stronger LLJs than for weaker LLJs. Monthly statistics were used to address the large seasonal variation in LLJ frequency (see Fig. 4). These results were then used to motivate the LLJ definition utilized for both ERA5 data and lidar profiles throughout this study. For each of the three sites, Mann–Whitney U tests were applied to the full time series distributions of single-level meteorological variables during LLJ and non-LLJ hours. The null hypothesis that the distributions of single-level meteorological variables from the LLJ and non-LLJ populations have the same median value was then tested, using a significance level of 0.05.

2.7 Wind speed profile prediction (WSPP)

To extrapolate from the ERA5 single-level variables to get the wind speed in the profile, an ML model was independently trained for each level of lidar measurements; i.e., in the case of Utö, 36 independent RF and 36 independent NN models were set up. The RMSE was utilized to optimize the ML models during training and was calculated as

$$\begin{array}{}\text{(2)}& \text{RMSE}=\sqrt{\sum _{i=\mathrm{1}}^n{\displaystyle \frac{({\widehat{u}}_i-u_i{)}^{\mathrm{2}}}{n}}},\end{array}$$

where ${\widehat{u}}_i-u_i$ represents the residual, i.e., the difference between the predicted wind speed, u_i, and the lidar wind speed at the same height, ${\widehat{u}}_i$, at time step i, and where n is the number of samples.

2.8 LLJ classification task

In the classification task of predicting the LLJ, one RF model and one NN model were generated for each site. The models are optimized for the SEDI, using the LLJ occurrence as given by the lidar observations as the ground truth. The SEDI is a base-rate independent score which is non-trivial to hedge, and due to its independence of the frequency of an event, the SEDI is useful for evaluating predictions of rare events (Ferro and Stephenson, 2011). These types of events are difficult to evaluate due to the degeneration of traditional forecasting scores as events increase in rarity. The SEDI is calculated using a 2 × 2 contingency table consisting of frequency of correct rejections (cr), false alarms (fa), hits (h), and misses (m) in the predictions. Using the hit rate (H)

$$\begin{array}{}\text{(3)}& H={\displaystyle \frac{h}{h+m}}\end{array}$$

and the false-alarm rate (F)

$$\begin{array}{}\text{(4)}& F={\displaystyle \frac{\text{fa}}{\text{cr}+\text{fa}}},\end{array}$$

the SEDI can be calculated as

$$\begin{array}{}\text{(5)}& \text{SEDI}={\displaystyle \frac{\text{log}F-\text{log}H-\text{log}(\mathrm{1}-F)+\text{log}(\mathrm{1}-H)}{\text{log}F+\text{log}H+\text{log}(\mathrm{1}-F)+\text{log}(\mathrm{1}-H)}}.\end{array}$$

A SEDI of 1 indicates perfect prediction skill, while a SEDI of 0 indicates no more skill than random chance following the climatology. A negative SEDI indicates that the prediction is worse than what would have been expected from a climatological prediction.

Figure 4 Monthly Mann–Whitney significance test results for ASIT, MMIJ, and Utö. Three LLJ criteria (see Sect. 2.6) are evaluated with increasing levels of strictness. Circles denote months with at least 30 LLJ samples as extracted with each criterion and are scaled by the monthly LLJ relative frequency for that site, also shown in the bottom panels. Colored circles indicate that median values of the single-level ERA5 variables are significantly different during LLJ and non-LLJ hours; i.e., the Mann–Whitney U-test null hypothesis is rejected at the 0.05 significance level. Plus and minus signs denote whether the medians of the variables for the LLJ samples are significantly lower or higher than for the non-LLJ samples.

3 Results

3.1 Sensitivity study of LLJ predictor significance

Results for the sensitivity study of the significance of single-level meteorological variables during LLJ occurrence are presented in Fig. 4. The three LLJ definitions, crit. 1–3, are evaluated across each site, and Mann–Whitney U tests are conducted to evaluate whether median meteorological conditions during LLJ occurrence are significantly different than those during non-LLJ hours. These statistical tests are conducted to further motivate the choice of which ERA5 single-level variables to be utilized in this study and the selection of which LLJ definition to use for training and testing the ML models. Note that this test is not applicable for wdir10, as the coordinates for that variable are circular. For all three sites, circles are plotted only if the sample size is greater than 30 LLJs in that month, resulting in only 6 months plotted for MMIJ, pertaining to warmer months and crit. 1, and the circles are colored if the meteorological conditions, i.e., the single-level ERA5 variables, have significantly different monthly medians during LLJ occurrence compared to hours without LLJs. All single-level ERA5 meteorological variables tested have significantly different medians during LLJ hours when compared to non-LLJ hours in at least 1 month at each site. Each of the three LLJ criteria with increasing strictness were evaluated, and only Utö had sufficiently high LLJ sample sizes to include results for all three criteria, due to a relative higher frequency of LLJs at Utö and a larger data set in general when compared to the other two sites. Increasing the LLJ criteria strictness results in reductions of relative LLJ frequency, with more than a 50 % reduction in LLJs meeting crit. 2 as compared to crit. 1.

The highest LLJ frequency is observed in the warmer months across all three sites, with peaks at approximately 4 % of hours in June for MMIJ and ASIT (crit. 1), and at approximately 25 % of the hours in May for Utö (crit. 1). For all months, and all sites, the SHF demonstrates a significantly higher ensemble median values during LLJ occurrence when compared to hours without LLJs (Fig. 4, denoted by the + signs under the monthly circles at each site). Please note that the ERA5 uses the ECMWF convention for vertical fluxes, with the positive direction downwards; i.e., a positive shift of the SHF is a shift towards more stable atmospheric conditions. The low cloud cover (LCC) is usually at lower values when LLJs are occurring, indicating a lower number of low clouds in LLJ conditions. No clear trend is observed for the surface net radiation (Rn), while the CAPE is usually lower when LLJs occur, indicating that LLJs more frequently occur during less convective conditions. Further, conditions are usually dry and the MSLP is generally higher during LLJ occurrence. Finally, LLJs tend to appear at higher SST values than the monthly median, and the ws10 is generally lower during LLJ occurrence.

Given the evidence for significantly different meteorological conditions during LLJ occurrence for all variables considered at all sites in at least 1 month, the choice of ERA5 single-level variables for model training and development is validated. Further, given more consistent results and the larger sample sizes associated with LLJs extracted using crit. 1, the least strict criterion, this criterion is selected and is implemented for LLJ identification and model training, validation, and testing in the following sections.

Figure 5 Predictor importance by height of each variable selected by the ML methods to predict the wind speed in the profile for each of the three sites and the two ML methods: (a) RF and (b) NN. The predictor importance denotes how important the predictor is to minimize RMSE between the model predictions and the ground truth lidar observations. ERA5 variables with highest predictor importance are indicated with colored circles. The predictor importance is calculated for the validation period.

3.2 Results for WSPP

Following the validation of the choice of ERA5 single-level variables and the LLJ definition, the ML models are developed and the ERA5 single-level variables are further examined for predictor importance in WSPP (see Sect. 2.7). The run time to find the optimal set of predictors varies between height levels and among the sites but generally ranges from 4 to 8 min per height level on a standard laptop. The predictor importance for the validation period is calculated for each vertical level, and the highest values of PI are represented with colored circles in Fig. 5. For all sites, both ML methods show that the predictor with the highest importance is, as expected, ws10 followed by SHF. For each site, these results are consistent across nearly all vertical levels (50–300 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$). However, for ASIT the NN displays more dispersion identifying the important variables than the RF. The consistency in predictor importance, ws10 and SHF, across both RF and NN indicates replicability of the model development and also assures the validity of the physical importance of the predictors. Further, to test the robustness of the method and the sensitivity to the data split (Fig. 3), the time series have been randomly split into training, validation, and testing 10 times for all sites. Then the RF has then been trained according to the protocol described in Sect. 2.5 and the predictor importance calculated. Results from this sensitivity analysis (not shown) clearly separate the ws10 as the most important predictor across all folds and for all sites and SHF as the second most important predictor. The physical implications of the results regarding predictor importance are discussed in further detail in Sect. 4.

Figure 6 (a–c) RMSE for wind speed profiles for ERA5 (gray), the RF (purple), and the NN (yellow) for ASIT, MMIJ, and Utö. The values of RMSE are calculated relative to the lidar observations, and the results are valid for the test period. Panels (d–f) present RMSE for ERA5 and the ML models for the subset of the test data with LLJs (crit. 1) in the lidar profiles.

For each height level, RMSE values considering the lidar observations as the ground truth are calculated for the test period, both for the ERA5 profile and the resulting extrapolated profiles from the ML models (see Fig. 6a–c). Extracting only cases with LLJs present in the lidar profile, the RMSEs of ERA5 and the ML models are presented in Fig. 9d–f. Although the WSPP is set up for height-by-height wind speed prediction not taking LLJs specifically into account, the ML models perform better than ERA5 when an LLJ is present in the lidar profile. This is evident for both the RF and the NN at heights of approximately 50–150 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$, in which the RMSE between the WSPP and the lidar profiles is approximately 10 (NN) to 20 (RF) percent lower than that of ERA5. Considering all time steps (panels a–c), the ML models demonstrate minor reduction in RMSE over ERA5, with both the RF and the NN yielding an approximate 0 %–10 % reduction in RMSE at heights up to 150 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$ The greatest enhancement relative to the ERA5 direct model output is at levels closest to the surface, a consequence of using surface variables from ERA5 as input for the ML models. The RF model performs consistently for RMSE reduction in the ASIT test period, with an approximate 5 %–10 % reduction in RMSE at all heights (50–200 m), when compared to ERA5. For ASIT and MMIJ, the RF gives better results, lower RMSE, than NN across all heights. However, for Utö, the RF and the NN exhibit similar dispersion in RMSE, which may indicate that the NN performance is affected by a lack of sufficient training data at the other two sites, 27 %–36 % less training data. These discrepancies are discussed further in Sect. 4. In general, both the RF and the NN result in favorable RMSE reductions at heights relevant to wind energy, approximately 50–200 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$, when compared to the interpolated ERA5 wind speed profiles, and especially in cases of LLJs. In the same sensitivity test described above, running the RF for 10 random splits of the time series at each site individually, the RMSE profiles are similar to what is presented here and for ASIT even showing slightly larger reductions on average.

Figure 7 Distributions of LLJ core height (a–c) and core speed (d–f) for the three sites for the test period. For LLJ identification, crit. 1 is applied. The horizontal lines in panels (a–c) mark the height levels in the lidar measurements, i.e., the possible heights of LLJ cores. The circles mark the medians of each distribution, respectively. A threshold of at least 10 LLJs is set to plot the statistics, and thus, ERA5 data are missing for ASIT and MMIJ.

As the ML WSPP task generates wind speed profiles, these can be assessed for LLJs using crit. 1. Morphology of the LLJs identified in the ML profiles is calculated and compared with those from the lidar and interpolated ERA5 profiles (Fig. 7). The medians, denoted by circles in Fig. 7, of the jet core heights and speed distributions show a lower dispersion for the core speeds than the core heights for all three sites, indicating that core speed distribution is easier to predict than core height distribution. For ASIT and MMIJ, ERA5 did not produce enough LLJs in the test period to generate a representative distribution. For all three sites, both the RF and the NN generally predict higher LLJ core heights than those given in the lidar profiles. The NN generally outputs a bimodal distribution for predicted LLJ heights, particularly for ASIT and MMIJ (Fig. 7ab). This may indicate a threshold effect that the NN associates certain predictor values with lower or higher LLJ core heights. In comparison, distributions of LLJ core speed as predicted by the NN more closely agree with the lidar observations when compared to both the RF model and ERA5. The task of LLJ core height prediction thus appears to be more difficult than the task of LLJ core speed prediction. It is possible that other surface-level variables may be more conducive to LLJ height prediction than the variables used herein, and future investigation is warranted.

Figure 8 Predictor importance for single-level ERA5 variables at all three sites for the RF model in the binary (yes/no) LLJ classification task. Circles are both colored and scaled relative to their predictor importance. Results are for the validation period.

3.3 Binary LLJ prediction

Finding the optimal predictors for the LLJ classification tasks takes, depending on the site, between 2.5 and 6.5 min on a standard laptop. The predictor importance for the LLJ classification task during the validation period was assessed, and results are presented in Fig. 8. Here, results are only provided for the RF, as the NN becomes unstable when some predictors are missing. Higher predictor importance is associated with a greater contribution to the maximization of the SEDI. Similarly to predictor importance results for the regression task, predictors with the highest importance for wind speed profile LLJ classification are SHF and ws10. The physical significance of these predictors is discussed in further detail in Sect. 4. Testing the different LLJ criterion, crit. 1 to crit. 3, for Utö (the site with highest sample size of LLJs), the predictor importance remains consistent, with SHF and ws10 having the highest predictor importance (not shown).

Figure 9 SEDI plots for (a) ASIT, (b) MMIJ, and (c) Utö. On the lower abscissa, the percentage of hits is plotted, i.e., the hit rate, and on the upper abscissa the percentage of misses. On the left ordinate the percentage of correct rejections is plotted and on the right ordinate the percentage of false alarms, i.e., the false alarm rate. The background coloring indicates the SEDI, which is also shown by the isolines. What would be a perfect SEDI (100 % hits, 0 % false alarms) is marked by the black pentagon in the top right corner of each panel. Results for the test period and SEDI for ERA5 (blue circles), RF (purple symbols), and NN (yellow symbols) are plotted.

In Fig. 9, SEDIs for LLJ prediction in the test period are presented for each site. Included in the figure are also LLJ SEDIs for the interpolated ERA5 wind speed profile and for profiles generated in the WSPP regression task. Generally, the RF performs better than the NN, with the RF models resulting in SEDIs of approximately 0.7 for all sites. Similar to the regression task, the NN model appears to suffer from a lack of training data, with NN results most closely matching RF results for Utö (SEDI for RF: 0.72; SEDI for NN: 0.65). However, both models perform considerably better in LLJ identification than the ERA5 wind profiles. For both the RF and the NN, the classification models (diamonds in Fig. 9) perform notably better than the regression models (squares) for LLJ prediction. These results indicate that the classification task adds marked accuracy to predicting LLJ conditions when compared to the regression task or ERA5 alone. It is however possible that combinations of the two tasks may be explored for even further improvement.

Figure 10 Seasonality of binary LLJ (crit. 1) predictions for (a–c) interpolated ERA5 wind speed profiles, (d–f) the RF WSPP regression task, (g–i) the RF classification task, (j–l) the NN WSPP regression task, and (m–o) the NN classification task. Seasonality is expressed as monthly proportions of correct rejections (light gray), false alarms (yellow), misses (red), and hits (blue), calculated relative to the ground truth lidar profiles for the test period. In panels (m–o), also the total number of LLJs identified in each month in the test period is plotted.

The seasonality for LLJ predictions for the classification and WSPP tasks within the test period is evaluated in Fig. 10 in terms of the false alarms, misses, hits, and correct rejections, calculated relative to the ground truth lidar profiles. The seasonality is also plotted for the interpolated ERA5 profiles to compare and contrast ML model performance relative to ERA5 on a monthly basis. Generally, ERA5 is consistently missing LLJ occurrence across all months, with no hits at all (Fig. 10a–c). The ML models improve upon these results by increasing the monthly frequency of hits, particularly in the warmer months and particularly for Utö. For all sites, the classification models from both RF and NN correctly predict the majority of LLJs, and the RF WSPP also correctly predicts the majority of LLJs for Utö. However, all ML models introduce a high rate of false alarms, and this is particularly prominent for the classification models. This may be due to the fact that the models are optimized using the SEDI and that the RF is optimized using a cost matrix. Further refinement of the model optimization and development may reduce the high rate of false alarms.

4 Discussion

Generally, the ML models developed in this work offer an improvement over ERA5 in the regression task (WSPP) – predicting coastal wind speed profiles from single-level variables. For both ASIT and MMIJ, the RF and the NN result in lower RMSE than ERA5 in the lower half of the height range swept by the turbine blades (hub height and down) of a typical modern offshore wind turbine, when compared to the ground truth lidar profiles (Fig. 6). For Utö, the performance of the ML models is similar to that of ERA5. Given these findings, it may be possible to utilize trained ML models to improve historical wind speed profiles from ERA5 (non-linear bias correction), as the ML models provide improved information on the wind speed profile based on ERA5 predictors, while the ERA5 wind profile itself is not always very representative for local behavior. Further, it is likely that similar ML models could be developed using direct field observations, i.e., from buoys and met masts, to be used for real-time wind profile predictions. However, it should also be mentioned that – as the ML methods applied here focus on reducing the RMSE – there is a possibility that the quality of the prediction is inferior in other aspects when compared to ERA5. For example, analyzing the wind speed distributions at 150 m height for the three sites (not shown), ERA5 better represents the full distributions, while both RF and NN tend to underestimate the extremes and overestimate the peak of the distributions. This could have further implications, e.g., in non-linear calculations such as in estimations of the power production. We strongly recommend to optimize the ML models using the metric (or combination of metrics) that are of importance for the application and to be aware that an improvement for one metric does not necessarily result in an improvement for all other metrics.

In particular, both the RF and the NN exhibit high fidelity when compared to ERA5 for reducing the RMSE for hours in which an LLJ is present. For the ML classification task of binary LLJ prediction, both ML models produce higher SEDI than ERA5, although both models suffer from a high rate of false alarms, an artifact of optimizing for the SEDI and a consequence of the cost matrix for the RF method. Since correct rejections are numerous, it might be beneficial to optimize the ML methods for a measure not including those, such as, for example, the F1 score or the critical success index (CSI). Utilizing a larger set of predictors, the false alarm rate could be improved (however, see further discussion below why not more predictors were included in the study). Figure 10 brings up many questions relevant for further investigation, such as (1) is there anything specific about LLJs that are missed or hit by the ML (e.g., hypothetically the more intense LLJs could be easier to predict), and (2) are persistent LLJs more likely to be predicted?

Regardless of the statistical differences between the model architecture of the RF and NN models, both methods show high PI for SHF and ws10. In at least 1 month at each site, all of the predictors demonstrate significantly different median values during LLJ hours when compared to non-LLJ hours as described by results from the Mann–Whitney U tests (Fig. 4). However, the median SHF, which varies between negative and positive values depending on the month, exhibits significantly higher values during LLJ hours, when the median SHF always is positive, for all three sites and across all months. In the WSPP regression task, the PI is consistent across all vertical levels, indicating that dynamic readings of ws10 and SHF are highly important in reducing the RMSE of the predicted wind speed profiles. This high predictor importance is also observed for the classification task, indicating that ws10 and SHF maintain high PI even when Monin–Obukhov similarity theory is invalid, i.e., due to the presence of an LLJ. The high predictor importance of SHF may be attributed to its relation to stability, particularly for air–sea temperature differences. Note that, throughout this study, the wind direction (wdir10) was included as values in the range 0–360°. Using the sine and cosine, encoding of the wind direction to better handle the circular coordinates of the variable was tested for the analysis of predictor importance in binary predictions of the LLJ using the RF method, and although this representation causes minor differences in predictor importance, the main conclusion that SHF and ws10 are the most important predictors still holds. It is also worth noting that, although warm air advection over relatively cold water in spring and early summer, creating stable conditions, is often mentioned as one of the key formation mechanisms for offshore/coastal LLJs, it is the stability of the atmosphere rather than the wind direction itself that is most important; i.e., the SHF is more important than wdir10 (no matter the encoding) for LLJ prediction.

Both the RF and the NN are sensitive to the quantity of training data available, with both methods having higher fidelity in both ML tasks (classification and regression) for Utö, the site for which the most lidar data are available for training. In particular, the NN exhibits higher sensitivity to a paucity of training data when compared to the RF. This discrepancy highlights the challenge in choosing adequate ML models for various applications; although the NN is traditionally more complex and thus more capable of making more sophisticated statistical inferences, it is more sensitive to the body of training data and thus may produce weaker results than would be expected a priori (Wang et al., 2017). Initial testing, with a larger set of single-level variables from ERA5, shows improved results for both models, both in terms of the RMSE and SEDIs, with the NN ahead of RF. However, as high cross-correlation between the variables makes the analysis of predictor importance finicky, we decided not to include it. Although not applied in this study, spatial variation in the single-level ERA5 data could also be used as predictors for the ML models, making it possible to identify conditions on both the mesoscale and synoptic scale that favor LLJ formation, e.g., coastal effects and fronts.

It is of course also interesting to test whether the ML models created for Utö could be used to predict the wind conditions at ASIT or MMIJ. While this research question lies beyond this study, it is possible to hypothesize that, as wind conditions at coastal sites are sensitive to the local conditions, e.g., if the wind is directed from the land or the sea, a model optimized for one of the sites is of relatively low value for the other sites. Switching from model output to locally performed measurements, these site-specific characteristics of the ML models will be even more pronounced. However, in homogeneous surroundings, e.g., far offshore, it is likely that the models will be more general. The decrease in performance of the ML models with increasing distance to the origin of training data is suggested to be further analyzed.

Different optimization metrics could be introduced depending on the wind energy application under consideration; i.e., if a high number of false alarms would be detrimental, a different score that minimizes false alarms could be utilized. Further, given the improvement in results for Utö as compared to the other sites, it is possible that longer data sets in areas of higher LLJ frequency may be the best candidates for employing the ML methods presented here.

5 Summary and conclusions

Two ML tasks are investigated using multiple years of ERA5 reanalysis and lidar profiles at three offshore sites (the US Northeastern Atlantic Coastal Zone, the North Sea, and the Baltic Sea) of relevance to wind energy. The regression task of accurately extrapolating the wind speed profile, up to a maximum of 315 m a.s.l., is investigated through development of two ML methods, RF and NN, using nine selected single-level ERA5 variables as predictors and observed wind speed profiles by vertical scanning lidars as the ground truth. All selected single-level ERA5 variables are physics-based and chosen either for their relevance in describing synoptic-scale and local-scale atmospheric conditions, their ability to be measured using real-time observations within a wind farm, or both. The ws10 and the SHF have by far the highest PI in WSPP. Further, these two variables are also the most important in predicting the LLJ and can help in assessing expected production from a farm and loads on a turbine. This indicates that both variables are highly important for describing local-scale and mesoscale processes that influence wind speed profile development.

At heights below approximately 200 $\mathrm{m}\mathrm{a}.\mathrm{s}.\mathrm{l}.$, both the RF and the NN have lower RMSE when compared to ERA5 for predicted wind speed profiles during hours in which an LLJ was present within the lidar profiles. Further, both models exhibit markedly higher SEDIs than ERA5 for binary low-level jet predictions using the single-level ERA5 variables as predictors.

Future work could employ the use of more training data when developing the models. This would be especially helpful for the NN, which demonstrates a high sensitivity to a relative paucity of training data for both ML tasks. However, the RF would also benefit from more training data, and thus consistent and accurate long-term measurements with high vertical resolution are important. The results presented herein show promise for utilizing field measurements to make real-time predictions of coastal wind speed profiles, particularly during conditions or at heights for which the Monin–Obukhov similarity theory is invalid, i.e., in the case of LLJs or negative shear in general. Results also show promise for making wind speed profile predictions from models that may lack sufficient vertical resolution at heights relevant to wind energy.

Appendix A: List of abbreviations

ANN	artificial neural network
ASIT	Air–Sea Interaction Tower
CAPE	convective available potential energy
CSI	critical success index
ECMWF	European Centre for Medium-Range Weather Forecasts
ERA5	ECMWF Reanalysis version 5
IEA	International Energy Agency
IFS	Integrated Forecasting System
LCC	low cloud cover
LCoE	levelized cost of energy
Lidar	light detection and ranging
LLJ	low-level jet
ML	machine learning
MMIJ	Meteorological Mast IJmuiden
MSLP	mean sea level pressure
NN	neural network
NWP	numerical weather prediction
PCHIP	piecewise cubic Hermite interpolating polynomial
PI	predictor importance
precip.	precipitation
RF	random forest
RMSE	root mean squared error
Rn	net radiation at the surface
SEDI	symmetric extremal dependence index
SHF	surface sensible heat flux
SST	sea surface temperature
wdir10	wind direction at 10 $\mathrm{m}\mathrm{a}.\mathrm{g}.\mathrm{l}.$
WHOI	Woods Hole Oceanographic Institute
WRF	Weather Research and Forecasting
ws10	wind speed at 10 $\mathrm{m}\mathrm{a}.\mathrm{g}.\mathrm{l}.$
WSPP	wind speed profile prediction