One method for correcting lidar turbulence includes modeling the spatial averaging effects of the lidar probe volume. This method involves convolving the true radial velocity field with a spatial weighting function that is controlled by the lidar beam pattern e.g.,. Spatial weighting functions for both pulsed and continuous wave lidars are relatively straightforward e.g.,. However, modeling the true velocity field requires knowledge of the three-dimensional turbulence structure, which can be described by the spectral velocity tensor, Φij.

The spectral velocity tensor can be modeled through the use of the turbulence model, as in , , , and others. Fitting the model requires three parameters: a turbulence dissipation rate parameter, a length scale, and a parameter that describes the anisotropy of the flow. Values for these parameters can be estimated by using high-frequency sonic anemometer data and can also be approximated from lidar data. However, the turbulence model is technically only valid in the surface layer under neutral conditions and is not valid in complex terrain.

To reduce variance contamination caused by the DBS and velocity–azimuth display VAD; techniques, proposed a new six-beam scanning technique for Doppler lidars that utilizes the variance of the radial velocity. tested the six-beam method with a scanning lidar at the Boulder Atmospheric Observatory in Erie, Colorado, and compared six-beam variance estimates to estimates from sonic anemometers on a tower at the site. found that while the six-beam method did generally reduce variance contamination in comparison to estimates from a WC lidar, errors in the different radial velocity variance estimates caused large errors and even negative values in the resulting u and v variance estimates. Better estimates of the radial velocity variance are likely needed from lidars to obtain accurate results for the six-beam technique.

While single lidars require measurements around a scanning circle to estimate the three-dimensional velocity field, multiple scanning lidars can be pointed toward a particular volume of air to obtain turbulence estimates with much higher spatial resolution e.g.,. To collect turbulence measurements, multi-lidar systems must be temporally and spatially synchronized with a high degree of accuracy. Synchronization techniques have been developed for a set of user-customized scanning lidars but are currently not easily implemented on most other scanning lidars. In addition, scanning lidars are much more expensive than commercially available vertically profiling lidars, particularly if more than one scanning lidar is required for operation.

Structure functions describe the spatial correlation of a variable at different separation distances e.g.,. If the turbulence is isotropic and the turbulence length scale is large, the structure function can be approximated by the model and used to estimate the velocity variance. used scanning lidar data from a field campaign to calculate structure functions in both the along-beam and azimuthal directions and fit the functions to the model to obtain estimates of the velocity variance. The lidar data used by were obtained from a series of plan-position indicator (PPI) scans with high azimuthal resolution, which is typically not available from a scanning strategy used by a vertically profiling lidar.

As discussed by , the spectral density of a particular radial velocity, vr, is essentially a weighted count of all the positions within the probe volume where the radial velocity is equal to vr. The weighting occurs because the intensity of the lidar beam is highest at the center of the probe volume and drops off for distances in either direction from the probe volume center. The ensemble-averaged spectrum can then be related to the probability density function of the radial velocity at each position within the probe volume. Given this relation, the unfiltered (“true”) variance can be obtained from the second central moment of the Doppler spectrum. If the lidar is mounted on a turbine nacelle and pointing upstream, as in , it can be assumed that the wind field is homogeneous along the lidar beam and that the probability density of vr is approximately uniform along the probe volume. However, if a ground-based, vertically profiling lidar is used, the mean wind field will not be uniform along the lidar's line-of-sight and the effects of shear must be taken into account when estimating the unfiltered variance from the Doppler spectrum . Currently, this method is more clearly defined for continuous wave lidars, as the Doppler spectra of pulsed lidars are affected by the finite length of the probe volume in addition to turbulent fluctuations.

Several methods are currently available for obtaining more accurate turbulence estimates from Doppler lidars. Only a few methods were discussed here; a more extensive discussion of turbulence retrieval techniques can be found in and . Most of these methods require the fitting of models and the use of very specific scanning strategies that can currently only be achieved with expensive scanning lidars. The Doppler spectrum method is promising for continuous wave lidars but requires knowledge of the Doppler spectrum obtained at each lidar beam position, which is usually not available in the output of vertically profiling systems. Thus, there is clearly a need for a turbulence estimation method that can be implemented on vertically profiling lidars that use DBS and VAD techniques and that does not require high-resolution data from a sonic anemometer. Details of the new turbulence estimation method proposed in this paper are discussed in the next section.

After the lidar data are processed, different techniques are used to remove noise and spurious data. A standard way to remove outliers from a time series is to use a spike filter e.g.,. A basic spike filter was evaluated for the model in addition to several methods developed by .

The spike filter routine used in this work was based on one of the lidar preprocessing steps presented by . First, the difference between adjacent velocity values, Δv, is calculated for all velocity measurements in a 10 min period, defined by the following equation: Δvi=vi+1-vi.

A data point vi is defined as a spike and removed from the dataset if the following conditions are met:

Δvi>2IQRΔv,Δvi-1>2IQRΔv,ΔviΔvi-1=-1, where IQRΔv is the interquartile range of all values of Δv within the 10 min period. This spike filter eliminates data points defined by a large decrease in velocity followed by a large increase in velocity (or vice versa).

The methods involve the use of the lidar's velocity spectrum or auto-covariance function to estimate the amount of noise in the variance measurements from the lidar. In the lidar velocity spectrum, power at the high-frequency end of the spectrum is assumed to be largely attributed to white noise. The average power at the high-frequency end of the spectrum is integrated across all frequencies to estimate the variance due to noise. An estimate of noise variance can also be made by assuming that noise is random and completely uncorrelated with the velocity signal. Thus, the value of the auto-covariance function at lag 0 is a sum of the noise variance and actual velocity variance. The noise variance can be estimated by extrapolating the auto-covariance function from non-zero lags to lag 0. The difference between the auto-covariance function at lag 0 and the extrapolated value is the noise variance.

Two methods were considered to mitigate the effects of volume averaging: structure functions and spectral extrapolation. As discussed in Sect. , structure functions can be estimated using available lidar data and fit to models to estimate turbulence parameters e.g.,. By fitting the lidar data to a model, the reduction in turbulence due to volume averaging is mitigated. Although the estimation of structure functions with a lidar is optimized with the use of a high-resolution PPI scan, as in , structure functions can also be estimated from DBS scans. A longitudinal (i.e., along-beam) structure function can be estimated from a DBS scan by using velocity data from different range gates along the same radial. An azimuthal structure function can be estimated by combining data from different azimuthal directions at the same height, although the fit is likely to be poor for a DBS scan due to the small number of azimuthal angles where data are collected.

Another method of mitigating volume averaging is to model the lidar velocity spectrum and use the model to extrapolate the spectrum to higher frequencies. The high-frequency part of the modeled spectrum can then be integrated to obtain an estimate of the variance that is not measured by the lidar as a result of spatial and temporal resolution e.g.,. The standard Kaimal spectrum used for wind energy e.g., requires three parameters for fitting: the mean wind speed, the variance, and a length scale. The first two parameters can be calculated from the lidar data directly, while the last parameter must be estimated. In this work, the length scale was estimated in two different ways: by minimizing the difference between the actual velocity spectrum and the Kaimal spectrum (Spectral Fit 1) and by calculating the integral timescale from the raw velocity time series, which can then be related to the integral length scale through Taylor's frozen turbulence hypothesis (Spectral Fit 2).

Methods to reduce variance contamination include the six-beam technique developed by , discussed in Sect. , and the use of Taylor's frozen turbulence hypothesis with data from the WC's vertical beam . The six-beam technique can be adapted for the WC DBS scans by estimating the variance from the five different radial beam positions (four off-vertical and one vertical) and solving a system of five equations to determine the variance and covariance components. Although the covariance of the u and v components cannot be determined with this method, the three velocity variance components can be estimated . Taylor's frozen turbulence hypothesis can be used to relate temporal changes in the vertical velocity measured by the WC's vertical beam to spatial changes in the vertical wind field across the WC scanning circle. Spatial changes in the w component can then be used to reduce contamination in either the raw wind speed (Taylor 1) or the variance directly (Taylor 2).

For both the ARM site and the wind farm, all possible combinations of the physics-based corrections described in Sect.  for the u, v, and w components were evaluated. Initially, the model combination that minimized the overall TI MAE was selected as the optimal model combination for that particular site. Data were filtered to avoid mast shadowing, and 10 min periods where the mean wind speed was less than 4 m s-1 were not used to evaluate L-TERRA, as the standards outlined in IEC 61400-12-1 restrict remote-sensing classification to wind speeds between 4 and 16 m s-1.

At the ARM site, the original TI MAE was 1.5 %, and MAE values that resulted from the application of L-TERRA ranged from 1.31 to 2.73 %. MAE values above the original value of 1.5 % indicate that for these model combinations, L-TERRA actually increased overall error in WC TI. For many of these model combinations with high MAE values, the MAE increased for stable conditions, while decreasing for unstable conditions and vice versa, indicating that some model combinations work better for particular stability conditions than others. Many of the high MAE values were also associated with model combinations that used the Lenschow noise removal techniques with the 4 s VAD scans. The Lenschow techniques are more aggressive with noise removal than a spike filter and also involve directly reducing the variance due to noise, rather than removing spikes in the raw velocity time series. It is possible that the noise apparent in the original WC data artificially brought the WC TI estimates closer to the sonic TI, and removing the noise variance decreased the WC TI values, bringing them further from the sonic values and increasing the MAE. In addition, spectra and auto-covariance functions derived from the 4 s data are expected to be less accurate than those derived from the higher-resolution 1 s data, which could affect the accuracy of the Lenschow techniques.

TI MAE values also varied a large amount at the wind farm, with an original MAE of 1.46 % and L-TERRA MAE values ranging from 1.38 to 2.9 %. Similarly to the ARM site, many of the higher MAE values were associated with model combinations that decreased the MAE for certain stability conditions while increasing the MAE for other stability conditions.

Overall, the model combination that minimized the TI MAE was nearly the same for both sites and is shown in the first row of Table  (the only difference between the optimal model combination at the sites was in the variance contamination module). The application of this initial L-TERRA model combination resulted in a modest reduction in lidar TI MAE from 1.5 to 1.31 % at the ARM site and from 1.46 to 1.38 % at the wind farm (not shown). Several slightly different model combinations produced similar MAE values at both sites, suggesting that there may actually be multiple optimal combinations of L-TERRA at each site when the MAE is minimized. For example, at the wind farm, the minimum MAE of 1.38 % was achieved when the first Lenschow method was used to reduce noise from the raw 1 Hz wind speeds and the first correction using Taylor's frozen turbulence hypothesis was employed to reduce variance contamination. However, MAE values within 0.01 % of 1.38 % were also achieved when different noise removal techniques were used instead of the first Lenschow technique or when no noise removal technique was used at all. Thus, at this site, noise removal appeared to have a very minor impact on the WC TI values. It may be useful to consider other parameters in determining the optimal model combination, such as regression line statistics or the sensitivity of TI error to atmospheric stability. However, minimizing the MAE is a standard approach for determining optimal model combinations and provides a useful baseline combination for evaluating L-TERRA.

By examining the change in lidar TI after each step in L-TERRA, it was determined that some corrections decreased error under stable conditions but increased error under unstable conditions and vice versa. This is not surprising, as the magnitude and sign of TI errors was strongly dependent on atmospheric stability at both sites (Tables , ) as a result of the different factors that affect TI error under different stability conditions. Thus, optimal model combinations were next determined separately for the three different bulk stability classes to form a stability-dependent version of L-TERRA (L-TERRA-S). Optimal model combinations were very similar for both sites and are shown in Table .

For stable and unstable conditions, a spike filter was the optimal noise removal technique. Only the model chain for stable conditions included a volume averaging correction, likely because volume averaging effects on TI are largest under stable conditions. For unstable conditions, using the velocity time series from the VAD technique produced the largest reduction in MAE. While the raw output time series from the WC is available at 1 Hz (Sect. ), the VAD technique is typically applied once per full scan to derive the three-dimensional wind vector. For the WC, this results in an output data frequency of 0.25 Hz for the VAD technique. The lower temporal resolution of the VAD technique likely served to artificially reduce some of the effects of variance contamination, as smaller scales of turbulence were not measured.

The impact of each of the different physics-based correction modules on the TI error is shown in Fig. . Overall, MAE steadily decreased after the application of the different corrections, with the largest decrease occurring for the volume averaging module. For unstable TI values, the noise reduction module had the largest impact on reducing MAE and bringing the regression line slope closer to 1. The variance contamination module served to further reduce the MAE and regression line slope, bringing the slope from approximately 1.05 to 1.00, but the R2 value of the regression line decreased. Similarly, the variance contamination module reduced the regression line slope for neutral TI values, bringing it closer to 1, but caused the R2 value of the regression line to decrease. This resulted in an MAE value of 1.48 % for neutral TI values after L-TERRA was applied, which is the same as the original MAE value for neutral conditions. The corrections performed best on stable TI values, with MAE values steadily decreasing and R2 values increasing with each correction. The volume averaging module caused the largest change in stable TI values, with the regression line slope increasing from 0.90 to 1.01 after the application of the volume averaging module. In summary, all the physics-based corrections in L-TERRA appear necessary to the correction of WC TI, though the importance of each correction depends on the stability. The variance contamination module likely needs to be improved for certain types of unstable and neutral conditions, as it increased TI error for some unstable and neutral TI values.

Scatterplots of ARM site TI data after L-TERRA-S was applied are shown in Fig. b, and corresponding regression statistics are shown in Table . L-TERRA-S served to bring the majority of WC TI estimates closer to the 1 : 1 line, resulting in regression line slopes of 1.00, 1.01, and 1.00 for stable, neutral, and unstable conditions, respectively. In addition, the overall TI MAE decreased from 1.5 to 1.25 %. However, as discussed in the previous paragraph, R2 values for neutral and unstable conditions decreased slightly. Thus, although L-TERRA-S improved the accuracy of most lidar TI estimates, it also increased scatter for neutral and unstable conditions.

Results for the wind farm were similar, with overall MAE decreasing from 1.46 to 1.19 % and regression line slopes becoming 1.00, 1.04, and 1.05 for stable, neutral, and unstable conditions, respectively (Table ). R2 values for neutral and unstable conditions also decreased slightly.

One possible explanation for the increase in scatter at both sites could be the misclassification of atmospheric stability by the shear exponent, α. As discussed in Sect. , opposing stability classifications were made based on Ri and α during approximately 5 % of the 30 min cases where both stability parameters were available. For L-TERRA-S, an incorrect stability classification could lead to the application of corrections that are not well-suited for the actual errors in the lidar TI. To examine the impact of incorrect stability classification on remaining TI errors, time periods were identified where the WC TI error at the ARM site was above the 95th percentile of all TI errors after L-TERRA-S was applied; these TI values represent outliers in Fig. b and contribute significantly to the low values of R2.

Of the 56 TI outliers identified that were associated with valid values of Ri and α, 16 points were classified as unstable by both Ri and α, 5 points were classified as neutral by both parameters, and 4 points were classified as stable by both parameters. The remaining 33 points were classified differently by Ri and α, with nearly half (14) of the points being classified as neutral by Ri and slightly unstable or slightly stable by α, 12 points being classified as stable or unstable by Ri and neutral by α, and the remaining 5 points being classified as stable by Ri and slightly unstable by α. The majority of these stability misclassifications appear to occur near neutral conditions, where small errors in measurement of the wind shear or temperature gradient could lead to a different stability classification. Thus, it may be useful in the future to use a blend of correction techniques for points classified as near-neutral or use additional parameters to classify stability from lidar data. However, as opposing stability classifications accounted for fewer than 10 % of the large TI errors apparent after the application of L-TERRA-S, it is likely that other factors were also responsible for the large amount of scatter still apparent in the TI data. For example, it is likely that the current physics-based corrections in L-TERRA-S do not fully capture all the factors that affect lidar TI error, resulting in large errors that remain for some data points. In the next section, machine-learning techniques are evaluated as a potential method to model the remaining physics that impact lidar TI error.

To determine appropriate predictor variables for the machine-learning module, a sensitivity analysis was conducted for the WC TI error at both sites. The sensitivity of the lidar TI error to the various predictor variables in Table  was quantified following the guidelines in Annex L of the new committee draft of IEC 61400-12-1 . First, predictor variables were binned and bin means of the TI percent error corresponding to each bin were calculated. A least-squares technique was then used to calculate a regression line between the predictor bin centers and the bin means of the TI percent error. Sensitivity, defined as the product of the regression line slope and the standard deviation of the predictor variable, was then calculated for each predictor. The sensitivity gives the approximate change in the TI error for a change in the predictor variable that is equivalent to 1 standard deviation of the variable.

Sample plots showing the response of TI percent error to different variables at the ARM site are shown in Fig. . Raw WC TI error was extremely sensitive to the four variables depicted in Fig. , with larger TI percent errors for lower wind speeds and SNR values and TI errors changing sign from negative to positive for decreasing shear parameter values and increasing raw TI values. After L-TERRA-S was applied, TI error sensitivity to shear parameter and raw lidar TI decreased significantly. This decrease in sensitivity demonstrates a major advantage of L-TERRA-S, as it implies that lidar TI error is no longer strongly dependent on atmospheric stability. However, while L-TERRA-S decreased sensitivity slightly for mean wind speed and SNR, a high dependence of TI error on these two variables is still apparent in Fig. c and d.

Overall, the six variables for the ARM site with the highest sensitivity values after the application of L-TERRA-S were as follows: integral timescale (vertical), SNR, corrected TI, integral timescale (horizontal), mean wind speed, and shear parameter. (For highly correlated variables, the variable with a higher sensitivity was retained in the list.) These six variables were then used to train a random forest with the wind farm data.

Results from the application of the trained random forest on the ARM site L-TERRA-S TI values are shown in Fig. a. The application of the random forest increased MAE values from 0.78 to 0.89 % for stable conditions and from 1.48 to 1.6 % for neutral conditions and decreased the MAE from 1.59 to 1.53 % for unstable conditions in comparison to the results from L-TERRA-S. For all three stability classifications, R2 values dropped significantly and a positive bias was induced for low TI values.

To determine the cause of this positive bias, the sensitivity values from both sites were compared for the six input variables. While the regression line slope and sensitivity values for the vertical integral timescale, SNR, mean wind speed, and shear were very similar at both sites, sensitivity values for the horizontal integral timescale and corrected TI differed substantially. In particular, the sensitivity of TI error to the corrected TI was 6.86 % at the ARM site and over 5 times larger at 39.6 % at the wind farm. After the removal of the horizontal integral timescale and the corrected TI from the input parameter list, the bias at low TI values largely disappeared (Fig. b), suggesting that the positive bias was related to the different sensitivities associated with two of the input parameters. However, even with these two parameters removed from the input parameter list, the MAE values still increased in comparison to L-TERRA-S, while R2 values decreased. Results from the MARS model were similar to those from the random forest models.

These results highlight two major limitations of using machine-learning techniques to improve lidar TI accuracy in L-TERRA: (1) the most significant input parameters can change from one site to another and will not be known a priori for a new site, and (2) the sensitivity of TI error to different input variables depends on the training site and the particular lidar and reference measurements used. To investigate the effect of the training dataset used, a random forest was also trained on 75 % of the ARM site data and then applied to the remaining 25 %. Training and testing a random forest on data from the same site did decrease MAE values in comparison to results from L-TERRA-S, but R2 values still decreased slightly for neutral and unstable conditions. More importantly, using this technique would preclude L-TERRA from being applied at a new site that does not have a met tower.

Although machine learning can be a useful tool for turbine power prediction e.g.,, it does not appear to be an ideal technique for correcting lidar TI error. Thus, the next steps in the development of L-TERRA will involve further refining the physics-based corrections in L-TERRA-S to improve TI estimates in a more robust manner. Rather than relying on modeled patterns, physics-based corrections directly relate lidar measurements to TI errors and substantially improved the accuracy of lidar TI estimates at both sites evaluated in this work (Fig. , Tables  and ). However, the current physics-based corrections do not completely eliminate TI error, indicating that the physics that cause TI error are not being entirely captured in L-TERRA-S. Future work will involve the development of a lidar uncertainty framework that outlines all possible causes of lidar error. Different parts of the framework could then be quantified through the use of a simulated flow field and virtual lidar, as in .

Results from the sensitivity analysis conducted in this section will greatly assist in determining areas of focus for the lidar uncertainty framework. For example, TI error at both sites was extremely sensitive to the integral timescale of the w wind component, which is a proxy for the dominant temporal scale of turbulent eddies in the vertical direction. Thus, lidar TI error appears to be strongly affected by the scales of vertical motion present in the area enclosed by the lidar scanning circle, which will contribute to the degree of variance contamination. Currently, no physical models exist that could account for these effects, and so we suggest that this could be a fruitful research direction. In future work, the virtual lidar tool will be used to examine how changes in the vertical flow field across the lidar scanning circle impact TI estimates and how information from the lidar can be used to approximate and remove these effects.