An Overview of Wind Energy Production Prediction Bias, Losses, and Uncertainties

The financing of a wind farm directly relates to the preconstruction energy yield assessments which estimate the annual energy production for the farm. The accuracy and the precision of the preconstruction energy estimates can dictate the profitability of the wind project. Historically, the wind industry tended to overpredict the annual energy production of wind farms. Experts have been dedicated to eliminating such prediction errors in the past decade, and recently the industry is recording near-zero average energy prediction bias. Herein, we present an overview of the energy yield assessment errors 10 across the global wind energy industry. We identify a long-term trend of reduction in the overprediction bias, whereas the uncertainty associated with the prediction error is prominent. We also summarize the recent advancements of the wind resource assessment process that justify the bias reduction, including the improvements in modeling and measurement techniques. Additionally, because the energy losses and uncertainties substantially influence the prediction error, we document and examine the estimated and observed loss and uncertainty values from the literature, according to the proposed framework in 15 the International Electrotechnical Commission 61400-15 wind resource assessment standard. From our findings, we highlight the opportunities for the industry to move forward, such as the validation and reduction of prediction uncertainty, and the prevention of energy losses caused by wake effect and environmental events. Overall, this study provides a summary on how the wind energy industry has been quantifying and reducing prediction errors, energy losses, and production uncertainties. Finally, for this work to be as reproducible as possible, we include all of the data used in the analysis in appendices to the 20 manuscript.


Introduction
Determining the range of annual energy production (AEP), or the energy yield assessment (EYA), has been a key part of the wind resource assessment (WRA) process. The predicted median AEP is also known as the P50, where the actual AEP would exceed that threshold 50% of the time. For years, the experts in the field have been discussing the difference 25 between the predicted P50 and the actual AEP, where the industry often overestimates the energy production of a wind farm (Hale, 2017;Hendrickson, 2009Hendrickson, , 2019Johnson et al., 2008). A recent study conducted by the researchers at the National Renewable Energy Laboratory (NREL) concludes 3.5% to 4.5% of average P50 overprediction bias, based on a subset of wind farms in the United States, after accounting for curtailment (Lunacek et al., 2018). https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License.
The financial implication of such P50 overestimation is influential. Healer (2018) states that if a wind project produces 30 a certain percentage lower than the P50 on a 2-year rolling basis, the energy buyer, also known as the offtaker, has the option to terminate the contract. For a 20-year contract, if a wind farm has a 1% chance of such underproduction within 2 years, the probability of such event taking place within the 18 2-year rolling periods is 16.5%, where 100% -(100% -1%) 18 = 16.5% (Healer, 2018), assuming each 2-year rolling period is independent. Therefore, projects with substantial energy-production uncertainty experience the financial risk from modern energy contracting. 35 The WRA process governs the accuracy and precision of the P50, and a key component in WRA constitutes the estimation of energy-production losses and uncertainties. Wind energy experts have been using different nomenclature for in WRA, and inconsistent definitions and methodologies exist. To consolidate and ameliorate the assessment process, the International Electrotechnical Commission (IEC) 61400-15 working group has proposed a framework to classify various types of energy-production losses and uncertainties (Filippelli et al., 2018, adapted in Appendix A). We illustrate the categorical and 40 subcategorical losses and uncertainties in Figs. 1 and 2. Note that the proposed framework is not an exclusive list of losses and uncertainties because some institution-specific practices may not fit into the proposed standard.  Table A2.
The wind energy industry has been experiencing the monetary impacts caused by the challenges and difficulties in predicting energy-production losses and uncertainties over the lifetime of a modern wind project, which can continue to operate beyond 20 years: • For the financial impact of P50 prediction error, losing 1 GWh translates to about 50,000 to 70,000 Euros 55 lost in the industry (Papadopoulos, 2019).
• Regarding the uncertainty of wind energy production, reducing 1% of energy uncertainty can result in $0.5 to $2 millions of economic benefits, depending on the situation and the financial model (Brower et al., 2015;Halberg, 2017).
• In one study that quantifies the economic influence of wind speed uncertainty, changing 1% of such 60 uncertainty can lead to 3% to 5% change in net present value of a wind farm (Kline, 2019).
Experts in the industry have presented many studies on the P50 prediction error, energy loss, and uncertainty for years, and the purpose of this evaluation of past literature is to assemble previous research and deliver a meaningful narrative.
This manuscript is unique and impactful because this is the first comprehensive review and analysis of the key parameters in the WRA process across the industry. The three main research questions of this study include: 65 -Is the P50 prediction bias changing over time, and what are the reasons for the changes?
the same year, we select the one closest to zero, because those numbers reflect the state of the art of P50 validation of that year ( Fig. 3). Along the same line, we use the paired P50 errors to indicate the effects from method adjustments (Fig. 4). To track the bias impact of technique changes from different organizations, we combine the closely related, ongoing series of studies from a single organization, usually by the same authors from the same institutions (each line in Fig. 4).
We also derive the trend of P50 prediction errors using linear regression and investigate the reasons behind such trend. 95 In the linear regressions presented in this article (Figs. 3,8,and C1), we implement the 95% confidence interval, generated via a bootstrap, using the regplot function in the Seaborn Python library (Waskom et al., 2020). The confidence interval describes the bounds of the regression slope and intercept with 95% confidence. Furthermore, we present the 95% prediction interval in Fig. 3, which depicts the range of the predicted values (e.g. the P50 prediction bias in Fig. 3) with 95% confidence, given the https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License. existing data and regression model. In short, the confidence interval illustrates the uncertainty of the regression function, 100 whereas the prediction interval represents the uncertainty of the estimated values of the predictand. In addition, we evaluate the regression analysis with the coefficient of determination (R 2 ), which represents the proportion of the variance of the predictand explained by the regression.
For loss and uncertainty, we need to interpret a small subset of the data, because these data are only sparsely available.
When a source does not provide an average value, we perform a simple arithmetic mean when both the upper and lower bounds 105 are listed. For instance, when the average wake loss is between 5% and 15%, we project the average of 10% in Fig. 6, and we present all the original values in Appendix B. If only the upper bound is found, then we project the data point as a maximum: the crosses in Fig. 6 are used as an example.
We categorize the data to the best of our knowledge to synthesize a holistic analysis. On one hand, if the type of loss and uncertainty from a source uses marginally different terminology from the IEC proposed framework, we first attempt to 110 classify it within the IEC framework, we gather other values in the same category or subcategory from the same data source, and we select the minimum and the maximum. As an illustration, if the total electrical losses from the substation and the transmission line are, respectively, 1% and 2%, we then label the total electrical loss with the range of 1% to 2%. On the other hand, when the type of loss and uncertainty illustrated in the literature largely differ from the IEC framework, we label them separately (Figs. 7 and 11). Because a few studies contrast wake loss and nonwake loss, where nonwake loss represents every 115 other type of energy loss, we also include nonwake loss in this study (Figs. 6 and 10). When a type of uncertainty is recorded as simply "extrapolation," we label it as both horizontal and vertical extrapolation uncertainties. We also divide the reported losses and uncertainties into two groups, the "estimated" and the "observed", where the former are based on simulations and modeling studies, and the latter are quantified via field measurements.
Unless specifically stated otherwise in Appendix B, we present a loss value as the percentage of production loss per 120 year, and we document an uncertainty number as the single standard deviation in energy percentage in the long term, usually for 10 years or 20 years. The wind speed uncertainty is stated as a percentage of wind speed in m s -1 , and the uncertainty of an energy loss has the units of percent of a loss percentage.
This manuscript evaluates a compilation of averages, where each data point represents an independent number. The metadata for each study in the literature vary, in which the resultant P50 prediction errors, losses, and uncertainties come from 125 diverse collections of wind farms with different commercial operation dates in various geographical regions and terrains.
Therefore, readers should not compare a specific data point with another. In this study, we aim to discuss the WRA process from a broad perspective. Other caveats of this analysis include the potentially inaccurate classification of the data into the proposed IEC framework; the prime focus on P50 rather than P90, which also has a strong financial implication; and the tendency to report extreme losses and uncertainties caused by extraordinary events, such as availability loss and icing loss, in 130 the literature. Our data sources are also only limited to publicly available data or those accessible at NREL. We perform a rigorous literature review from approximately 170 sources, and the results presented in this manuscript adequately display the current state of the wind energy industry. https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License.

Bias trend 135
We identify an improving trend of the mean P50 prediction bias, where the overprediction of energy production is gradually decreasing over time (Fig. 3), and the narrow 95% confidence interval of the regression fit justifies the long-term trend. Such an improving trend is not strictly statistically significant (Fig. 3a), even after removing the studies based on small wind farm sample sizes (Fig. 3b). However, the low R 2 implies that less than half of the variance in bias can be described by the regression, and more than half of the variance is caused by the inherent uncertainty between validation studies that does 140 not change over time. The average bias magnitude also does not correlate with the size of the study, either in wind farm sample size or wind farm year length (not shown). Note that in some early studies, the reported biases measured in wind farm differ from those using wind farm year from the same source; we select the error closest to zero for each independent reference because the bias units are the same (Sect. 2).
145 Figure 3: The trend of P50 prediction bias: (a) scatterplot of 63 independent P50 prediction error values, where R 2 is the coefficient of determination and n is the sample size. Negative bias means the predicted AEP is higher than the measured AEP, and vice versa for positive bias. The black solid line represents the simple linear regression, the dark grey cone displays the 95% confidence interval of the regression line, the light grey cone depicts the 95% prediction interval, the horizontal black dashed line marks the zero P50 prediction error. (b) as in (a), but only for 57 studies that use more than 10 wind farms in the analyses. The vertical violet bars 150 represent the estimated uncertainty bounds (typically presented as one standard deviation from the mean) of the mean P50 prediction errors in 16 of the 57 samples. Table B1 summarizes the bias data illustrated herein.
The uncertainty of the average P50 prediction error quantified by the studies remains large, in which the mean standard deviation is 6.88% of the 16 data sources' reported estimated P50 uncertainty (violet bars in Fig. 3b). The industry started to report the standard deviations of their P50 validation studies in 2009 and it is becoming more common. With only 155 16 data points, we cannot identify a temporal trend of the uncertainty in P50 prediction bias. Even though the mean P50 prediction bias is approaching zero, the industry appears to overestimate or underpredict the AEP for many individual wind projects.

Reasons of bias changes
To correct for the historical P50 prediction errors, some organizations publicize the research and the adjustments they 160 have been conducting for their WRA processes. We summarize the major modifications of the WRA procedure in Table 1.
Most studies demonstrate mean P50 bias improvement over time (Fig. 4), and the magnitude of such bias reduction varies. In two studies, the authors examine the impact of accounting for windiness, which is the quantification of long-term wind speed variability, in their WRA methodologies. They acknowledge the difficulty in quantifying interannual wind speed variability accurately, and their P50 prediction errors worsen after embedded this uncertainty in their WRA process (vertical dash lines 165 in Fig. 4).  Table B2. Table 1: Categories of method adjustments to improve the wind resource assessment process and the respective data sources.
• degradation of long-term meteorological masts.
Consider meteorological effects on power production e.g., • wind shear, • turbulence, • air inflow angle, and • atmospheric stability. AWS Truepower, 2009;Brower et al., 2012;Elkinton, 2013;Johnson, 2012;Ostridge, 2017 Improve modeling techniques e.g., • turbine performance, • wind flow, • wake, • complex terrain flow, • surface roughness, and • wind farm roughness. Elkinton, 2013;Johnson, 2012;Ostridge, 2017;Papadopoulos, 2019 Improve in measurement and reduce in measurement bias e.g., adjust for dry friction whip of anemometers AWS Truepower, 2009;Johnson, 2012;Ostridge, 2017;Papadopoulos, 2019 Correct for previous methodology shortcomings e.g., • loss assumptions, and • shear extrapolation Ostridge, 2017;Papadopoulos, 2019 4 Energy-production loss 175 The prediction and observation of production losses are tightly related to the P50 prediction accuracy; hence, we contrast the estimated and measured losses in various categories and benchmark their magnitude (Figs. 5, 6 and 7). The total energy loss is calculated from the difference between the gross energy estimate and the product of gross energy prediction and various categorical production efficiencies, where each efficiency is one minus a categorical energy loss (Brower, 2012). Of the total categorical losses, we record the largest number of data points from availability loss, and wake loss display the largest 180 variability among studies (Fig. 5). For availability loss, the total observed loss varies more than the total estimated loss and displays a larger range (Fig. 6a). The turbine availability loss appears to be larger than the balance of plant and grid availability losses; however, more data points are needed to validate those estimates (Fig. 6a). Except for one outlier, the turbine performance losses, in both predictions and observations, are about or under 5% (Fig. 6b). Large ranges of environment losses https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License. exist, particularly for icing and degradation losses, which can extremely deplete AEP (Fig. 6c). Note that some of the icing 185 losses indicated in the literature represent the fractional production loss from the abnormal production halts for an extended period in the winter, rather than a typical energy loss percentage for a calendar year. Electrical loss has been assured as a routine energy reduction with high certainty and relatively low magnitude (Fig. 6d). Of all the categories, wind turbine wake results in a substantial portion of energy loss, and its estimations demonstrate large variations (Fig. 6e). The magnitude of estimated wake loss is larger than that of the predicted nonwake loss, which consists of other categorical losses (Fig. 6e). The 190 observed total curtailment loss exhibits lower variability, yet with larger magnitude than its estimation (Fig. 6f). From the eight studies that report total loss, the predictions range from 9.5% to 22.5% (Fig. 6g). We do not encounter any operational strategies loss under curtailment loss in the literature, and thus the subcategories in Fig. 6 do not cover every subcategory in Table A1.     Losses that inhibit wind farm operations have considerable monetary impacts. For example, blade degradation can lead to a maximum of 6.8% AEP loss that translates to $43,000 of annual loss for a single turbine in the IEC Class II wind regime , where the maximum annual average wind speed is 8.5 m s -1 . Generally, the typical turbine failure 210 rate is about 6%, where 1% reduction in turbine failure rate can lead to around $2 billion of global savings in operation and maintenance (Faubel, 2019). In practice, the savings may exclude the cost of preventative measures for turbine failure, such as hydraulic oil changes and turbine inspections.
We categorize two types of energy-production losses additional to the proposed IEC framework, namely first few years of operation and blockage effect (Fig. 7). For the former loss, a newly constructed wind farm typically does not produce 215 this time-specific and availability-related production loss. Regarding the later loss, the blockage effect describes the wind speed slowdown upwind of a wind farm . Wind farm blockage is not a new topic (mentioned in Johnson et al., 2008) and has been heavily discussed in recent years Papadopoulos, 2019;Robinson, 2019;Spalding, 2019). Compared to some of the losses in Fig. 6, the loss magnitude of first few years of operation and blockage is 220 relatively small, where they contribute to less than 5% of AEP reduction per year (Fig. 7).  Table B4.
For trend analysis, we linearly regress every subcategorical energy loss ( Fig. 6 and Table B3) on time, and we only find two loss subcategories demonstrate notable and statistically confident trends (Fig. 8). The measured curtailment loss and 225 the observed generic power curve adjustment loss steadily decrease over time, and the reductions have reasonable R 2 (Fig. 8).
No other reported losses with a reasonable number of data samples display remarkable trends (Fig. C1).  illustrates the 95% confidence interval, R 2 is the coefficient of determination, and n is sample size.
Past research further documents the uncertainties of AEP losses. Except for an outlier of measuring 80% uncertainty in wake loss, the magnitude of the uncertainty of wake loss is analogous to that of nonwake loss (Fig. 9). Moreover, the industry tends to report the uncertainty of wake loss according to the larger number of data sources (Fig. 9). One data source reported that intermonthly variability can alter AEP losses for more than 10% (Fig. 9). Note that the results in Fig. 9 represent the 235 uncertainty of the respective production loss percentages in Fig. 6 and Table B3, rather than the AEP uncertainty.

Energy-production uncertainty
The individual energy-production uncertainties directly influence the uncertainty of P50 prediction. Total uncertainty 245 is the root-sum-square of the categorical uncertainties; the assumption of correlation between categories can reduce the overall uncertainty, and this assumption is typically consultant-and method-specific. (Brower, 2012). Except for a few outliers, the magnitude of the individual energy-production uncertainties across categories and subcategories is about or below 10% (Fig.   10). The energy uncertainties from wind measurements range below 5%, after omitting two extreme data points (Fig. 10a).
The estimated long-term period uncertainty varies the most in historical wind resource (Fig. 10b), which indicates the 250 representativeness of historical reference data (Table A2). Horizontal extrapolation generally yields higher energy-production uncertainty than vertical extrapolation ( Fig. 10c and d). For plant performance, each subcategorical uncertainty corresponds to the respective AEP loss ( Fig. 6 and Table A1). The range of the predicted energy uncertainty caused by wake effect is about 6% (Fig. 10e). The estimated uncertainty of turbine performance loss and total project evaluation period match with those observed ( Fig. 10e and f). Overall, the average estimated total uncertainty varies by about 10%, whereas the observed total 255 uncertainty appears to record a narrower bound, after excluding an outlier (Fig. 10g).
In the literature, we cannot identify all the uncertainty types listed in the proposed IEC framework; hence, the following AEP uncertainty subcategories in Table A2 are omitted in Fig. 10: wind direction measurement in measurement; on-site data synthesis in historical wind resource; model inputs and model appropriateness in horizontal extrapolation; model components and model stressor in vertical extrapolation; and environmental loss in plant performance. 260  Table B6 numerates the production uncertainties.
Similar to energy losses, other types of AEP uncertainties not in the proposed IEC framework emerge. The magnitude of the uncertainties in Fig. 11 is comparable to the uncertainties in Fig. 10. The power curve measurement uncertainty in Fig.   11, specifically mentioned in the data sources, could be interpreted as the uncertainty from the turbine performance loss.  Table B7.
The energy-production uncertainty from air density and vertical extrapolation depends on the geography of the site.
For instance, the elevation differences between sea level and the site altitude, as well as the elevation differences between the mast height and turbine hub height affect the AEP uncertainty . For simple terrain, the vertical 275 extrapolation uncertainty can be estimated to increase linearly with elevation . A common industry practice is to assign 1% of energy uncertainty for each 10 m of vertical extrapolation, which could overestimate the uncertainty, except for forested locations (Langreder, 2017).

Wind speed uncertainty
Energy production of a wind turbine is a function of wind speed to its third power. Considering wind speed, either 280 measured, derived, or simulated, is a critical input to an energy estimation model, the uncertainty of wind speed plays an important role in the WRA process. We present various groups of wind speed uncertainties in the literature herein (Fig. 12).
The bulk of the wind speed uncertainties are about or less than 10% in wind speed. Many studies report estimated uncertainty from wind speed measurement, however its magnitude and discrepancy among the sources are not as large as those from wind speed modeling or interannual variability (Fig. 12). Notice that some of the wind speed categories coincide with the IEC 285 proposed framework of energy uncertainty, and others do not. The absence of standardized classification of wind speed uncertainties increases the ambiguity in the findings from the literature and poses challenges to the interpretation of the results in Fig. 12. We also lack sufficient samples of measured wind speed uncertainties to validate the estimates. https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License. 290 dark purple dot, dark green dot, and dark purple cross represent the mean estimated wind speed uncertainty, the mean observed wind speed uncertainty, and the maximum of estimated wind speed uncertainty from each independent study respectively. The units are in percent of wind speed. For clarity, the grey horizontal lines separate data from each category. Table B8 documents the wind speed uncertainties displayed.
Wind speed uncertainty greatly impacts AEP uncertainty, and the method of translating wind speed uncertainty into 295 AEP uncertainty also differ between organizations. For example, 1% increase of wind speed uncertainty can lead to either 1.6% (AWS Truepower, 2014) or 1.8% increase in energy production uncertainty (Holtslag, 2013;Johnson et al., 2008;White, 2008b). Local wind regimes can also affect this ratio. For low wind locations, AEP uncertainty can be three times the wind speed uncertainty, while such ratio drops to 1.5 at high wind sites . Reduction in wind speed measurement uncertainty of 0.28% could reduce project-production uncertainty by about 0.15% (Medley and Smith, 2019). Using a 300 computational fluid dynamics model to simulate airflow around meteorological masts can reduce wind speed measurement uncertainty from 2.68% to 2.23%, which translates to 1.2 million British pounds of equity savings for a 1-GW offshore wind farm in the United Kingdom (Crease, 2019). https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License.

Opportunities for improvements
Instead of the magnitude of the prediction errors, the uncertainties immersed in the WRA process dominates the 305 accuracy of AEP prediction. Different organizations have been improving their techniques over time to eliminate the P50 bias (Table 1), and as a whole we celebrate such advancements; nevertheless, challenges still exist for validation and reduction of the AEP losses and uncertainties. Even though the average P50 prediction bias approaches zero, the associated mean P50 uncertainty remains at over 6%, even for the studies reported after 2016 (Fig. 3b). For a validation study that involves a collection of wind farms, such uncertainty bound implies that sizable P50 predication errors for particular wind projects can 310 emerge. In other words, statistically, the AEP prediction is becoming more accurate yet is imprecise. Moreover, from an industry-wide perspective that aggregates different analyses, the variability on the mean P50 bias estimates is notable, which obscures the overall bias-reducing trend (R 2 below 0.5 in Fig. 3). Specifically, the magnitude of the 95% prediction interval at over 10% average P50 estimation error (Fig. 3b) suggests a considerable range of possible mean biases in future validation studies. Additionally, the uncertainties are still substantial in specific AEP losses ( Fig. 9), AEP itself (Figs. 10 and 11), and 315 wind speed (Fig. 12). Therefore, the quantification, validation, and reduction of uncertainties requires the attention of the industry collectively.
To reduce the overall AEP uncertainty, the industry should continue to assess the energy impacts of plant performance losses, especially those from wake effect and environmental events. On one hand, Wake effect, as part of a grand challenge in wind energy meteorology (Veers et al., 2019), has been estimated as one of the largest energy losses (Fig. 6e). The AEP loss 320 caused by wake effect also varies, estimated between 15% and 40% ( Fig. 9), and the unpredictability of wakes contributes to the AEP uncertainty on plant performance (Fig. 10e) and the wind speed uncertainty (Fig. 12). Although the industry has been simulating and measuring the substantial energy depletion caused by wake effect, its site-specific impact on AEP for the whole wind farm as well as its time-varying production impact on downwind turbines remains largely uncertain. From a macro point of view, compared to internal wake effect, external wake effect from neighboring wind farms is a bigger known unknown 325 because of the lack of data and research. On the other hand, environmental losses display broad range of values, particularly from icing events and turbine degradation (Fig. 6c). In general, the icing problem halts energy production in the short run, and blade degradation undermines turbine performance in the long run. Diagnosing and mitigating such substantial environmental losses would reduce both loss and uncertainty on AEP. Overall, the prediction and prevention of environmental events are critical, and the production downtime during high electricity demand can lead to substantial financial losses. 330 Additionally, the industry recognizes the role of remote-sensing instruments in reducing the uncertainty of energy production and wind speed from extrapolation, such as profiling lidars, scanning lidars, and airborne drones (Faghani et al., 2008;Holtslag, 2013;Peyre, 2019;Rogers, 2010). The latter can also be used to inspect turbine blades (Shihavuddin et al., 2019) to reduce unexpected blade degradation loss over time. Industrywide collaborations such as the International Energy Agency Wind Task 32 and the Consortium For Advancement of Remote Sensing, have been promoting remote-sensing 335 implementation in WRA.
Experts in the field have been introducing contemporary perspectives and innovative techniques to improve the WRA process, including time-varying and correlating losses and uncertainties. Instead of treating energy loss and uncertainty as a static property, experts have studied time-varying AEP losses and uncertainties , especially when wind plants produce less energy with greater uncertainty in later operational years (Istchenko, 2015). Furthermore, when different 340 types of energy-production losses or uncertainties interact and correlate with each other, the resultant compound effect can become larger than the total value from a linear (Brower, 2011) or root-sum-square approach (Istchenko, 2015). For example, an icing event can block site access and decrease turbine availability, and even lead to longer-term maintenance problems (Istchenko, 2015).
Overall, more observations are necessary to validate the estimates listed in this article. In this article, the ratios 345 between the measured and predicted values are 1 to 1.9, 2.3, and 7.3, for energy loss, energy uncertainty, and wind speed uncertainty, respectively. The small number of references on measured uncertainties indicate that we need more evidence to further evaluate our uncertainty estimates. Public disclosure of summary statistics would continually improve insight into the WRA process and increase the AEP estimation accuracy for the industry as a whole, even when the data are site-specific or organization-centric. 350

Conclusions
In this analysis, we compile and present the ranges and the trends of predicted P50 (i.e. median annual energy production) errors, as well as the estimated and observed energy losses, energy uncertainties, and wind speed uncertainties embedded in the wind resource assessment process. Although the mean P50 bias demonstrates a statistical trend that it approaches zero over time, the notable uncertainty of the mean prediction error reveals the imprecise prediction of annual 355 energy production. The dominant effect of prediction uncertainty over the bias magnitude calls for further improvements on the prediction methodologies. To reduce the mean bias, the industry experts have made method adjustments in recent years that minimize the energy-production prediction bias, such as the applications of remote sensing devices and the modeling advancements of meteorological phenomena.
We present the wind energy production losses and uncertainties according to the proposed framework by the 360 International Electrotechnical Commission (IEC) 61400-15 working group. Wake effect and environmental events undermine wind plant performance and constitute the largest loss in energy production, and validating the wake and environmental loss predictions requires more field measurements and detailed research. Moreover, the variability of observed total availability loss is larger than its estimates. Meanwhile, the decreasing trends of measured curtailment loss and observed generic power curve adjustment loss indicate the continuing industry effort to optimize wind energy production. Additionally, different 365 categorical energy uncertainties and wind speed uncertainties demonstrate similar magnitude, with a majority of the data below 10%. More observations are the solution to better understand and further lower these uncertainties. https://doi.org/10.5194/wes-2020-85 Preprint. Discussion started: 10 July 2020 c Author(s) 2020. CC BY 4.0 License.
In our findings, we highlight the potential future progress, including the importance of accurately predicting and validating energy-production uncertainty, the impact of wake effect, and innovative approaches in the wind resource assessment process. This work also includes a summary of the data collected and used in this analysis. As the industry evolves 370 with improved data sharing and rigorous research, we will increasingly be able to maximize energy production and reduce its uncertainty for all project stakeholders.

Data availability
Appendix B includes all the data used to generate the plots in this manuscript. 375  For the P50 prediction error, Fig. 3 and Fig. 4 use the data from Table B1 and Table B2, respectively. For the various categories and subcategories of losses, Figs. 5, 6, 8 and C1 portray the values in Table B3. Fig. 7 illustrates the losses outside of the IEC proposed framework listed in Table B4. Fig. 9 summarizes the uncertainty of production loss percentages in Table   B5. Figs. 10 and 11 represent the AEP uncertainty data included in Table B6 and Table B7, respectively. Fig. 12 displays the 390 wind speed uncertainty data in Table B8.  Fig. 3. The "Bias (%)" column represents the average P50 bias, where a negative number indicates an overestimation of actual energy production. The "Uncertainty (%)" column usually illustrates one standard deviation from the mean.    Year