We collected preconstruction EYAs or key data from them over the course of multiple years. We built on the work of by working with many of the same wind plant owners and operators who provided EYA estimates to collect plant-level MOR containing monthly gross energy production, curtailment losses, and availability losses. Our data collection spanned MORs for 94 wind plants in six countries, covering 115 EYAs for six wind plant owners and 19 specified consultants. Of those, 76 were in the United States (72 with corresponding EIA data), and 70 had a COD of 2011 or later – a reversed COD bias from where only 23 of the 56 plants had a COD of 2011 or later. In total, we collected 707 complete wind plant years of data to conduct this study.

Figure  shows a variety of key project characteristics collected from both the EYAs and the MORs and their distributions to highlight the breadth of plants represented in this study. The collected EYA data were provided as either the full document or as a data sheet of the essential information, so the project metadata were not universally available, meaning there are fewer results for some variables than others. There are some projects with CODs prior to 2010, but most plants came online between 2012 and 2017. Additionally, the provided project MORs tended to contain 3.5 years or between 8 and 10 years of operational data as opposed to 2 years or less, providing sufficient data for modeling.

For each wind plant, we also collected the ERA5 and MERRA-2 reanalysis datasets for long-term wind resource corrections. The MERRA-2 data are made available at 0.5° × 0.625° resolution, and the ERA5 data are available at 0.25° resolution, so we identified the closest latitude and longitude to each plant's owner-provided centroid and used the atmospheric data corresponding to that point. For each point, we downloaded the hourly wind speed, wind direction, temperature, humidity, and surface air pressure and aggregated these readings to the monthly mean reading for each variable. Figure  demonstrates the high degree of correlation between gross energy production (discussed further in Sect. ) and the MERRA-2 and ERA5 wind speed data as measured by the Pearson correlation coefficient. While MERRA-2 demonstrates a higher average Pearson correlation coefficient (0.83) than the ERA5 data (0.80), their median values show a reverse trend, with ERA5 having a 0.90 median correlation and MERRA-2 having a 0.88 median correlation. The close alignment of reanalysis wind speed and gross energy suggests that for most projects, the reanalysis wind conditions are suitable for analysis.

We found that the five projects with a correlation coefficient less than 0.4 represented our most extreme outliers in the operational analysis results, so we removed them from the study. We also found that each of these projects either had upstream plants introduced following their construction, causing external wake losses, or is located in complex terrain, such as downwind from a mountainous region or among rolling hills. Further, reanalysis wind conditions were not able to fully account for complex terrain near ground level, as they are modeled from atmospheric conditions, nor could they account for every location within their coarse-grid structure. While a coarse grid means the reanalysis conditions are more or less representative of the data for that plant, there is no correlation between distance from the modeled grid point and the plant's centroid for either the poorly matched plants or the data set writ large.

The first set of analyses assessed the similarities between the P50 biases using the annual net energy provided through the EIA and MORs. To annualize the energy production, we summed each consecutive 12-month period starting from the COD, excluding any partial years at the end of the time series. We then calculated the percentage difference between each annualized reading and the EYA-estimated P50.

To compute the long-term-corrected P50, P90, and uncertainty estimates, we utilized the Monte Carlo AEP analysis method in the National Laboratory of the Rockies' OpenOA software , as described in , outlined below, and shown as a flowchart in Fig. . The method includes the following steps:

Compute the density-corrected wind speeds for each reanalysis product, and average the hourly wind speed, wind direction, and temperature to a monthly level to align with the operating data.

The long-term correction is similar to the approach taken in , with a few essential differences in the data used and how they are filtered. The reanalysis datasets were of a different vintage, and only the wind speed data were used in . No curtailment or availability data are provided through the EIA, so there was no gross energy calculation, nor was a 30 d normalization applied in . The outlier filtering in was based on the standard deviation from the mean, different from the more robust approach taken in this study.

For each iteration in the Monte Carlo analysis, we randomized the usage of the variables listed in the top section of Table . The bottom section represents variables that are varied between analyses but held constant within an analysis. Additionally, we defined what OpenOA uses as a default value, how each simulation of the Monte Carlo analysis applies that value, and what variations of the Monte Carlo analysis we ran. For the simulation's parameter choice, OpenOA randomly chooses one of the input variables, randomly samples from a normal distribution using the input value as the distribution's center or a uniform distribution with the input value acting as the lower and upper bound, or does not vary the parameter's value at all between runs. The choice for the default values was not documented when the model was originally built; however, it is expected that the uncertainty in energy losses will be higher than the net energy, as the losses are derived based on the estimated potential energy production rather than directly measured at the revenue meter. For the combined loss threshold parameter, we found that typically between 4 and 16 months will be removed from an analysis, representing as little as 9 % and as much as 28 % of the available records. Despite removing many records for some plants, this filtering significantly reduced the overall long-term-corrected uncertainty. We used the Huber loss method because it is more robust to outlier data , such as high curtailment or low-availability months, which are not representative of regular plant operations. For each combination of model configurations, we ran 5000 Monte Carlo iterations, including bootstrapping the monthly operational data and randomly sampling different analysis parameters to understand the uncertainty in the operational assessments for each plant. The rightmost column of Table  represents the input parameter variations that we ran to identify the analysis settings that yield the most accurate and reliable estimates.

To select a representative model for each of the combinations, we chose the lowest total rank by comparing the R2 of each bootstrapped model and the total uncertainty (σAEP/μAEP). Using each of these metrics, we summed the ranking of the lowest total uncertainty and maximum median R2. We selected the lowest total rank model as the representative model for each plant, as it would have minimized uncertainty and maximized how well the model fits the test data. Using the 5000 iterations, we computed the P50, P90, and uncertainty of the AEP estimations.

When computing the results, we also gradually filtered out time periods and projects to arrive at a more accurate assessment of each wind plant and the industry more broadly. We removed the first year of operational data to account for non-representative issues that arise in the first year of operations. Then, we removed projects with a COD earlier than 2011 to account for changes in EYA methodology. Finally, we considered the combination of both filters.

As a first step, we confirmed the validity of the approach to rely on monthly EIA net energy data to determine AEP from the original study . This step ensures that we are able to make a reasonable comparison because there is not a specification for how the EIA data are provided by plant owners or what processing steps (if any) are performed prior to being provided to end users. To do this, we compared the EIA net energy output to the MOR net energy, where the EIA identifier was provided with the EYA data. As shown in Fig. , the correlation between the MOR energy production received from plant owners and operators and the energy reported by the EIA was strong (ρ=0.98). The line of best fit for the EIA and MOR data yields y=0.99x+0.30, indicating that the EIA data are highly consistent with those reported by wind plant owners. This result indicates that the methods used by to set a baseline for P50 bias remain valid overall, though significant discrepancies exist for individual months.

We then compared the EYA AEP P50 bias for both the net EIA energy and the net MOR energy without a long-term correction to the bias resulting from , as shown in Fig. . We took the best-case scenario, where we removed the first operational year of the wind plant and removed plants with a COD prior to 2011. This approach is similar to the approach taken in , which reported a 5.5 % overprediction when using a long-term correction. Here, the average P50 biases for the uncorrected EIA and MOR data are -14.9 % and -11.5 %, respectively. However, there is a large variation in the bias for individual plants (19.3 % for EIA data and 13.5 % for MORs), with positive biases (underprediction) for many plants. As expected with uncorrected values, there is a significant number of outliers for AEP overprediction, particularly for the EIA data, which has 55 % more wind farm years of data. In general, this finding indicates that there is still an overprediction of AEP P50 in the EYAs, given that it is at least double the long-term-corrected value reported in .

The next phase of the analysis was to estimate the long-term-corrected AEP and uncertainty. As described in Sect. , we found each project's ideal analysis settings; we report on the long-term-corrected results in this section. We found that 84 % of the best combinations included the use of the Huber loss algorithm outlier detection and filtering, 74 % used the reanalysis temperature data, and 33 % used the reanalysis wind direction. For the uncertainty loss thresholds, we found that 21 % of the best combinations used the OpenOA default value, and another 46 % used thresholds with a minimum of the 80th percentile. Combined, 47 % of the best model combinations also used the outlier detection and temperature data but no wind direction data.

In Fig. , we demonstrate the overall long-term-corrected P50 bias by COD where there is modest yet inconsistent improvement in the P50 bias between 2011 and 2019 relative to projects prior to 2011. However, in 2019 and later, the P50 bias worsens, though it is worth noting that there are fewer projects and therefore less operational data in our dataset in this COD grouping, so the results may not be representative. Even with later COD P50 bias increases, the post-2010 COD wind plants exhibit an improving P50 bias, -7.4 % vs. -6.6 %, as shown in the text box of Fig. , which is in line with the baseline result trends from . To further test whether or not there is a relationship between CODs prior to 2011, we ran a two-sided Welch's t test for years prior to 2011, 2011 through 2022, and 2011 through 2019 to account for the effect of small sample sizes in projects coming online later against the P50 bias of all samples and those dropping the first year of operational data. For all combinations, a p value greater than 0.2 was yielded, demonstrating that there is not a strong relationship between the EYA estimation methods prior to 2011 and those used starting in 2011.

We similarly found that there was a possibility of a relationship between states, provinces, or small countries with at least four plants and their long-term-corrected P50 bias. In Fig. , it can be seen that states such as Oklahoma, New Mexico, Nebraska, and New York tend to have above-average P50 biases compared to some states like Texas and California that have many below-average P50 biases. When running a one-way ANOVA , we found a p value of 0.003, so we ran Tukey's test against the pairs, only finding a significant difference between California and Oklahoma (p value 0.03) and California and New Mexico (p value 0.019). However, when we ran a one-way ANOVA to further inspect the relationship between original equipment manufacturers (OEMs) and long-term-corrected P50 bias, we found no relationship, with a p value of 0.043 for OEMs with at least four data points. For other key distinguishing variables, there were not enough data available relative to the total plants analyzed to warrant further testing.

We also demonstrate the P50 bias for each consultant in Fig.  to determine if there were any significant variations across the 19 consultants. Most deviation from the average P50 bias occurred in consultants with a small sample size or where the spread was large. In particular, consultants 1, 3, 6, and 8 had a large spread in their individual estimates; only consultant 5 demonstrated a high level of precision in their estimates. The unspecified grouping, corresponding to no consultant name being provided, yielded the largest sample size with some of the smallest spread in EYA P50 bias. Nevertheless, when performing a one-way ANOVA test for all consultants with at least three data points, we found a p value of 0.7263, confirming there is no meaningful difference between consultant predictions of P50 energy production.

Using the 5000 simulations, we then computed the P50 and P90 as their respective 50th and 10th percentiles and the uncertainty as σAEP/μAEP. The boxplots in Figs. , , and demonstrate the distribution of the per-project spread of the P50, long-term P90, and long-term uncertainty bias, respectively, for each of our four down-selection criteria. Compared to the results, we can see a slightly worsening P50 bias. We observe that the short-term P90 bias is positive, indicating that the P90 of the estimated operational AEP exceeds the EYA AEP P90 value on average. However, this result does not necessarily suggest that the EYAs use overly conservative means for estimating high-probability energy production levels; to reach this conclusion, we would need to demonstrate that the estimated operational AEP exceeds the EYA P90 predictions 90 % of the time.

As indicated, the uncertainty bias for the long-term-corrected energy estimates relative to the 1-year EYA uncertainties appears to suggest that the EYAs are routinely conservative estimates of uncertainty, with the average bias in the -52 % to -48 % range. It should be noted, though, that the uncertainties for the operational AEP estimates and the EYA AEP predictions are not defined equivalently. The operational AEP uncertainties represent the uncertainty in the long-term AEP based on realized energy production during an initial period of operation. EYA estimates, on the other hand, represent the uncertainty in the predicted AEP before the plant is built. The EYA estimates are expected to have a higher degree of uncertainty because there are a significant number of individual model uncertainties and potential sources of loss to consider when predicting energy production prior to building a wind plant.

To better understand the uncertainty and prediction errors, we also computed the prediction error of the EYA P50 and P90 estimates for each plant relative to each Monte Carlo iteration's AEP estimation for the corresponding plant (percentage difference between the EYA reading and each iteration's AEP estimate), as shown in Figs. –. In Fig. , which shows the distribution of P50 prediction errors including IAV, we smooth the error histogram to highlight the centering around the mean values. Importantly, we found that the standard deviation of the sample errors ranged from 11.9 % to 13.7 %, which closely aligns with the average 1-year uncertainty in the corresponding EYAs of 10.5%±2.6%, shown on the right side of Fig. . However, when comparing the EYA long-term uncertainty to the P50 prediction errors that exclude IAV, as shown in Fig. , our standard deviation ranges from 11.4 % to 13.1 %, which is much larger than the corresponding EYA long-term uncertainties of 7.5%±2.1%. As opposed to the direct comparison in Fig. , these results indicate that the EYA uncertainty, especially the long-term uncertainty, is being underestimated.

In line with our project-level P90 estimate comparisons (though more positive), the individual P90 prediction errors for each Monte Carlo iteration center around positive values (Fig. ); however, the 10th percentile of the prediction errors is roughly -10 % for all cases, demonstrating that EYA P90 estimates are not conservative enough for both the 1-year and the long-term P90 predictions. Ideally, the 10th percentile should be 0 %, meaning the realized operational AEP exceeds the EYA P90 predictions at least 90 % of the time. Note that we can still see the impact of outlier data in the spread of the distribution and the presence of additional peaks.

Considering the continued existence of EYA P50 bias, we then calculated the true probability of exceedance value (PXX value) of the EYA-reported P50 based on the long-term-corrected distribution and the PXX value of the long-term-corrected P50 with respect to the EYA distribution statistics. We derived the respective PXX values by first finding the number of standard deviations by which the estimated P50 value was from the comparative mean, x=(P50ltc-μeya)/σeya. Then, we calculated the probability based on the cumulative distribution function of the number of standard deviations and subtracted it from 1 to ensure the overestimates yield a lower PXX value than the P50. We followed the same process for the long-term-corrected P50 with respect to the EYA AEP distribution by computing the number of standard deviations (x=(P50eya-μltc)/σltc), calculating the probability from the cumulative distribution function, and subtracting it from 1 to account for the overprediction bias. The results of these calculations for all plants are shown in Fig. . We found that the EYA P50 represented a P22 on average, with a standard deviation of 30 percentage points, and that the long-term-corrected P50 represented a P72 with respect to the EYA AEP distribution, with a standard deviation of 26 percentage points. The two distributions roughly mirror each other and form clusters, where the EYA P50 is significantly overestimated. For the long-term-corrected distribution, the EYA P50 is -0.95 ± 1.61 standard deviations from the long-term-corrected mean, with the bottom 25 % of the distribution being at most -1.64 standard deviations. Similarly, the long-term-corrected P50 is on average 2.16 ± 2.84 standard deviations from the mean EYA-estimated AEP, with the top 75 % of the data being at least 3.99 standard deviations from the mean. Table  shows that when comparing the subset of plants with CODs after 2010 and when removing the first year of operational data, the results improve slightly. For example, the EYA P50 PXX value relative to the long-term-corrected distribution corresponds to a P24 compared to a P22 when viewing all plants with all available operational years. Similarly, the long-term-corrected P50 based on the EYA distribution shifts from a P72 to a P70 when filtering out plants with CODs prior to 2011 and removing the first year of data.

Another way to understand what the P50 and P90 values represent based on the long-term-corrected distribution is to compute what percentage of the P50 and P90 prediction errors (as shown in Figs. –) is greater than 0 %. In Table  we demonstrate the close alignment these prediction error results have with those presented in Fig.  and Table . The EYA P50 corresponds to a P19 to P20 based on the long-term-corrected AEP distributions without IAV, depending on the scenario, and the long-term EYA P90 corresponds to a P65 to P69, depending on the scenario. For both the short-term P50 and the P90, the PXX values improved with the long-term-corrected AEP distributions, with IAV yielding a realized P22 to P24 for the EYA P50 value and a P75 to P77 for the EYA 1-year P90 value, depending on the scenario.

Finally, because EYA availability and curtailment loss estimates were provided for 36 projects, we also computed the P50 value for each of the long-term-estimated operational availability and curtailment losses in the Monte Carlo simulations by project, as shown in Fig. . We have excluded a detailed comparison between the EYA-estimated curtailment losses and operational curtailment losses, as 31 of the EYAs provided an EYA-estimated 0 % curtailment loss, and only two EYAs estimated 0 % availability losses, which were removed from the comparison. As such, the results of the availability comparison are shown in Fig. . We found that the average difference between the operational P50 values and EYA estimates for availability losses was 1 percentage point, indicating a slight underestimation of availability loss; however, the median value is 0.22 percentage points, and the standard deviation is 3.63 percentage points. Taken together, this suggests an inconsistent understanding of availability across the board, even if it contributes a minimal amount to the AEP P50 biases. Further, it should be noted that availability data were provided in mixed formats – energy lost and time-based availability – so the comparisons are not wholly on the same basis.

The curtailment P50 bias only consists of seven comparisons, so the underestimation of losses by 13 ± 23 percentage points may not be representative of reality. In fact, many of the EYAs with curtailment loss information state outright that situations where the energy grid may force the power plant to stop sending power are not considered, so the comparison of realized curtailment and EYA-estimated curtailment likely provides little insight without further details or delineations.