General comments

Dear Barber et al.,first of all I'd like to  say that I completely agree with the underlying motivation for this work. It is important to focus on how inaccuracies in wind speed predictions translate in energy and show that it’s a complex matter with a lot of variables (like wind direction, wind speed distribution, shear…) impacting the final results. It is very easy to be “right for the wrong reason”. This is why so many studies focus on wind speed in one direction. The aim of those studies is to improve flow models and the only thing the scientific community can do is to design and conduct experiments while minimizing the amount of variables that could make the interpretation of results impossible.

Specific comments

As I said, the motivation and scientific question behind this work is very relevant. However I notice some decisions in your implementations which confuse me a bit. I don’t understand if you decided to obtain results by including some of the uncertainty components of the AEP assessment process on purpose or you tried to avoid them. For example, you decided to base your results and comparison on long term corrected wind speed data being hopefully aware of  increased uncertainty that you take with you during the AEP comparisons. How do you know that certain differences in AEP are not due to  a long term correction far from perfect? Ok, but let’s say that we want to include  long term correction uncertainty on purpose to see how it  translates in AEP, why did you decide to avoid vertical extrapolation from the measurement height to the hub height. Uncertainty of vertical extrapolation is another important source of uncertainty in the AEP assessment process. I would have avoided or included both.

I think that using wind speeds measured by nacelle anemometers for the purpose of the article is quite a stretch. It would require quite a lot of analysis (flow inclination, rotor speed, pitch settings) in order to make the statement “but the ration will still be valid”. I would have not used site 4.

I am a bit puzzled how you can obtain a non zero error when you compare the wind speed at the calibration location when you consider one height only. (figure 2)

My last specific comment is that you used given power curves for different wind turbine models for different sites. I assume that each of them are different with respect to generator/rotor area ratio and they will have different rated wind speed. Would it have been better to use only one for all sites so that the results are not affected by the power curve steepness? Given power curves are also tricky because they almost always need site specific adjustment. One way to make the study power curve independent would have been to use WPD (wind power density) as a metric instead of AEP.

Technical corrections

Line 51: It would be nice to mention that flow calculations in WindPro can be based on WAsP CFD (EllipSys3D) or WAsP linearized flow model (IBZ). I assume you used the IBZ model.

Line 56: WindSim can simulate more directions . But 12 were used for this analysis.

Line 107: I am missing some details of the MCP method used (linear least square, matrix, etc) and some metric for the reader to evaluate the accuracy of such a step (maybe a table with R^2,  measured and long term corrected mean wind speed?).

Line 159 and Table 1: i don’t think you explain the meaning of the abbreviation HSE or OST before using them

Line 260: Did you apply RIX correction? It’s quite known that WAsP IBZ results need RIX correction for complex sites which will make a difference in terms of accuracy.

General Summary

The authors conclude that in complex terrain, the wind energy community should consider the impact of wind direction to overall annual energy production in the wind resource assessment process. The authors display the results from 7 sets of wind flow models and 5 simulation locations. This sensitivity study leads to some interesting results, yet the authors should elaborate on their thought process and their findings. The authors also omit some key details to support their arguments and to uphold scientific reproducibility.

Major Comments

1. Section 2.1.1: It would be very useful to use a table to summarize the similarities and differences among the wind models. For example, both WF-1 (WindPro) and WF-2 (WindSim) calculate TI directly using the input met mast data. For a sample table, the columns can be WF-X’s, a row can be “TI calculation”, and the respective WF-1 and WF-2 elements can contain the same information. Such table can be constructed in many ways, at the discretion of the authors. Moreover, in a separate table, the authors should also discuss the pros and cons of different wind models, such as computational time (relating to line 84), labor required to set up a model run, recommended resolution, specific assumptions made in the code, when would an analyst use one model over the other, etc. This would guide the readers on how the wind models differ.
2. Section 2.2: Similarly, a table to summarize the 5 (or 4, since Site 5 is confidential) simulated sites would be useful. The table shall include heights of measurements, the duration of measurement period, the number of turbines, etc. For Site 2, it is confusing that its measurements seem to be available at multiple heights, and in Section 3.2 only the measurement of 1 height is discussed.
3. Lines 257 to 259: Can the authors explain why the validation errors are substantial? Given the model errors are very similar across the models (except for WF-1), what insights can we derive from this? Along the same line, Site 1 does not seem to be the outlier, because Sites 3 and 5 record validation errors of similar magnitude.
4. The manuscript lacks consistency in analyzing the workflows and the sites. In lines 218 to 219, the authors reasoned why Site 4 does not need long-term extrapolation. In lines 314 to 315, the authors ignore the importance of vertical extrapolation in a project that focuses on complex terrain, which does not sound convincing. The authors should discuss under what circumstances a typical process is skipped for a site and explain why. Given the specific treatments each site requires, summarizing the information using a graphic or a table would be useful.
5. Wind direction is emphasized in the Abstract, but among the plots and tables throughout the manuscript, only the wind roses in Figure 1 mention wind direction. For example, lines 318 to 321 discuss AEP differences among wind direction sectors, without referring to any plots or tables to support the arguments. Similar problems can be seen from lines 328 to 331, from lines 338 to 342, from lines 351 to 357, and from lines 387 to 389. Moreover, the authors should emphasize the role of wind direction sectors in AEP calculation in more detail earlier, in which its role is not introduced until Section 2.1.5. The authors include and illustrate the (wind-direction) weighted wind speed results in some parts of the paper, but the current analysis does not fully support the arguments made in the text.
6. Lines 369 to 379 contain the key message of the paper. The authors should also discuss which parts of the WRA process in their case studies that lead to the low correlation between wind speed error and AEP error. What should readers focus on among all the steps in the WRA process? Which steps of the WRA process are embedded with the most sensitivities?
7. Each panel in Figure 12 consists of few data points, and the argument of low correlation between wind speed error and AEP error is partially a product of the lack of data samples. For instance, the authors fit a linear regression with only 4 data points in Figure 12 (g) through (j). Strictly speaking, such technique and visualization does not treat statistics properly. The authors should address the issue of low data samples in the text. One alternative is to examine the correlation between wind speed error and AEP error by combining the data across the 5 sites.
8. Overall, the manuscript needs a careful check on copyediting: line 137 uses “1 ms^-1” and line 150 uses “1 m/s”. The naming convention of the wind sites and model runs is also not uniform. For example, WF-5aT is used in lines 345 and 352. Is it equivalent to WF-5b, which is only found in line 86 throughout the manuscript? The reference style of the citation is sometimes incorrect, as seen in lines 26 and 170.

Minor Comments

9. Line 11 to 12: This sentence is confusing, please consider rephrasing it.
10. Line 17: The brackets are not necessary.
11. Line 23: The authors should also briefly explain what the steps, data types, and organizations are.
12. Line 27 to 28: Why does the full name of CREYAP use double quotes but ‘complex’ (line 20) and ‘workflows’ (line 46) use single quotes? Please be consistent.
13. Line 34 to 35: Use “In their work” instead? “In this work” can be interpreted as the work done in your manuscript.
14. Section 2.1.1: What is the default or available number of wind direction sectors for WF-3 and WF-4?
15. Line 85 to 87: What are the differences between WF-5a and WF-5b?
16. Lines 94 to 95: WF-7 can use more descriptions.
17. Lines 99 to 102: Consider splitting the sentence into two.
18. Line 99: Which of the WF-X’s are counted as CFD simulations? Is WindSim considered as one?
19. Line 128: What is “speed-up factor”? Is it simply the wind speed difference between the validation location and the calibration location?
20. Line 142: What is “10/60 hours”?
21. Line 153: Who are the research partners?
22. Line 158: Is the “less than 5% variation” in terms of power, energy/AEP, or wind speed? Do the authors mean “more than 5%”? This sentence somewhat contradicts with lines 161 to 162 of “5%”.
23. Figure 1: The legends of the map are too small, and the dot colors are blended with the topography color scheme.
24. Lines 214 to 216: This is a critical assumption and needs more attention. Did the authors look at the turbine availability or operation log to verify such assumption?
25. Line 221 and 227: How about WF-7?
26. Table 2 to 6: The “Wind model” row is not necessary, as they are explained in Section 2.1.1.
27. Table 4: Is there a reason why WF-7 is only applied for 1 case?
28. Line 233: Do the authors mean 2021 instead of 2001? Also seen in lines 291 and 309.
29. Figure 2: The authors can consider using the same y-axis scale for plots (a) and (b).
30. Line 248: How many measurement heights were used and what were the heights? Same for lines 250, 254, and 255.
31. Line 345: What is the difference between WF-4 and WF-4T? Why is WF-4T used here but not WF-4? Does WF-4T relate to Calculation 4 or 4.1 in Table 1?
32. Figure 4: Can the authors explain why the calibration errors are so low for WF-1, WF-2, WF-3, and WF-6? Similar patterns are seen in Figures 5(a) and 6(a).
33. Line 297: Do the authors mean “time series at the validation location”?
34. Lines 365 to 367: This sentence is vague. Please explain what the “wide range of different effects” are.
35. Lines 369 to 370: This sentence is confusing. How did the authors conclude “absolute wind speed has a larger effect on AEP accuracy” based on “weighting based on wind speed frequency does not change the correlation between wind speed errors and AEP errors”? The authors need to explain their logic more.
36. Line 378: Correlation between what?
37. Figure 12: The axes labels should be “AEP errors” and “Wind speed errors” to avoid confusion.