SGD is built upon the steepest descent algorithm. Early SGD algorithms added a moving average term (sometimes referred to as “momentum”) to avoid spurious oscillations . The conventional Adam SGD algorithm uses two moving averages: one of the gradient and one of the squared gradient. The ratio of these moving averages is used to determine the search direction. SGD algorithms are often combined with a learning rate scheduler, where the step size of the gradient descent is gradually decreased, allowing the optimization algorithm to hone in on the best solution. While the Adam algorithm is already designed to dynamically change the step size, including a learning rate scheduler can further improve the performance. The conventional Adam algorithm is designed for unconstrained optimization algorithms. In the following, we extend the algorithm to allow for deterministic constraints, which is a case that is common in mechanical engineering and unusual in the context of training neural networks. The basic idea is to aggregate the constraints to a penalty term with units that are consistent with the objective. The penalty term is designed so that, initially, the penalty gradients are of similar magnitude to the AEP gradients and so that the penalty gradients overwhelm the AEP gradients as the optimization continues. The SGD algorithm is shown in Algorithm 1, where s0 is the initial turbine position; i is the iteration number; αi is referred to as the constraint multiplier; γ(s) is a penalty function; P(s,u∞(k),θ(k)) is the wind farm power associated with the inflow speed and direction, u∞(k) and θ(k); K is the number of samples employed in each SGD iteration; β1 and β2 are constants; T is the number of SGD iterations; S is the learning rate scheduler; and ηi is the learning rate. By default, the early stopping option is false.

The spacing between turbines is enforced using a penalty term, 7γs=∑∀i,j>iminxi-xj2+yi-yj2-NDD2,0.

Similarly, the distance outside of boundaries is enforced using a penalty term. When considering square wind farms this penalty term is defined as γb=∑i=1Ntmaxxi-xub,02+maxxlb-xi,02+maxyi-yub,02+maxylb-yi,02, and, in the case of circular boundaries, it is defined as 8γb=∑i=1Ntmaxxi2+yi2-R,02, where Nt is the number of wind turbines.

The total penalty, γ, is defined as the sum of these two penalty terms, 9γ(s)=γs(s)+γb(s). The gradient of the penalty term, γ, is scaled before being added to the negative gradient of the AEP using the scaling factor, αi.

In Algorithm 1, the learning rate (ηi), constraint multiplier (αi), number of SGD iterations (T), and the samples per SGD iteration (K) are all free parameters. These parameters can be optimized to perform well for individual wind farm optimization problems. But there is no guarantee that these particular parameters will perform well for other wind farm problems – and this meta-optimization can be expensive. In the machine learning community, these parameters are sometimes optimized using evolutionary, grid search, or Bayesian optimization approaches . In addition, it is common to schedule the learning rate to decay as the optimization proceeds .

We propose a method for setting free parameters to ensure that all units are consistent. The only free parameters we manually set are the number of optimization iterations and the number of power samples per iteration. The optimization generally becomes more accurate and more expensive as these parameters increase, and users are free to balance this trade-off as they see fit. Our formulation does not guarantee that all intermediate solutions satisfy the constraints, especially in the beginning of the optimization. The constraint multiplier begins on a comparable scale to the AEP and is scheduled to increase so that the constraint gradients overwhelm the AEP gradients as the optimization progresses. The number of iterations, T, can be based on a prescribed computational budget.

We initially attempted this approach using the widely used default parameter values in the original Adam algorithm, β1=0.9 and β2=0.999. The parameters can be thought of as adding momentum to the moving averages of the gradient and squared gradient, m and v. We found that these default values gave too much emphasis to gradients from the penalty function, launching the turbines away from the boundaries in a dramatic fashion. Instead, we suggest the parameters β1=0.1 and β2=0.2, which encode a shorter memory of the presence of the penalty. With these new default parameters, and the learning rate defined below, we observed successful convergence for a wide variety of test cases.

The learning rate, ηi, can be interpreted as converting m^/v^ (with unity units) to distance (units of m). In this study, the learning rate is scheduled to decay according to 10S(t=0)=η0S(t=T-1)=ηT, where T is the number of optimization iterations, η0 is the initial learning rate, and ηT is the scheduled final learning rate. This final learning rate can be thought of as a solution tolerance for the design variables. In this study, we set ηT=0.1 m.

The initial learning rate, η0, is based on a length scale parameter, L, which corresponds to a reasonable initial step size for the optimization. By setting the initial learning rate according to 11η0=L=D/5, where D is the turbine rotor diameter, we encourage the turbines to move at most L distance every optimization iteration.

The learning rate is scheduled to decay as 12Sη0,δ,t=η0∏i=0t11+iδ, where δ is a parameter that controls the learning rate length, such that the final learning rate is ηT. The parameter δ is numerically set as 13δη0,ηT,T=argminδ|ηT-Sη0,δ,T|.

The constraint multiplier, αi, can be interpreted as converting the gradient of constrained square distances (in units of m) to AEP gradients. The initial constraint multiplier, α0, is set as the mean absolute AEP gradient divided by the length scale, L, so that the separation constraint has a similar scale to AEP gradients, 14α0=mean|∇AEPs0|L, where mean[|∇AEP(s0)] is the mean of the absolute AEP gradient of the initial guess with respect to each component of the gradient. During each iteration, the constraint multiplier, αi, is scheduled to increase based on the inverse of the learning rate, 15αi=α0η0ηi.

The wind rose samples, (u∞(i),θ(i))∼π(u∞,θ), are randomly selected based on the direction frequency and direction-specific Weibull shape and scale parameters. Note that the tilde (∼) denotes a shared probability distribution. After a direction is sampled, the wind speed is sampled as a continuous Weibull-distributed random variable, 16u∞(θ)∼Wu∞,A(θ),k(θ), where the probability density of the Weibull distribution, W, is given by 17Wu∞,A,k=kAu∞Ak-1exp⁡-u∞Ak.

The SLSQP algorithm is selected to be the deterministic optimization algorithm to act as a benchmark to the SGD approach. SLSQP is a conventional deterministic optimization approach. It is employed in many open-source engineering design codes and has been used in previous comparisons of optimization algorithms .

The spacing and boundary constraints are passed to the optimizer as individual inequality constraints. The spacing constraints are defined as 18Cij=xi-xj2+yi-yj2-NDD2∀i,j>i, where C is an upper triangular matrix of nonlinear inequality constraints.

Square wind farm boundaries are represented using four inequality constraints per turbine, 19Di1=xi-xuDi2=xl-xiDi3=yi-yuDi4=yl-yi, and the circular wind farm boundaries are represented with one inequality constraint per turbine, 20Di=xi2+yi2-R, where D is a matrix of boundary constraints that must be less than or equal to 0.

The performance of SGD is compared to the deterministic counterpart, considering wind farms with 100, 144, 225, and 324 turbines, with square boundary constraints, using 20 different initial starting conditions to obtain statistically significant results. The AEP, constraint violation (γ), and time elapsed associated with each optimization solution are plotted in Fig. . The SGD approach consistently yields higher AEPs than the SLSQP approach when the number of scheduled SGD iterations, T, is 2000. There is a large range of computational times associated with the SLSQP approach, though the computational expense of SLSQP generally grows much larger than SGD as the number of turbines is increased. This is largely due to the nature of the turbine spacing constraint, the size of which grows as the number of turbines squared. SLSQP takes about as much computational time as the SGD approach with 500 scheduled iterations when there are 100 turbines in the square farm. As the number of turbines grows, the average time required by SLSQP becomes more costly than SGD with 2000 scheduled iterations. The computational cost of SLSQP is a strong function of the initial layout, and the variance of the SLSQP optimization time also increases with the wind farm size. This is due to the complex interaction between the linear boundary constraints and nonlinear spacing constraints. As the proposed SGD formulation does not offer an automatic way to set the number of SGD iterations, T, results are shown for different values of T. When T is increased, the optimizer finds solutions with larger AEPs, with a computational cost that is approximately proportional to T. The SGD solution consistently improves as more optimization iterations are scheduled (larger values of T). Results associated with 1000 SGD iterations tend to yield similar AEPs to the SLSQP approach, and results with 2000 SGD iterations tend to yield higher AEPs than the SLSQP designs.

The final layouts associated with one of the random initial conditions used in the 324-turbine analysis, when T=2000 iterations, are shown in Fig. . The SGD approach generally identifies solutions with the majority of turbines packed into the side boundaries. The deterministic algorithm also packed turbines into the edges of the farm, although not as many turbines were packed into the east and west boundaries as in the SGD results. The layouts found using the SGD approach tend to have interior turbines that generally appear to be more aligned in the north–south direction than in the deterministic solutions.

The results of the 100-, 144-, 225-, and 324-turbine wind farm optimization cases are summarized in Table . The mean time, mean constraint violation, and mean and standard deviation of the AEP are reported with respect to the 20 random initial starting conditions. Constraint violation is reported as γ/Nt to quantify the mean length of the constraint violations of each turbine. The final constraint violations can be reduced by lowering the ηT parameter. In all of these cases, the SLSQP optimization resulted in solutions with zero constraint violations. This is likely because of the linear formulation of the boundary constraints – when a solution satisfies the spacing constraint, any solutions that satisfy the boundary constraints can quickly be found. SGD with 2000 iterations generally yields solutions with AEP that are 0.3 %–0.5 % higher than the solutions found using SLSQP. This is likely because the SGD algorithm is able to better explore the design space by initially relaxing the constraints, allowing for some initial constraint violations.

To ensure that the previously presented results are not specific to square wind farms, we performed a similar set of analyses examining circular wind farms. The results yield similar trends to the analysis using square wind farms – SGD becomes significantly less time-consuming than SLSQP as the number of turbines increases and generally yields solutions with slightly larger AEPs.

Circular winds farms were optimized using 20 random initial layouts, examining different farm sizes, using the SGD and SLSQP optimization algorithms. The results are summarized in Fig. . The circular wind farm optimization generally took longer than the square wind farm when using the SLSQP optimizer. This is likely due to the more complicated nature of the circular boundary when using Cartesian coordinates. These results are similar to the results in Sect.  – as the number of wind turbines and scheduled SGD iterations increases, SGD tends to find solutions with larger AEPs in less computational time.

The results of the circular wind farm optimization are compared between the SGD and SLSQP optimizers in Table , where SGD is scheduled to run for 2000 optimization iterations. SGD generally results in about 0.5 % more AEPs in significantly less time than SLSQP as the number of turbines is increased. The result area is also compared in Fig. . The SGD optimizer generally results in more turbines on the boundary edge than the SLSQP optimizer.

There are several parameters in the SGD algorithm that were tuned to perform reasonably well. In this section, we investigate the sensitivity of the SGD optimization results with respect to the early stopping option in Algorithm 1, the number of Monte Carlo samples per optimization iteration, the learning rate schedule, and the initial and final learning rates.

As the optimization progresses, the constraint multiplier, αi, becomes large (approaching 10 as the learning rate approaches 0.1), and the gradients of the AEP are overwhelmed by the gradients of the penalty, which take very little time to compute. This situation can be addressed by using the early stopping option in Algorithm 1. The solution tends to terminate quickly when the optimizer only follows the deterministic gradient (the optimization engine terminates when the constraint gradients are 0). Figure  shows the AEP, constraint violation, and computational time associated with five random initial layouts, using threshold parameters of 0.01, 0.05, and 0.1, as well as the SGD algorithm as applied in the previous sections, without the early stopping option activated. The use of each early stopping option results in layouts without constraint violations. As the threshold parameter is increased, the AEP is slightly reduced, and the total computational time decreases. A threshold parameter of 0.1 results in approximately 0.3 % reduction in AEP and 44 % reduction in computation time.

The optimization results presented in this study used 50 power samples per iteration (K=50). We found this to produce high-quality results without incurring unacceptable computational expense. Figure  shows the behavior of the SGD approach associated with different values of K, considering 100 turbines with 2000 scheduled optimization iterations. As K increases, the optimization finds solutions with larger AEPs. There is a small increase in time elapsed and a large increase in the final AEP between the K=5 and K=50 cases, while there is a large increase in time elapsed and a small increase in the final AEP between K=50 and K=200. As K increases, we expect the maximum AEP to reach a plateau and the time and memory required to increase indefinitely. In future work, we plan to explore scheduling K to change as the optimization progresses.

This study used an exotic learning rate scheduler. We tried several schedulers and observed this one to be the best at finding sufficiently large AEP solutions that reasonably satisfied the imposed constraints. Figure  shows the behavior of the SGD algorithm associated with the presented learning rate scheduler, referred to here as the product scheduler, as well as an exponential and a linear decay scheduler. The exponential scheduler quickly diminishes the learning rate, causing the SGD algorithm to become stuck in local minima. The linear transition from large to lower learning rates prevents the SGD algorithm from having sufficient time to follow enlarged constraint gradients. It is possible that the algorithm could be improved by using separate schedulers for the learning rate and constraint multiplier. For instance, it might be more effective to use a linear scheduler to decrease the learning rate and an exponential scheduler to increase the constraint multiplier. We leave this question for future work.

The initial and final learning rates, η0 and ηT, have units of distance and correspond to the initial and final step size of the optimization algorithm. The final learning rate can be interpreted as the degree to which the constraints are to be satisfied, since this will be the step size the optimization algorithm uses when αi is large and the constraint gradients overwhelm the AEP gradients. This is illustrated in the left panel of Fig. , which shows the results of several SGD optimizations using different initial and final learning rates. On average, there is a linear relationship between the constraint violation of the solution and ηT, where the average final constraint violation is approximately 2 times ηT. The maximum observed constraint violation is approximately 4 times ηT. In addition, there is a trade-off between the AEP and constraint violation of the final solution. This trade-off is influenced by the initial and final learning rates. It is important to tune the initial learning rate. An initial learning rate that is too low will result in very little exploration. Initial learning rates that are too high will result in a rapid influx of penalty violations that overwhelm AEP gradients throughout the optimization. From our experiments, we found a step size of one-fifth of the rotor diameter to produce satisfactory results, as shown in the right panel of Fig. .