Wind farm layout optimization is one of the main phases of offshore wind farm design and involves a wide range of implementation choices, spanning aspects such as flow modeling, data selection, and objective definition. This study quantifies the impact of 18 different implementation choices across 9 categories on the results of wind farm layout optimization. The comparison is performed using both a conventional approach based on performance metrics and the concept of <em>effective layout distance</em>, a novel metric introduced to measure differences between sets of layouts based solely on turbine positions. The results show that the hyperparameter values of the optimization algorithm, the choice of wake model, the use of cost of valued energy as objective, and the adoption of a control co-design approach have a high impact on the optimal layouts and therefore require careful consideration during problem formulation. In contrast, choices such as rotor averaging models and the use of revenues based on varying electricity prices as objective have a negligible effect on the resulting layouts and can therefore be selected based on simplicity and computational efficiency.