Awesome Post! Thank you very much, it's very helpful!

Hi, I'm a doctoral student in cognitive psychology and I find this simulation very interesting, but your inputs are quite confusing to me in respect with what I usually have from the lm model, that is betas and std errors.
I'm looking to simulate a linear model given the coefficients of a previously fitted model (some kind of reverse engineering :). How can I generate datas that fit perfectly (betas and std errors) the model ?

You'll need your estimated betas as your "true" betas, your X's (possibly as a model.matrix if you have factors), and the estimated residual error. The latter is called "sigma" in the lm model object and it is that term that you need as an estimated SD of the normal distribution of the errors. The SE of the betas are ultimately calculated from the residual variance (I think most textbooks on regression will show the math).

I hope that gives you a place to start! If you get stuck after gathering up these pieces for the simulation you might try Stack Overflow or Rstudio Community for more help. :)

Hi, Nice article! I'm slightly confused by what interpretation to draw from one of your final statements;

"Here the proportion of models that correctly rejected the null hypothesis, given that we know it’s not true, is just over 56%. That’s an estimate of statistical power."

Isn't it sort of bad that if we were to just guess from "There is a difference (reject null)" and "There isn't a difference (accept null)" 1000 times, we would expect to conclude the right answer about the same number of times as the model (~50%)?

Your interpretation looks spot on to me. Getting the "right" answer only 56% of the time isn't very good.

Such a result shows that the variability in the data is large compared to the size of the effect I am estimating, so the statistical power is relatively limited. I'd need a bigger sample size if we want to detect the effect with a statistical hypothesis test; figuring out the power and sample size is one reason to do a simulation.