The thing is, would you call someone who had a stroke "brain-damaged"? Would you say that to Kirk Douglas or Garrison Keillor or Ray Bradbury -- all of whom had strokes and then kept on doing public appearances? I mean, that is the definition of what a stroke does -- it damages your brain!
But no, people understand that a stroke might affect someone's speech, and might slow them down in certain ways, but if they're lucky enough and have quality care, they can resume most of their lives.
No one would call them brain-damaged, even though it's literally true, because the term is pejorative. Pure and simple.
As someone who's HAD a stroke (4 years on, now), I can say I agree with you 100 %