If you use RCBS type dies the brass is worked more than if you use Lee collet dies for neck sizing. If you full length, then yeah annealing will prolong fatigue. I bought brass already annealed, and used brass not annealed with no problems. 10 plus loadings and still good.
When I was making 35 Whelen and 338/06 cases out of 06 cases I annealed the cases because of the expansion and sizing etc.. Maybe I did some 300 HH before I blew them out to 300 Bee but that was a lot of rounds ago. Those are the only cases I have ever anealed in 50 years of reloading
I just recently annealed a batch of Norma .300WSM cases due to work hardening. The kit I used was from Hornady and included the special heat sensitive paint. The annealing process went very well and I noticed an immediate difference in brass flexibility and bullet seating.