My wife and I had a similar discussion years ago- though not about possible infidelity or me no liking her body.
I was a varsity tennis player in high school and an avid cyclist through college. During the summer weekends it was nothing for me to go out for 60-70-100 mile bike rides. I then drifte away from biking for a long time until in 1998 I decided to buy a new bike.
I bought a mountain bike, then started mountain bike racing in 1999. Later picked up a cyclocross bike and then a new road bike over the next 7 years.
In order to be competitive, not just with others, but with myself versus previous events, it does take some dedicated training. My wife said she is a "bike widow" in the summer.
But to the initial sentence, yes it does take some level of selfishness to be able to train and workout. It also takes focus and determination. One cannot always say "don't feel like it today"
I have a bike in the basement permanently on a trainer and also recently bought a used treadmill.
But on the flip side- I do almost all the cooking and am always ready for intimate activity if she is- probably more so LOL.