Don't feel bad, and no there's no need to justify. The rewarding part is nice, definitely motivational to make a change in life. However don't think of exercise to lose weight and fit back into your clothes, then you will feel like chasing a rabbit hole. I was in the same boat last year, I picked up exercising again at first trying to lose weight, but I was very disappointed as I didn't lose any. I actually gained a few pounds after religiously exercising almost everyday.
Everytime I weighted myself, I felt hopeless about exercising and dieting. Fortunately I didn't just stop. I actually felt healthier and happier and have gotten better sleep. So while it's great to have the vanity benefits, your mindset should be focus on making a positive change to yourself and your lifestyle. The weight loss is only secondary. With healthy diet and exercising, your body will have the positive energy and you'll feel satisfied.