YES!!
My mom never cleaned her diamond (and did everything with it on). the dirt and grime building up in the setting was, she was later told, a prime factor in the diamond popping out of the setting suddenly one day and being lost.
She had insurance but no proof (sales receipt, appraisal) so never got it back.
Please do clean your diamond and also have prong and setting regularly checked by a jeweler (who can also do a better cleaning than you can). any jeweller is happy to do this for you (they hope you will see something you lke in the store while they clean).
Nope. The dirt will not eat into it like acid, or anything like that. I suppose if it was incredibly dirty, you'd be less likely to notice damage caused by other things (i.e., you banged it on something and chipped it), but the big problems with never cleaning a ring are, a) it looking cruddy, and, b) that being just plain unhygienic.