Tracy, you put the money down for him to hold while you checked out the diamond. You did not promise to buy or you would have purchased it right away. If he screams that it cost him money to get the diamond in just for him. Tell him really, I thought you were a diamond wholesaler and it was your diamond.
The $500 is yours. You can stop payment on a check. You can call the credit card company and tell them. Don't let them run over you.
Good Luck!