Social Security office. I went to the office in Melville and did it all in a matter of 15 minutes. Just bring your drivers license and marriage certificate.
Then do DMV, call your credit card companies and don't forget to tell human resources at work so they can change your name there as well.