I am hot gluing some acrylic gems to a clay pot - would this work?

It will work, but will also be very fragile. Also depends on if the pot is glazed or not. Specialized epoxy would likely work best. Go to your local hobby or hardware store and they can help you pick out the best product. Bring both the pot and a couple gems so they know exactly what you are wanting to fasten.