My phone actually feels hot

My phone actually feels "hot" to the touch at times. Usually if I have been using it awhile, but still the idea of it getting hot is really odd, and it is to the point it can be uncomfortable to hold. I'm concerned I am causing damage but at the same time I need to use my phone. Any idea what this could be or what I could do to remedy this? Thanks.
