Made a rookie mistake on my first build - powered up my VTX without connecting the antenna. Now whenever I plug in the battery, the video transmitter gets extremely hot to the touch within seconds. The video feed still works, but I'm worried I've damaged something.

Do I need to buy a new VTX, or is this normal operating temperature? How hot is too hot for these things?

VTX will get hot sitting idle even with the antenna connected. That is normal. As long as your video appears OK, then you don't need a new VTX. However, running without antenna can cause damage over time due to reflected power.

Never ever power on your VTX without antenna, all the energy sent to the antenna will bounce right back when the antenna is absent, and instantly cause overheat and might permanently damage your video transmitter. If video still works you might be lucky, but monitor it closely for degraded performance.

I did the exact same thing on my first build! Yes, vTX's get extremely hot sitting on the bench, as Wayne said a fan, or maybe undo the connector on the vTX (if it has one instead of being soldered), and of course try to never power the vTX without an antenna. Mine survived but always use proper cooling when bench testing.