I don't bother with the plastic or rawhide mallet, but I use either a brass or a nylon punch between a small steel hammer and the sights.
The real trick to simplifying the job is supporting the barrel solidly. For that, you might consider drilling a hole that's roughly the barrel diameter...