Smaller is fine -- I'd guess you're starting (with the conversion to diesel) the equivalent of a 6 cylinder gasoline car engine.
We use an AGM so that the charging profile is the same. We have AGM's for our house bank. You should do the same. Ours house batters are big Lifelines. We use a smaller spiral Optima (http://www.optimabatteries.com/en-us/technology/spiralcell-technology) battery. It's for automobile, aviation, and marine use. They're small, can be mounted upside down (if you wanted) and seem pretty easy.
The question is usually the alternator and regulator connection. It would be best if they were on their own alternator or off an alternator controller with their own dedicated connection. But having them on a combiner will work. If you have shore power, similarly you should think that through.
You should have a battery switch. IF, God forbid, you have a short, you want to disconnect the battery vs. burn down the boat. You also might want to do some work in the engine room and want the circuit clean.
It's good to have a little cross connect switch, in case you need to start the engine or whatever.
The question is usually the alternator and regulator connection. It would be best if they were on their own alternator or off an alternator controller with their own dedicated connection. But having them on a combiner will work. If you have shore power, similarly you should think that through.