So likely I would have to remove the counter and trim to get at all the old stuff, to remove it, so I could put in the new stuff... I am not that skilled at finished work... so to me, it is a giant ball of pain.

I do mechanical and electrical quite well... so I am half tempted to just go in with a cutting tool, drop in a brand new finished refrigerator box, and then cover up the ragged edges with some sort of trim. That I can do... but it won't be as nice as the built in box that exists now.