The vibration as you run the RPM is probably caused by one of the prop blades not swinging out. When you shut the engine down, one of the blades can be vertical and gravity feathers that blade and little bits can temporarily lock it in place. Then it is essentially is a one bladed prop. Dropping the rpm back allows the blade to open up. As far as rough idling, does it happen when the engine is cold and goes away as it warms up??? If the former, it could be incompleat combustion in one cylinder until the engine warms up. My Yanmar does that.