The radius in the groove is sized for the wire. There's no room for a rope.
There are dual use sheaves which have a groove sized for the rope, with a smaller groove in the bottom sized for wire. Of course, there are rope-only sheaves that don't have the smaller groove.
Whether wire or fiber, the rope wants to be supported for about 120 degrees as it lays in the groove. The dual use sheaves are a compromise, but the groove for the wire doesn't detract that much from the support area for the larger fiber rope.
A new sheave is the answer, assuming there's enough room in the sheave box for the size fiber rope you want to run. Many of the older boats with the all-wire halyard reels didn't need or provide room in the sheave box for rope.
You may have to size the fiber rope to the space available and then pick the rope technology to get the strength you need. If you still don't get there, perhaps a 2:1 halyard would work - halyard fixed at top of mast, sheave at the head of the sail, back to the mast head, then down and back to the winch. The only down side to this is that when you raise the sail, you end up with a LOT of line in the cockpit (and the high tech stuff doesn't coil that easily).