First, there needs to be some angle between the swivel to halyard block so that the swivel halyard won't bind

Secondly, your hoist position may have the head bound by the halyard exit.

Thirdly, do you have a saggy forestay? When going to weather, does it noticeably sag? Can you see the middle telltale fron the stern quarter while beating? Do you need to add backstay pressure in light air to take-out the sag? There is some sag, but it's common to have an overly loose forestay. It SHOULD NOT be the halyard that controls that on a boat with a forestay!

Finally, sometimes the swivel bearings get flattened.

I would go up the rig with the jib pulled out and see what's happening. This is best done at dock in light air unless you are a foredeck man ;^))). You should have someone tension the rib and work the fuller.

On our working jib, you really don't need a lot of tension in light to moderate air -- just enough to tack the wrinkles out of the luff. Even though we have Kevlar (equivalent) sails, I release the tension after furling when we put the boat away for the day. This to keep non elastic stretch down and reduce the strain on the furling bearings.

When we are going to weather in stronger winds, we flatten our sail by both halyard and backstay tension.