A lot of this is about your tolerance for delays, altering choice of destination, and decisions about acceptable pace.
You wrote the other day that you start your engine if boat speed goes below 8 knots. I would call that your "speed floor". My boat cruises under power at 8 knots, but we sail until we slow to about 4 knots (that varies depending on situation) when we're not in a hurry. Yes we sail pretty well in light air, so we sail more than most others, but the point I'm trying to make is the "speed floor" is really the major deciding factor. Our average speed is surprisingly low compared to our boat's performance potential.
That means to sail at ~8 knots your J40 is still going to need ~15 knots of wind (depending on whether light-air headsails are deployed and depending on sea-state, current and angle of apparent wind). What percentage of the time are winds below that level, or at the wrong angle, and are you willing to delay departure for more favorable sailing conditions?
You'd also need to decide whether the reduction in comfort is worth it. No more full-size fridge, and the space you lose is in the most voluminous part of the boat (they both have bows and sterns, so you'll be giving up space in the middle).