From your post, I have the impression that you are too heavily focused on the ‘Fitness’ number.
On the other hand, as an amateur (ex-)racer, I would expect that you are familiar with the terms ATL/CTL/TSB (in Intervals Fitness/Fatigue/Form).
So I could be wrong in this, but still think it’s important to make sure that you understand what the Fitness number represents.
If in doubt, first read the first posts in this thread:
FITNESS PAGE - a guide to getting started - #2 by MedTechCD
TL:DR: The Fitness number is not representative of performance level, it simply shows how hard you worked over the last weeks. Someone with a Fitness number double of someone else, isn’t necessarily going to outperform him.