It probably depends how much of the total ride you end up above Z2.
I referred to a recently published paper in another thread
“A 3 minute all out test performed during an endurance-based training session did not alter metabolism or perceived exertion, nor did it affect delayed exercise capacity beyond the effect of the 3-h prolonged exercise itself. There appears to be no physiological basis for the strict exclusion of short, high-intensity bursts during endurance training.”
Zone 2 with some intense intervals. Harmful to Z2 effect? - #35 by ivegotabike
I would also say that keeping the pedalling time as Zone 2 work is the primary objective. But, if the work by other parts of the body needed to stay on the bike on the gnarly bits is significant, it will all add up as you describe.
I am not someone who would say “just go slower on the downhills on endurance days” but I expect there are people out there who might.