Normally, my rest days involve not doing much. It's an almost constant factor that threads through them all. What I wouldn't do on a rest day is ride up something like the Col de Peyresourde, 9.8km long, 669m of ascent, an average of 6.8%. I certainly wouldn't start the ride by throwing in an extra 300m of ascent over Col de Shortcut before getting to the foot of the Peyresourde.
I'd enjoy a Coke and an ice cream, maybe, and even a 15km descent into Bagnères de Luchon followed by a pleasant coffee and a wander round a bike shop. Following that with a ride up the Col du Portillon (7.9km, 8.4% average, sections to 14%) into Spain would be sheer madness.
Oh, no. I wouldn't do any of that on a "rest" day.
What I definitely would do is spend a couple of hours drinking beer and eating tapas, before stuffing myself with a selection of excellent Spanish food, wine, port and beer.
I must have imagined that whole middle section of the day. And faked those Col signs too.