I find I just can't ride with a back rest, I too get pains with or without the back rest so I choose not to use it.
My lower back is fused at L5-S1 and when using the back rest I tend to get sciatic pain under my but and down the back of my legs, without the back rest I tend to get pain between my shoulders and neck.
So I pick the lessor of the two and which one I can heal up and recoup from the fastest which is Not to use the backrest.
I also found over the years that my seating position and fatigue plays a big part in the pain I feel while riding. If I sit up straight and my feet straight down and not stretched out I feel the best , also keeping my head straight and not tilted down. As the day goes on with longer rides I tent to slouch and that puts my head in a different position which then starts the pain.
Once I feel it coming on it reminds me to sit up and then I roll my shoulders forwards and backwards and the same with my head and that helps. If all else fails I stop and go to the trunk and take some Advil or Aleeve.
We do 2-300 mile lunch rides every weekend so this is a ongoing thing for me and I come to realize it is me and not the bike. Years of back and neck issues along with a bad shoulder I think for me its just the cost of doing something you like. Somedays I dont hurt at all and others are just a real drag, alot of it depends on my work and what I did prior to rideing to agravate my current situation.
I will say that on a 10 day road trip I hurt for the first day and after that I feel pretty good, so like my doctors had told me once I stop doing dent repair ( I do Paintless dent repair ) I will fell better and heal up. Cant retire yet and I like what I do , so for now it is what it is.