That will be some of it
You seem to have found part of the problem-if there is one. Of course-take the advice here to make sure you do not have structural problems, then tune the rig centered in the boat and straight side to side. Set prebend to match you mainsail (lots of posts on this here), and as you already know, make sure you set the boat up the same way on each tack (trim, heel, etc.).
If you have a knotmeter, and it is on centerline, it will give more meaningful tuning info than a GPS. If the transducer is off to one side, or not very good, use the GPS-sounds like you understand the variables-but GPS is not the way to tune a boat in general-too slow to respond (many of them anyway), and is not giving you what you want. It gives groundspeed and direction over the ground, not boatspeed or heading..But if that is what you have, so be it.
Again-if the rig is off more than an inch to one side, you will notice the clew of the headsail higher off the deck on one tack than the other-when trimmed in the same amount of course. You can use a mark on each genoa sheet measured at the same distance from the clew, and see where it is reference something fixed-like the cabin top edge, a winch, turning block, etc. If the marks are the same on each sheet, and they line up with whatever marker you use on each tack, you have it trimmed the same amount on each tack.
Then you can see if the clew is higher on one side you can confirm if the rig is off to one side or not (a double check against the rig measurements you are aready doing)...
BTW-It is best (unless you have a hydraulic tranny) to leave the prop in reverse while sailing. Leaving it spinning slows you down, and even worse, the tranny has no lubrication if the engine is not running, so the shaft is spinning away without being lubricated. Lock the prop.
If you have a 2 blade, when you are the dock, dive down and with the engine in neutral, rotate the shaft until the blades are straight up and down-then using a magic marker, nail polish or tape, mark the top of the shaft near the coupling (somewhere where you can see it). When you will be sailing for any period of time, shut the engine down, put it in reverse to stop the spinning, then neutral again so you can reach down and rotate the shaft to the vertical, marked position, then engage reverse again to lock it. This will reduce drag considerably and help sailing speed! We do this all the time with folding props, but the mark indicates the blades are horizontal, not vertical, so they fold back and cannot "drop a blade" if you slow down...
Good luck!
S