You're original reason when it was proposed was that you would never fight in a gym. Driving, driving distance or other "inconveniences" were never an issue. You stated quite vociferously that you would only fight in a street fight. But you never ever stated driving or traveling as a reason not to scrap.
Now you did not mention the issue of fighting in a gym, now the issue is traveling outside of your house.
Sounds to me like "excuse du jour", and again, I don't blame you to avoid a scrap with the guy.
Don't quote 6'7 280 big tough burly shit when you can't even keep your excuses straight.
6'7 280 could be this guy......

size != skill