I would say it won't make a big difference, but you can't choose how to distribute HPs and MPs. So that might make an actual difference. Also while the stat differences don't appear to have a lot of difference, what would concern me is the diminishing returns aspect. As you level it will take more points to increase a single stat by 1 point. So at some point because of diminishing returns, having that little 4-6 point difference might come in handy. Just depends on how SE has this set up though.
I also want double eyepatches, to increase my pirate bravado.