It seems like physical level will only effect your own experience points (not skill points). So your physical level will grow more slowly than someone rank 1 and level 1, but that makes sense. Your skill points should grow at the same rate (or as "same" as you can have when it's a "random" award per hit).
FFXI: Triane [Caitsith], Retired DRK75, THF75, RDM71
FFXIV: Triane Rhiga
[Besaid] Pugilist, Marauder, Fisher