Polarization-dependent center-of-mass motion of an optically levitated nanosphere