In Aptheker v. Secretary of State, 378 U.S. 500, 84 S.Ct. 1659, 12 L.Ed.2d 992 (1964), a case decided on constitutional grounds, the First Amendment seemed to play a more central role in the Court's analysis. "Since freedom of association is itself guaranteed in the First Amendment, restrictions imposed upon the right to travel cannot be dismissed by asserting that the right to travel could be fully exercised if the individual would first yield up his membership in a given association." Id. at 507, 84 S.Ct. at 1664.