What role do the proteins in a virus's outer coat play in the invasion of a host?

At the most, docking. Though docking is usually accomplished by glycoproteins studded on the capsid. Still, T even phages have landing extensions made of proteins that they use to land on on fix themselves to bacteria.

The actual in invasion is genetic material as the capsid is left outside, or molds with the cell membrane.