Bring first 5 ingredients to a boil in a saucepan. Reduce heat to low, and simmer, stirring occasionally, 15 minutes. Stir in vanilla. Cool syrup. Discard cloves and cinnamon stick.
Peel and section oranges and grapefruits, and place in a large bowl. Gently stir in syrup. Chill mixture, if desired. Slice bananas, and stir in just before serving. Garnish, if desired.