Combine the first 11 ingredients in a large saucepan, and bring to a boil. Add swordfish; reduce heat, and simmer for 5 minutes. Spoon mixture into a shallow dish, and cool. Stir in olive oil; cover and chill at least 24 hours. Remove swordfish from marinade with a slotted spoon, discarding marinade.
Arrange orange and lemon slices on a lettuce-lined platter. Top with swordfish, and garnish with fennel fronds, if desired.