Heat oven to 375° F. Place the halibut fillets in a 9-inch square baking dish. Pour the wine over the fish. Drizzle with 1 tablespoon of the oil and season with 1/2 teaspoon of the salt and 1/4 teaspoon of the pepper. Scatter the olives, rosemary, and garlic over the top. Bake until the fish is the same color throughout and flakes easily, about 15 minutes. Meanwhile, heat the remaining oil in a large skillet over medium heat. Add the potatoes and cook, stirring occasionally, until golden and tender, 15 to 17 minutes. Transfer to a paper towel-lined plate. Season the potatoes with the remaining salt and pepper and serve alongside the halibut and olives.