Heat oven to 400 F. In a saucepan, gently heat mashed potatoes and milk, stirring occasionally, until well combined. Remove from heat and stir in butter. Set aside. Heat olive oil in a large skillet over medium-high heat. Add meat-loaf mixture and cook, breaking up with a spoon, until browned, 5 to 7 minutes. Add carrot and celery, and cook 5 minutes. Sprinkle in flour, stir, then add chicken broth. Simmer 3 minutes and remove from heat. Add peas and whole potatoes. Spoon into an ovenproof casserole and top with the mashed potatoes. Bake until the top is golden, about 40 minutes.