Preheat oven to 400°F. Combine nutmeg, allspice, and 1 teaspoon each of the salt and pepper in a bowl; set aside.
Heat 1 tablespoon of the oil in a large skillet over medium-high. Add collard greens, and cook, stirring often, until bright green, about 2 minutes. Add garlic, and cook, stirring constantly, until fragrant, about 1 minute. Season with 1/4 teaspoon each of the salt and pepper.
Melt 2 tablespoons butter in a small, heavy-bottomed saucepan over medium. Whisk in flour; cook, whisking constantly, until mixture bubbles and flour is cooked but not browned, about 2 minutes. Whisk in warm milk; bring to a boil, whisking occasionally. Reduce heat to low, and cook, whisking occasionally, until sauce thickens, about 2 minutes. Add 1/4 cup of the cheese, stirring until melted. Season with 1/2 teaspoon each of the salt and pepper. Remove white sauce from heat.
Sprinkle 1 tablespoon of the panko on bottom of a 2-quart baking dish lightly greased with olive oil. Layer half of the sweet potatoes evenly over breadcrumbs; season with remaining 1/4 teaspoon each salt and pepper. Spoon collard green mixture over sweet potatoes; sprinkle with half of nutmeg mixture. Top with remaining sweet potatoes; sprinkle with remaining nutmeg mixture.
Pour white sauce over sweet potatoes, and cover dish with a piece of parchment paper. Bake in preheated oven until sweet potatoes are soft when pierced with the tip of a knife, about 40 minutes. Meanwhile, combine 2 tablespoons melted butter and remaining 2 tablespoons panko and 1/4 cup cheese. Reduce oven temperature to 375°F. Remove dish from oven, discard parchment, and sprinkle panko mixture evenly over potatoes. Bake, uncovered, until golden brown, about 25 more minutes. Transfer to a wire rack; let stand 10 minutes before serving.