Combine the sugar, lavender, and lemon zest in a food processor or blender and process until the lavender buds are broken down. Set the lavender sugar aside. 

Place the eggs (while still in the shell) in a bowl of warm water. The goal is to get the eggs slightly warmer than room temperature. This will warm them up and help them whip to a higher volume. Let them sit while you prep the cake pans and the ingredients. 

Preheat the oven to 350°F

Grease and flour two 9-inch cake pans and place some parchment paper in the bottom of the pans. (You can also use 8-inch pans if they are 3-inches tall)

Combine the butter, oil, and milk in the saucepan and warm it over low heat until the butter melts. Keep the pan on the lowest heat setting to stay warm.

Crack the warmed eggs in a large mixing bowl and add the lavender sugar. Beat on high speed for about 10 to 15 minutes. The eggs should triple in volume and become pale yellow in color.

Sift the flour, baking powder, and salt over the egg mixture and fold in with a spatula until it is just barely combined, you will still see some flour streaks.

Add the vanilla to the warm milk mixture and pour the milk into the batter. Stir the mixture by hand or on low speed. Scrape the sides and bottom of the mixing bowl with a spatula to make sure all the flour is incorporated. 

Divide the batter into the cake pans.

Bake for 22 to 28 minutes or until a toothpick comes out clean. (If you use 8-inch cake pans, the batter will be thicker and you will need to increase the baking time by 5 minutes)

When the cakes are finished baking, set the pans on a wire rack on the counter and let them cool in the pans. 

Place the sugar, water, and lavender buds in a small saucepan. Bring to a boil over high heat then reduce the heat to a simmer. Simmer for 3 to 5 minutes then remove the pan from the heat. Place a lid on the pot and let the syrup sit for 10 to 15 minutes to infuse the lavender flavor. 

Pour the syrup through a fine-mesh strainer and toss the lavender buds out. Reserve the lavender syrup for later. 

Set a strainer over a bowl and keep it readily available near the stove.

In a heavy non-reactive saucepan, beat the egg yolks, sugar and butter until well blended. Stir in the lemon juice, zest and salt. Cook over medium-low heat, stirring constantly, for about 10 to 15 minutes until thickened and has a glossy appearance. The slower you cook the curd, the thicker it will be. Try not to let the mixture come to a boil, if it begins to boil remove it from the heat for a bit then return it to a lower heat. You can tell if the lemon curd is thick enough if it coats the back of a spoon and you can run your finger through it and it holds a trail. 

When the curd has thickened, pour it into the strainer and press it with the back of a spoon. Discard the lemon zest and any lumps that are left behind in the strainer.

Place in the refrigerator to chill. The lemon curd will thicken more once it has chilled completely.

When the cake cools down, drizzle 4 to 5 tablespoons of lavender syrup on each cake layer. Place one cake layer on a platter. Spread the top with a thin coat of buttercream (this will prevent the lemon curd from soaking into the cake). Pipe a wall or dam of buttercream around the edge of the cake to prevent the lemon curd from seeping out. Spread half of the lemon curd inside the dam. 

Place the second cake on top of the lemon curd then cover the cake in buttercream. Add the rest of the lemon curd to the top of the cake. Pipe mounds of buttercream around the top edge of the cake.