Preheat the oven to 375ºF. Line an 8- by 4-inch loaf pan with parchment paper.
In a large bowl, use your fingers to rub the sugar and lemon zest together so that the oils in the citrus are released (this is optional but it really helps the lemon flavor pop!).
Whisk in the eggs, olive oil, and yogurt until smooth, then the lemon juice and almond extract. Whisk in the almond flour, oat flour, arrowroot, baking powder, salt, and poppyseeds.
Scrape the batter into the prepared pan.
Bake for 40 to 45 minutes, until a toothpick comes out clean. Cool in the pan for 10 minutes, then unmold the loaf and cool it completely on a rack.
Lemon Glaze:
In a small bowl whisk together the powdered sugar and lemon juice until smooth. Add the milk or cream 1 teaspoon at a time, until you have an easy-to-drizzle consistency.
Spoon the glaze over the cooled loaf. Allow at least 30 minutes for the glaze to harden, then garnish with lemon slices or zest, if desired, and slice and serve.
Stored in an airtight container or bag, the loaf will keep for 3 days at room temperature. The flavor is even better the next day!