Whisk the eggs in a bowl for about 1 minute until well combined.
Stir the crème fraîche and milk into the whisked eggs.
Fold the mashed goat cheese into the egg mixture. Season with pepper, fleur de sel, and nutmeg to taste—be careful not to over-salt, as the cheese is already salty.
Grease a flat gratin dish or 4 individual molds with fat.
Pour the egg mixture into the prepared dish or molds.
Top with the olives and sprinkle 1 tablespoon fresh thyme over them.
Bake for 30 minutes until the clafoutis wobbles slightly when gently pressed and has the consistency of an omelette—do not bake until firm as it will dry out.