In a large heavy saucepan, combine the first 5 ingredients. Stir in milk until smooth. Cook and stir over medium-high heat until thickened and bubbly. Reduce heat to low; cook and stir 2 minutes longer. Remove from the heat.
Stir a small amount of hot mixture into the egg yolks; return all to the pan. Bring to a gentle boil, stirring constantly; cook 2 minutes or until mixture is thickened and coats the back of a spoon. Remove from the heat.
Stir in butter and vanilla. Cool for 15 minutes, stirring occasionally. Transfer to 6 dessert dishes. Cover and refrigerate until chilled. Garnish with whipped cream if desired.