Beat egg yolks and sugar together in a bowl with an electric mixer until fluffy and pale yellow, 1 to 3 minutes. Heat milk in a large saucepan over medium heat, and whisk in the egg yolk mixture until smooth; boil for 1 minute. Mix in the coconut and cracker crumbs; bring to boil, reduce heat to low, and simmer the mixture until thick, about 5 minutes.
Beat the egg whites in a glass or metal mixing bowl with an electric mixer until they hold stiff peaks; gently fold into the coconut mixture, and mix in the vanilla extract. Serve warm or cold. Refrigerate leftovers.