Melt 1/4 cup butter, 2 ounces bourbon, and olive oil together in a large skillet over medium-high heat. Cook sea scallops in hot butter-bourbon mixture until browned and just about cooked through, 2 to 3 minutes per side. Transfer scallops to a plate and drain any excess liquid from the skillet.
Whisk remaining butter, remaining bourbon, brown sugar, cayenne pepper, salt, and black pepper together in the same skillet over medium-high heat until sauce reduces and alcohol evaporates, 1 to 3 minutes. Drizzle sauce over scallops.