It isn't uncommon for serial killers to taunt police with letters and phone calls like the killer in the show. As stated, this is due to their arrogance and need to dominate.
However none of them want to get caught, they do it to prove they are smarter than the people chasing them. And they're often right.