Adam isn't family. Like Bobby and Dean have both said it, Family doesn't end in blood, but it doesn't have to start there either. And Adam has made it clear he does not want to be a part of the Winchester's family. Also, Adam's a whiny bitch. Why would anyone want him back?