David first assembled his followers at the Cave of Adullam. They were his family and also the outcasts of society.

Source:

1^st Samuel 22