I havent denied my childrens father any access but i dont want them to stay at his house would i have a say in?

i have all-ways let my children's father see them every 2weeks and have them for the weekend, things started going wrong for him so he started drugs and drink, i really feel strongly now i don't want my children to stay there, would the courts give him that access even tho im still agreeing for him to visit them at home.