Prime Minister Kevin Rudd has rejected the Exclusive Brethren's call for an inquiry into their treatment.
More than 30 former members of the cult have sent a letter to Mr Rudd outlining their treatment and calling for an investigation.
A spokesman for Mr Rudd says his office has replied to the letter, saying that at this stage there will be no inquiry.
He says the most appropriate action is for people to take their concerns to the police, and a government inquiry would encroach on religious freedom.
Australian Greens spokeswoman Christine Milne says the welfare of children and families should come first.
"The issue here is not about religious freedom, it is about what this cult is doing to destroy families and effectively to undermine the law," she said.
The Exclusive Brethren were in the news in the run-up to last year's election over their high-level meetings with then-prime minister John Howard.
