Campaigners have warned plans to redevelop the stables near the castle and move a bat roost would have an “unacceptable impact” on the Grade II* listed site.