Residents and local councillors have raised concerns about possible pollution at Ty Llwyd quarry, with some describing a “toxic smell” at the site.