Some have described a “toxic smell” coming from the area and concerns have been raised by councillors about pollution at the quarry.