More than 50 bags of asbestos were found dumped on Gelligaer Common, near Fochriw.
Environment Agency Wales officers have launched an investigation to trace those responsible.
Some asbestos sheets appeared to have been correctly bagged for disposal before being dumped on the site, which is close to a local fishpond, according to the agency.
It has confirmed the material is contained and there is no danger to human health, local wildlife or pollution to the pond.
The asbestos was reported to Caerphilly County Borough Council by a member of the public on Saturday.
A spokesperson for Environment Agency Wales said: “It seems as if the people who did this knew what they were doing in terms of bagging the material. To then simply dump it like this is completely irresponsible.
“Material like this must be disposed of properly because it has the potential to harm the environment and human health. However, it does cost money to dispose of properly, which is probably the reason behind this crime.
“The cost of clean up is likely to run into thousands of pounds and will ultimately be picked up by the taxpayer.
“We are investigating to find those responsible but are appealing if anyone has any information, or has seen something suspicious, to report it to our hotline on 0800 80 70 60. Calls can be made anonymously and all information will be used as part of our investigation.”