If this is just the yellow mud's and compacted dirt, contrary to much of the public opinion this is not uranium or high levels of sulfuric acid. For the most part it is iron sulfides, The mine water mixes with the pyrites (iron sulfide) and as the mine’s air comes into contact this produce hydrogen ions and sulfates.
In the worst case and over an extended period of time and only in the case of stagnant water may cause sulfuric acid to be created which in turn leaches more of the metals out of the host rock.
Mostly, this is just yellow dirt caused by sulfides and metals pigment leaching.
Unless you have a method of testing the dirt or other chemical and metal levels.
The very best method for handling these material is the rule of thumb, "Stay out Stay Alive".