I'm reading a large collection of text files into a DataFrame (initially it will just have one column, value). The text files use HTML encoding (i.e., they have < instead of <, etc.). I want to decode all of them back to normal characters.
Obviously, I could do it with a UDF, but it would be super slow.
What is a good approach?