We recently had a hardwood floor installed by a relative and now we are filling some of the missed spots with wood filler. We used mineral spirits to remove the residue from the wood planks to no eval.

What can we safely use?

The solvent or cleaner one uses on a floor depends on the floor finish and what they are trying to clean off. For an oil-based color putty, use mineral spirits, for a latex, use water and keep at it, it will take time. For an acetone/laquer based wood patch, laquer or acetone may melt the floor finish. Use extreme caution. Remove cleaners residue with floor cleaner by Bona.

Have had pretty good luck with the mr. clean magic eraser.