hello:

I am trying to install hardwood 8' x 10' surrounded by tile. The tile is level, the floor is not. So I used the latax modified floor leveling compound. Its a little to high in some areas and not high enough in others.

I experimented with nailing a strip of hardwood down. The nails bend and curl under the hardwood and never penetrate into the plywood subfloor. I called the 1-800 number and they assured me that this is how its done and they have never heard of this being a problem. It takes very little effort to lift the boards up. Also the concrete leveling is breaking up where the nails attempt to penetrate.

1) is there a special nail or what do you recommend?
2) how much of a gap should I leave for expansion between the wood and tile?
3) is there a better leveling compound that I should use?
4) should we just install carpet like I wanted to in the first place?

TIA for any advice.

These cementitious leveling compounds are NOT for use under nail down floors, for exactly the reasons you are having problems. Either the nails bend, and/or the compound breaks up. I suggest you switch to an engineeered glue down floor.
In hindsight, the hardwood should have gone down first, followed by the tiles. This would allow you to leave a small expansion space for the hardwood, as well as making the levelling process an easier one.
Sorry that I dont have a "quick fix" to the sittuation you find yourself in now. Perhaps one of my colleagues here has encountered this situation before and solutioned it better than what I suggest.

You could lay a straight edge across the tiles over the subfloor where the wood goes. Then map out the highs and lows. Grind down (sand) the high spots and fill the low places with asphalt roofing shingles or thin plywood. This is the same procedure for flattening a wood subfloor prior to installing a nailed down floor.

I just thought of this today.

What do you think about cutting out the old floor.
puting shims on the joists
laying a new high grade sub floor and nailing the hardwood to that?

Too risky imo unless you can stay at least a foot or more off the tiled area when cutting out. Both the new and existing subfloor have to be securely supported and fastened around the perimeter. If your first patch was adequately done you could glue the wood down in this particular situation.