what are you actually trying to do? work out how much water there is above the player?
that's not going to work with traceline. traceline can only detect solids, it can't find water.
trace_inwater / trace_inopen merely state that *part* of the trace was inside water/empty. if they're both set then you know that it crosses a water surface, but you don't know where. probably you'll have to bisect it until you find where the surface is.
basically trace up from 0 to 1000 units. if it crosses into air, try 0 to 500, if it doesn't, try 500 to 750, etc, until the range is small enough that you don't care too much.
or you can spam some pointcontents calls.