Why do I get double/crossed vision when my eyes are relaxed?

I have astigmatism and I hardly wear my glasses because my prescription is very mild but whenever I relax my eyes I get double vision as if I were physically crossing my eyes, but I am not. Also when I am having a converstion with someone and I am looking at them I feel like my eyes cross and then I have to look away and then come back to the person to regain my focus but then seconds later it happens again. Has anyone had this problem? Could not wearing my glasses eventhough the prescriptionis low be the cause of it?

This is unlikely to be anything to do with your current spectacles, worn or not worn, nor any small degree of astigmatism.

Only a proper examination could pick up all the possibilities, including the rare ones, but the most likely is that you have a significant amount of esophoria or exophoria.

That is, left to their own devices ,your eyes would be more comfortable slightly turned in (eso) or out (exo) but under most circumstances this does not happen, though maintaining alignment requires effort.
Almost everyone has this to some degree, if measured accurately enough, and if "fully compensated" such that it causes no fatigue, eyestrain or blurred or double vision it is of little or no significance, even if the swing on test is quite spectacular (it can be).
But if "uncompensated" keeping the eyes straight takes more effort than can comfortably be sustained, especally when tired, resulting in eyestrain or actual double vision.

The best treatment can be prisms incorporated in the spectacles, eye exercises, or surgery to re-align the eyes, depending on individual factors.

Optometrist, retired.