Horowitz writes that ayin "is a twin letter":
Ancient Hebrew had two different ayin sounds. These sounds were represented in our alphabet by the letter ayin. One was a harsh, heavy ayin. This is now lost, and no longer used in Hebrew. The other was a soft, mild ayin. When the Greek Jews translated the Bible into Greek, they had to transliterate Hebrew names having the harsh ayin in it. They used the Greek letter gamma for it - so you can imagine how hard a sound it must have been.
This "ayin gayin" has even come all the way down to English. The Hebrew place names עמורה (amora) and עזה (aza) both of which have this strong ayin were transliterated into Greek as Gommora and Gaza. Didn't the odd forms of these place names in English ever puzzle you?...
Incidentally, Arabic, a close sister language of Hebrew, still pronounces these two ayins differently, and what's more writes them differently.
He then goes on to list a number of pairs of words that would seem to be related based on the letters in their roots, but each originally was based on a different ayin:
עצב sad vs. עצב fashion, shape
עופר deer vs. עפר dust
עור be naked vs. עור be awake
Steinberg writes that there are different alternating letters for the soft and hard ayins. The softer ayin would switch with the other guttural letters: alef (תעב תאב), heh (no example provided) and chet (ענק חנק). The harder ayin switches with gimel (רעש רגש), kaf (עטר כתר) and kuf (ערא קרה). Kuf (as we will see later) can sometimes switch with tzade, so we sometimes have the jump from ayin to tzade as well: רבע רבק רבץ , עוק צוק, ארעא ארקא ארץ.
Klein adds that in some cases, the ayin will fall out of a word, as in רות which may derive from רעות.
From the word ayin (the English word eye has an unrelated etymology) we get ein עין and maayan מעין - meaning "spring, fountain", and according to Klein and others this is a shorter version of עין המים - "eye of the water".
Klein says that the word maon מעון - residence is related to ein, but I don't exactly understand how.
From ayin we get the verbs עין - "to look carefully at", and also "to look askance at" - from which derives the adjective oyen עוין - "hostile".