I suppose on retrospect I cannot confirm 100% that my fall death was without agro. I know I didn't actively have any agro, but I suppose I could have had a straggler on me as I was running around looking for a quest giver and encountered the ledge by accident. I just remember that about halfway through the fall I was like **********, this one might be it" and sure enough, I died. I didn't notice if I had anything on my enemies list so perhaps I am mistaken.
I do know prior to that fall I had been actively trying to kill myself from falls so I could get a sense of what a safe fall distance would be and never managed to die. However, when I fell off that last cliff I must have fallen for 5 seconds easy. I even got a knot in my stomach, which is rare anymore. I just assumed at that point that there is a safe distance up to a certain point that you would land with 1hp. I was assuming something like double or triple your MAXHP based on whatever formula they use to determine fall DMG.
"On a mountain of skulls in a castle of pain, I sat on a throne of blood. What was will be, what is will be no more. Now is the season of evil!"
~Vigo the Carpathian