I've wondered that about the Jasmine arc, why Angel didn't lose his soul then. Maybe because it was a false sense of happiness? A happiness with underlying malicious intent, in that Jasmine wanted to take away free will? If that is the case, does that point more toward the 'one single moment' theory rather than the confluence of events theory? I don't know. I haven't watched S4 in quite awhile so my memory is a bit dim regarding specific details, so I could be way off base.

I do agree with you about the baby Connor aspect. Connor was this unbelivable miracle for Angel, and I think that no matter how ecstatic he was over his son, he was as equally terrified and worried. Not only with the normal parental fears, but Connor was everything Angel had ever wanted--the ideal of family and love--and I believe his fear of damaging his child because of who he (Angel) is, was just as compelling in his mind as the danger of Holtz and W&H.