You have frightened him/her. Maybe he/she won't invite you next time.