A screenshot of the actual quickaction would be helpful. We could troubleshoot the actual code, I do not use the OOTB quickaction, so I cannot know what it says.
My guess? You add the person via the quickaction, but there is a "business rule" that adds him/her again. It happened to me... several times