This is a reference to the state of mind of Gollum soon after he got the One Ring, by killing his cousin Deagol. For his deed, he's been rejected by everyone.
This also alludes to the hatred (and yet love) Gollum feels towards the One Ring. The Ring betrayed and cursed him, made him suffer and provoked his loneliness.
Just remember the beginning of the third movie 'The Return of the King', in which we see the "birth" of Gollum, overtaking Smeagol, after the discovery of the Ring and the murder of Deagol. Smeagol/Gollum says: "They cursed us. Murderer. Murderer they called us. They cursed us and drove us away. And we wept, Precious. We wept, to be so alone."
The "we" is due to the schizophrenic-like personality of Gollum, which we can perceive at many moments in the movie and in the books.