In a recent interview with Rohn Shmidt, The head cinematographer for the walking dead TV show, this question was asked:

Q: Would you rather be stuck in a prison with walkers or inmates?

A: I think the inmates. I'm very much a man of reason, so I'd rather have someone I could maybe talk with and negotiate with.

The implication here is that the inmates WILL be giving the group trouble in the next season- which was pretty much a given already, but it's nice to see suspicions confirmed. Sounds like the prisoners are openly hostile to the group at the outset of the season. maybe they've already gotten to the armory?

The rest of the interview can be found Here