Most classic games that we don't think of as memory games actually strengthen our recall just the same, often giving a workout to our sequential memory. For example, while true "card counters" are rare -- and generally distrusted -- almost any card game requires us to remember which cards have already been played, in a basic way.
But what about games like Clue, 20 Questions or even chess? In each case, remembering what's been played, guessed or ruled out is essential to knowing how to move forward. On a more sophisticated level, the best chess players are able to amass an understanding of their opponent's behaviors as the game is being played.
Or consider the Rubik's Cube. You could look at it as using very complicated locational memory -- remembering where each of the squares along the outside of the cube is located, and how to move them correctly -- but in practice, the Rubik's Cube is a bit more like chess. Remembering where you've been, and how to get back to a previous state once you've hit a brick wall, takes up more mental power than any other aspect of the game.
In all cases, however, it's a complicated mixture of physical, sequential and visual memory that games help us strengthen -- and, if you're having fun, you might not even notice.