It's supposed to be a jarring experience. It is supposed to shake you out of your expectation that every game in which you hold a gun is about using it and nothing more. This alone will easily turn off half the people who play Mafia II, but if you allow it to speak to you instead of tuning it out (and returning the game), you'll have taken the first step toward enjoying what is one of the most introspective and elevating gaming experiences ever created.
The fact is, Mafia II is the kind of game that developers have been threatening us with for years, and which is finally starting to become a reality: a game in which fun is not the entire point. Mafia II is not about fun. It is fun to play. There are even moments that are insane amounts of fun, but those moments are flashes in an otherwise deeply engaging experience that, while intensely entertaining, is not always "fun."
Yes, you can treat Mafia II like an amusement vending machine and ignore the story (at times) in favor of wringing as much mayhem-fueled entertainment out of it as you can, but to do so would be to ignore some of the best writing, voice acting, design and production (that is to say, the art of the industry) ever put into a game. Whereas, with a game like Grand Theft Auto IV, or Red Dear Redemption, you can be excused for avoiding the game's narrative in favor of simply riding around being a badass, but to do so with Mafia II would be like using a Picasso for wallpaper. In Mafia II, the story and setting and characters and mood and tone are not just polish or window dressing, they are the entire point of the game.
Mafia II is an open world game only in so far as it has created an seemingly open world, but that world is not sandbox so much as it is a stage set. It's entirely possible to bang on it until fun pops out, but the use for which it is intended is as an immersive environment in which the story plays out. As interactive as its systems may be, they are not games in and of themselves, but components of the larger experience of enjoying the story of Vito Scaletti and his two families; the Mafia crime family that serves as the backbone of the narrative and the family into which he was born.
At the beginning of the game, when Vito is returning home form the war, it's unclear which family will ultimately claim him. His mother and sister welcome him home with a hot meal and a place to sleep, but his friend Joey, a member of the local Mafia, welcomes him home with a drink, a pretty girl and false discharge papers, allowing Vito to avoid returning to the war. Comparing the two offers, one can hardly fault Vito for the choice he ultimately makes, but for a time, it seems he may not have to choose at all so long as he can keep his two lives separate.