It does really come down to availability of actual games for VR headsets. And the issue is that there probably aren't any, or any worthwhile ones in development. Let us speculate a moment.
Let's assume that all VR headsets are the same and just work on whatever platform. Let us also assume that we are a publisher and have "an amount of money" to spend to develop one unit of "game". Right now, as a publisher we can choose to spend our money to make our "game" for one or more of several platforms out there: PC, PS4, XBone, PS3/360, Wii U, 3DS, Vita, Android/iPhone/Windows Phone, others. Mobile development tends to go hand in hand, such that one game will routinely be peddled on all the smartphone platforms. AAA games are usually cross-platform for PC and both of Sony's & MS's latest console.
All the above said, VR exists on PC and PS4 only, as of right now (and according to this article won't be adopted by MS any time soon). Of the people who own PCs and PS4s, without specifying a number, I think it's fair to say that only a small percentage actually has VR hardware. Now let us look at what kind of game we can choose to make. We can make: a standard AAA game to use controller/keyboard for PC, PS4, XBone. We can make a game for VR only. We can make a game that works both with VR and without, and using a normal controller/keyboard.
Do we spend our money to make a VR only game, the kind that actually uses the tech to it's full advantage, making immersive, unique experiences BUT which only the small percentage of PC & PS4 users who own the hardware can enjoy? Do we make our game VR capable, but also able to work without, thus getting more potential customers, at the expense of making a unique, VR experience since one can actually play it without? Or do we spend our money to target the usual PC/XBone/PS4 crowd?
Financially speaking, it is an almost impossible sell to any corporate exec at a game publisher, to spend money and develop a full game with a AAA budget for a fraction of the gaming audience. Yet, that is exactly what needs to happen if the technology will ever have anything to showcase it other than tech demos. Making a game that can be played with or without VR isn't the answer, since any VR capability then will be an unneeded gimmick bolted onto a normal game (such as Skyrim/ME3/Ryse:SoR with Kinect voice commands).
As it is now, we have expensive hardware that doesn't offer anything unique, so it's a tough sell to us consumers. And we have hardware few consumers have, so it's a tough sell to game developers. Something needs to give if VR will ever take off...either the hardware needs to become widely adopted enough that it increases demand for VR games, or the supply of VR games needs to go up to convince people to buy the hardware. On top of this, retracting an earlier assumption, I'm fairly sure there are in fact differences between headsets that suggests not all VR experiences might work on all competing headsets, fracturing an already small audience further.
One other point that might be worth noting...the cost of the hardware is such that, if we're being brutally honest, isn't going to sell a great deal to audiences in second or third world countries. America/Canada, Euro, Aus/NZ and Japan are really the main potential audiences as most of Asia, Russia, S. America, Africa aren't places that $500+ shiny tech will sell in high numbers.
In my brutally honest opinion, I think overall VR is too hard a sell and will be relegated to the same niche currently occupied by Kinect (remember the shenanigans surrounding XBone/Kinect integration at launch and its subsequent reversal?) and other such things. The asking price of a) $500+ and b) a large, uncluttered room in which to fit it is a restrictively high barrier to entry even assuming c) there are sufficient people who want to game standing up, moving around and who don't get motion sickness/headaches. The fact that on top of these barriers, there's nothing waiting for us in VR-Land as of the time of writing (months after Vive and shortly after PS4 VR hardware have been on sale) mean it's an uphill battle few people are even fighting.