I think it's all relative. For example, friends that I hang out with all the time couldn't care less about football. The most important thing you've got to ask yourself is whether or not you see people in the stands when you go see a football game, and to my knowlege, there has been no shortage of people filling in the stands.
Don't worry, the NFL is still America's pasttime. Baseball is on a pretty big upswing (thanks to Red Sox and Yankees), but it still does not compare with the NFL.