I somewhat responded to this in the other thread about learning English as a foreign language. In 1985-87, I took 2 years of Japanese in college. Straight A's. I wasn't taking any classes, but from 87-90 I was still hanging out daily with Japanese friends I made during that time. From 91-93, I lived in Japan, employed by the JET program. From 94-99 I didn't have any involvement with learning Japanese, but in 99 I was introduced to a Japanese woman who came to the States for business several times a year, and I ended up marrying her in 2000. We've been married almost 17 years now. We speak Japanese around the house, we watch Japanese television every night, we go to Japan for vacation every 2-3 years.
And, I'm basically a functional illiterate. I can read at roughly a 2nd grade level. I can 'get' about 90% of what's being discussed on a variety entertainment or light comedy television show, but miss words a lot when the show is serious crime drama or has a lot of medical vocabulary.
On the other hand, my 13 year old daughter is pretty close to fluent for her age level, because my wife has been teaching her since age three. When we visit Japan, she actually attends a local school there for several days and has a class she belongs to. She's been 'growing up' with the same kids since first grade. Unfortunately next year will be her last time to go to school with them, as they will mostly split up the following year for high school.
Point is ... if you're not starting out young, really young, it's a serious struggle to gain real fluency. I have seen it done, but it takes living there and willfully abandoning English for the most part, which is becoming increasingly more difficult to do with the internet being at your fingertips now (and that most jobs for foreigners in Japan involve teaching English or translating, etc.). But if you want to be a functional illiterate like me, then get started asap. Totally doable lol.