Broken Fifth Metatarsal - When can I remove the Boot!

Hi,
First off, I'm thankful to have found an active community here. Here's my situation. I broke my fifth metatarsal over 8 weeks ago. At first it was non-displaced, and put in a CAM boot and advised no weight bearing for 6 weeks. But by the week 2 appointment, it had shifted. I have really struggled with this injury, and it has brought me some consolation to go against doctor's orders, and walk on it (albeit, very very little, like barely putting my foot down with the aid of crutches, or just in moving from my knee scooter to the toilet). So, my walking on it may have caused the shift, but I think it probably shifted when my toddler stepped on it. That is the only time it was really painful in that first two weeks. Either way, it got displaced, but not so bad that surgery was required.

I am now 8 weeks from the injury. I have not had any pain since the first week. I was allowed to start walking in the boot at week 6, and have had no pain with that (some fatigue at the end of the day, some heel pain from walking oddly in the boot, but no pain at the fracture site). My 4 week X-Ray showed very little callus growth, and my 7 week X-Ray showed very little more. My doctor wants to keep me in the boot for another 4 weeks before going back.

I know I am in the "home stretch" here, and I am able to function well with the boot on, so I am fortunate there. But I am still really struggling with this thing mentally. I am a mid-30s male in pretty good shape, and have taken additional supplements in hopes of a quicker recovery. I had really high hopes that I could ditch the boot last week. What's more, the doctor said he would be relying more on his physical examination (the fact that there is no pain) than on the X-Ray results. But, he cited the lack of results on the X-Ray as the reason for keeping me in the boot for 4 more weeks. I've read that it is very common not to see healing on the X-Ray until 8+ weeks, but many doctors consider their patients healed when the pain is gone.

I obsess over this injury, and read everything online. This definitely leads me to more depression.

Also, I am very discouraged by the conservative treatment. My doctor says it should still heal without surgery, but he scares me by even mentioning it. I have read so much online that said these fractures rarely, if ever, need surgery, and rarely have non-unions. Many doctors (so I've read online) would allow their patients to walk on this fracture initially, without the 6 week non-weightbearing period), and allow removal of the boot by week 6 or 8, or as soon as pain subsides. Why is my doctor being so conservative? Is there really a risk of non-union here, or a need for surgery? I know everyone's fracture is different, but I really feel it comes down more to the doctor that you get as far as how you are treated. Is there any reason for this to be treated so conservatively? I'm sure the displacement that occurred in the first week is setting me back, but everything seems to indicate this is an acceptable level of displacement.

Does anyone have similar stories about initially being non-displaced, but later displacing? And how did it end up for you?

Can anyone speak to feeling no pain after 6-8 weeks but still not showing healing on the X-Ray? Were you "released" without the boot, or instructed to keep it on?

Other than stop reading stuff online, does anyone have any advice for dealing with the depression?

Thanks for any help I can get.