Why are Fate fans allergic to aspects effecting mechanics?

Every time someone asks, "Hey, shouldn't this situation be harder because of this aspect?" the response is always:

"No, aspects don't do anything unless you invoke them or spend a fate point!"

• Fighting enemies inside a smokescreen?
  Nah, the smokescreen apparently doesn’t matter unless someone decides to throw a fate point into the mix. It’s just there for the ambiance, I guess.

• The bandit captain is literally on fire?
  Well, sure, it was a problem… until you used up that invoke last round. Now the flames are just a minor inconvenience unless you fork over another fate point. Guess being engulfed in flames isn’t that big of a deal anymore.

Why is it such a crime for the GM to:

• Make everyone in the smokescreen roll Notice at the start of their turn to see if they can even figure out where they are, let alone who to attack? Oh no, you failed? Maybe you swing at your buddy by accident.

• Decide that someone with an "On Fire" aspect should take 1 point of stress at the end of their turn until they put out the flames. You know, because being on fire is generally not good for your health.

• Say, "Hey, PC, you’ve got a 'Grappled' aspect on you. While you’re grappled, you can’t use Athletics to dodge, only Fight." You can't really neo dodge bullets while wrestling with someone.

It feels like there's this unwritten rule that aspects are just pretty little decorations unless someone decides to spend a fate point. But really, shouldn't some of these aspects have a consistent impact without needing a resource every single time?

Why are we so scared to make aspects actually matter outside of invoking them?