What is the clutch on a modern rear derailleur actually doing, and should I ever turn it off?

It is a one-way friction device, and the one-way part is the whole trick.

The derailleur cage is sprung to pull the chain tight. Over bumps, the chain slackens for an instant and then snaps back, which is the slapping you hear and, occasionally, how chains come off. The clutch adds friction that resists the cage being pulled forward: the direction it moves when the chain goes slack and then whips, while adding much less resistance to the cage moving backward, which is what happens when you shift onto a bigger sprocket.

So it is not a brake in both directions. If it were, shifting would be genuinely awful. It is closer to a damper that is stiff one way and soft the other, usually a wrapped band or a roller clutch on the cage pivot.

That asymmetry is why you can leave it on permanently, which is what it is designed for.

One-way friction is the bit that makes it make sense. It resists the direction chain slap pulls and not the direction shifting needs.

Testing it by hand in a direction you never load while riding is how everyone forms the wrong impression.

On question 2, you are not imagining it, but you are probably attributing it wrongly.

There is a small amount of extra effort at the lever with the clutch on. On a healthy, clean drivetrain it is barely perceptible. What makes it very perceptible is a cable that is dragging, a dirty or kinked outer, or a housing that has gone stiff. The clutch adds a constant load, and a marginal cable that was just about coping without it becomes obviously heavy with it.

So if shifting feels notably heavy with the clutch on and fine with it off, that is not a reason to leave it off. It is the clutch telling you the cable needs attention. Fresh inner and outer usually returns it to almost no difference.

Worth knowing because "I turned the clutch off, shifting improved" is a very common conclusion and it is treating the symptom.

For question 3, the honest list of when to switch it off is short:

• Removing the wheel, as you said. With the clutch on, the cage will not swing forward easily and getting the wheel past the chain is a fight. This is what the switch is mostly for.
• Working on the chain: anything where you want slack.
• Some people turn it off for cleaning so the cage moves freely.

That is genuinely it for riding. There is an old argument that it costs a watt or two of drivetrain friction, which is true and irrelevant unless you are racing on a smooth road, and on a smooth road the chain is not slapping anyway.

The failure mode to know about: the clutch loses effectiveness over time as the friction material wears or the grease in it dries. If the chain starts slapping again with the switch on, it usually needs servicing rather than replacing - most are adjustable, and it is a documented procedure for your model rather than something to guess at.

One thing worth checking on a bike that is new to you: whether the clutch has ever been serviced.

They have a service interval that essentially nobody observes, and a five-year-old derailleur that has never been touched often has a clutch that is either seized solid or doing nothing. If yours feels dramatically different between on and off, it is alive and well. If the two positions feel similar, the mechanism is probably dry and the switch is decorative at this point.

Cheap test, and it tells you whether you are getting the feature you think you are getting.