Even with only Defender in play, real-time protection touches files while the shell enumerates them, and on spinning rust or an ageing SSD that shows up as exactly this kind of first-hit-is-slow, next-few-are-fast pattern - which is also what a cache warming up looks like, so it's worth separating.
Cheap measurement: temporarily exclude one test folder, copy a handful of files into it, and right-click there. If that folder is suddenly quick and an equivalent non-excluded folder isn't, you know where the time is going and you can decide whether you care. Then remove the exclusion. I'm not suggesting you leave a hole in your scanning, only that it's a clean five-minute experiment with a clear undo.