I had issues streaming directly from one device to the other without transcoding on WiFi. (I know you’re wired! Heard me out.)
I found that, although it didn’t fix the issue, it did help to switch from using SMB to NFS. Something about the way the protocol works meant that SMB had enough of an overhead that it worsened my stuttering issues outside of the spotty WiFi connection. For sure it significantly sped up scrubbing access times as well.
It may not be the issue, but it may be a step worth checking just to see if it is a part of the issue.
For what it’s worth, 4k remuxes can have bitrate spikes well exceeding the limits of a single gbps wire. If you have a player with limited memory, or just limited cache settings, this may also be a part of the problem.
It’s all read only, yes, but I just use a group specifically for NAS access and put users that need it in there.
I use the NFS version from the debian repository; not actually sure which one, and didn’t even know that it mattered.