I have a question regarding the bittorrent protocol: Assume we have a private tracker where we register and login to download torrent files/access magnets. We know that clients announce to the tracker to get a list of peers and then the client itself connects to different peers to download separate pieces of the whole file. Once the client finishes downloading or aborts, it sends an AnnounceFinished message to the tracker for stats/graceful shutdown of the client.
Now, let's say the tracker wants to maintain a download/upload rate (like it's commonly done in real trackers) to limit leechers from downloading too much without uploading. How does the tracker get real/legitimate statistics about the download/upload volume of a peer when the actual download/upload only happens between the peers themselves? Do they need to be honest and cooperate and send the right stats to the tracker or can the tracker somehow enforce the correct data?
Do they need to be honest and cooperate
More or less, yes.
or can the tracker somehow enforce the correct data?
Not in the general case of multiple colluding peers. In special cases of a single/few selfish downloaders it may be possible to detect them by cross-checking statistics from multiple peers but this is fragile and will require some fine-tuned heuristics to minimize false positives while still getting decent true positives.
It's basically a kludge, not something the protocol is designed for.
For the downloaded statistic a tracker could also participate in a swarm as a bittorrent peer and see the progress of a client based on have messages, but that doesn't work for upload and isn't all that reliable either because there's no guarantee that it could connect to all peers in a swarm to observe them.