Below query is used to get the test cases present in a test set.But the response returns all the test cases present in the project irrespective of the test set id passed in the query.
I guess you mean the test instances in a test set and this URL works for me:
https://alm/qcbin/rest/domains/[domain]/projects/[project]/test-instances/?query={cycle-id[1782]}