There are some processors out there that don't have commercially released documents explaining what its instruction set is. Is there any way to find the instruction set through tampering or an external device if you have access to the processor?
In general no, simply looking at bits is not going to get you very far. Imagine one of those jumble puzzles but you have many more than 26 characters (the alphabet). You can do some analysis on bit patterns, but what will that give you, not knowing the architecture that could be an unconditional branch or some other popular instruction, without any knowledge you dont know what the popular instructions are. With a jumble puzzle assuming english for example you have inside knowledge what letters and words are common a can try to start from there, you have no inside knowledge in instruction width, etc.
If it happens to resemble a known instruction set you still have the same problem, have the modified the instructions, if the have taken undefines and made them new instructions you likely wont be able to tell what they are without inside the chip visibility (which jtag wont really give you at this level as the info jtag would give also requires internal information).