I've recently started messing around with Java sockets and telnet...
I want the user to be able to connect to the server, just type a letter and have it sent to the server, without pressing enter to send it. I'm sure there's no way for the server to set this up, but maybe telnet has a parameter or something which could allow this?
Maybe if I got the user to type stty cbreak or stty raw before running telnet, this would work? (UNIX only, I know!)
If I can get telnet to do this then it saves me having to write a special client just for this feature...
You should be able to do this with telnet option negotiation. The protocol defaults to half-duplex mode, and at a minimum for an interactive session, the server should negotiate the suppress go ahead option and echo option.
At the bare minimum you could just spit out ff fb 01 ff fb 03 (will echo, will suppress-go-ahead) at the begining of the session, then reply to any ff fd 01 (do echo) with ff fb 01 (will echo) and reply to any ff fd 03 (do suppress-go-ahead) with ff fb 03 (will suppress-go-ahead).
Edit to add that the linemode negotiation mentioned by Ben Jackson is a better answer. Suppress go-ahead won't be enough for most clients connecting on ports other than 23.
However I think the other problem you're running into is that Java is sending Unicode characters. For example, when you say (char)0xff, Java assumes you're referring to UTF-16 character U+00ff which is ÿ. It's probably sending it over the socket using UTF-8 encoding, so the telnet client sees two bytes: c3 bf which it passes on and displays as ÿ.
What you can do is explicitly tell Java to use ISO-8859-1 encoding. For example, you may have been doing something like this before:
out = new PrintStream(connection.getOutputStream());
out.print((char)0xff); // sends 0xc3 0xbf
out.print((char)0xfb); // sends 0xc3 0xbb
out.print((char)0x01); // sends 0x01
out.flush();
Instead, you can use the OutputStreamWriter to specify the encoding you want:
out = new OutputStreamWriter(connection.getOutputStream(), "ISO-8859-1");
out.write((char)0xff); // sends 0xff
out.write((char)0xfb); // sends 0xfb
out.write((char)0x01); // sends 0x01
out.flush();