Tiny_tds gem contains native extensions which is not supporting in Jruby. By setting JRUBY_OPTS to "--1.9 -Xcext.enabled=true" in user variables also not made it work. Please advice how to make it work in Jruby 1.7.6.
C-ext are not supported by JRuby. they were working to some extend in older versions but the support was never complete (and was dropped) and its very likely tiny_tds uses pieces of the MRI-C API that are not implemented.
people usually use alternatives on JRuby (assuming the gem does not provide native version for C-Ruby as well as JRuby - which many do). for connecting to MS-SQL there's JDBC drivers which will work under Java.
although the API is different than that of tiny_tds you can get emulate some of its pieces - depending how much of tiny_tds you really need, if its too much you should look into libraries such as Sequel or ActiveRecord which can work with MS-SQL using JDBC drivers.