I'm trying to implement lossy image compression algorithm with good ratio and (I hope) less artifacts that jpeg has. I plan to release encoder/decoder tool first. I've heard that there's some issues with jpeg2000 usage related to licensing and patents. So I want to avoid these problems with my algorithm (i've mentioned jpeg2000 only as example of license problems, my algorithm significantly differs from jpeg2000). I have also heard about problems using some things licensed under GPL. So the main question is all about what license to choose to make my work accessible for everyone and to prohibit some bad guys to do bad things to it to make it less accessible. I also want to know what disadvantages GPL has for image compression algorithm. Thanks.
There are a range of licenses you can choose from - see the Open Source Initiative site for details. Which you choose will depend on how strongly you feel about open source versus widest possible adoption of your software.
The GPL license ensures that all users of your library publish any changes they make to your code. With a few exceptions, it also compels them to make all of their product source code freely available. This means that GPL code may not be attractive to some commercial products so a GPL license may not be ideal if you want to encourage maximum adoption of your library.
At the opposite end of the spectrum, the MIT license is very liberal, effectively allowing users to do whatever they want with your library. It is suitable for both open source and commercial projects so would be a good choice to grow your library's user base. Users of your library are not obliged to publish any changes they make to your library so this license may fail your "bad guy" test.
A compromise choice might be the LGPL license. Users are obliged to publish any changes they make to your library but can often keep their code private. (This is typically true for desktop applications; it's much harder to do this for embedded or certain mobile applications).