self.button = gtk.Button(stock=gtk.STOCK_DELETE)
Only Shows: Delete
This is a recent change in GTK - the developers wanted icons not to appear on buttons. On Linux, this can be changed by editing the gconf key
/desktop/gnome/interface/buttons_have_icons
On windows, I think (I haven't actually tried this) that you need to set a value in your gtkrc file (for me it's in C:\Program Files\Gtk+\etc\gtkrc) and use a theme that supports icons (I think the default one doesn't).
You can also add gtk-button-images = 1 to your ~/.gtkrc-2.0 file after setting the theme which may over ride the option from gconf.
EDIT in answer to your comment:
Just like this answer, but in Python: In Gtk, how do I make a Button with just a stock icon?
For python, it's just
image = gtk.Image()
# (from http://www.pygtk.org/docs/pygtk/gtk-stock-items.html)
image.set_from_stock(gtk.STOCK_**)
button = gtk.Button()
button.set_image(image)
button.set_label("")