Search code examples
mysqllispcommon-lispclsql

cl-dbi query mysql from sbcl with error - Illegal :UTF-8 character starting at position 18


Today try to use cl-dbi and do some query to mysql. But it failed with following error:

debugger invoked on a BABEL-ENCODINGS:INVALID-UTF8-STARTER-BYTE in thread
#<THREAD "main thread" RUNNING {B3E2151}>:   Illegal :UTF-8 character starting at position 18.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):   0: [ABORT] Exit debugger, returning to top level.

((LABELS BABEL-ENCODINGS::UTF-8-DECODER :IN "/home/angelo/quicklisp/dists/quicklisp/software/cffi_0.14.0/src/strings.lisp")
#<unavailable lambda list>)

here is mysql character set:

mysql> SHOW VARIABLES LIKE 'character_set%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

mysql> SHOW VARIABLES LIKE 'collation%';
+----------------------+-------------------+
| Variable_name        | Value             |
+----------------------+-------------------+
| collation_connection | utf8_general_ci   |
| collation_database   | latin1_swedish_ci |
| collation_server     | latin1_swedish_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)

More test here:
1. I tried to convert whole database to utf-8 with suggestion from post, but it is still failed with the same error.

  1. 8/3/2015: Here I tried to query character_set and collation in current cl-dbi connection. It's result is following:
**Query:**
(setf query (dbi:prepare *connection*
             "SHOW VARIABLES LIKE 'character_set%';"))
(setf result (dbi:execute query))
(loop for row = (dbi:fetch result)
       while row
   do (format t "~A~%" row))
**Result:**
(Variable_name character_set_client Value latin1)
(Variable_name character_set_connection Value latin1)
(Variable_name character_set_database Value utf8)
(Variable_name character_set_filesystem Value binary)
(Variable_name character_set_results Value latin1)
(Variable_name character_set_server Value latin1)
(Variable_name character_set_system Value utf8)
(Variable_name character_sets_dir Value /usr/share/mysql/charsets/)
NIL
**Query:**
(setf query (dbi:prepare *connection*
             "SHOW VARIABLES LIKE 'collation%';"))
(setf result (dbi:execute query))
(loop for row = (dbi:fetch result)
       while row
   do (format t "~A~%" row))

**Result:**
(Variable_name collation_connection Value latin1_swedish_ci)
(Variable_name collation_database Value utf8_unicode_ci)
(Variable_name collation_server Value latin1_swedish_ci)
NIL

As "Fernando Borretti" suggested, after send "SET names 'utf8';", the query is completed successful and the issue is fixed. Here is test result for reference:

**Sending:**
(setf query (dbi:prepare *connection* "SET names 'utf8';"))

(setf result (dbi:execute query))
(loop for row = (dbi:fetch result)
       while row
   do (format t "~A~%" row))
**Query:**
(setf query (dbi:prepare *connection*
             "SHOW VARIABLES LIKE 'character_set%';"))
(setf result (dbi:execute query))
(loop for row = (dbi:fetch result)
       while row
   do (format t "~A~%" row))
**Result:**
(Variable_name character_set_client Value utf8)
(Variable_name character_set_connection Value utf8)
(Variable_name character_set_database Value utf8)
(Variable_name character_set_filesystem Value binary)
(Variable_name character_set_results Value utf8)
(Variable_name character_set_server Value latin1)
(Variable_name character_set_system Value utf8)
(Variable_name character_sets_dir Value /usr/share/mysql/charsets/)
NIL
**Query:**
(setf query (dbi:prepare *connection*
             "SHOW VARIABLES LIKE 'collation%';"))
(setf result (dbi:execute query))
(loop for row = (dbi:fetch result)
       while row
   do (format t "~A~%" row))
**Result:**
(Variable_name collation_connection Value utf8_general_ci)
(Variable_name collation_database Value utf8_unicode_ci)
(Variable_name collation_server Value latin1_swedish_ci)
NIL

Solution

  • Does sending SET NAMES utf8 help? I got the idea from the reply to this comment.