Search code examples
mysqlsqlhtml-entities

How do I search for invisible characters in a database table column?


The database table in question has a unique username column. When the usernames are listed in phpMyAdmin or SqlYog, a duplicate for at least one username appears.

When the username is displayed on a web page and the HTML is inspected, it can be seen that the duplicate username is preceded by an HTML entity ‏ (right-to-left mark) which is not shown on the page because it has no visual representation or associated symbol.

Trying to search for the entity:

SELECT * FROM mytable WHERE username LIKE '%‏%'; 

returns no results, because the invisible symbol not the entity is stored.

When exporting to Excel from SqlYog, it appears as †in Excel.


Solution

  • Remove the visible characters that are normally in usernames from each username, then check which usernames are still not empty but appear empty.

    SELECT username, 
           REGEXP_REPLACE(username, "[\\.,{}<>\'a-z0-9@_-]", '') AS second_username 
      FROM mytable 
      HAVING second_username <> '';