LUIS: Batch testing returning incorrect results for entities

Question

I'm training a LUIS model to recognize the intent to change the date of birth or expected date of birth for an individual in a database. In addition to a datetimeV2 entity it is expected that a number:MemberID or number:FamilyID will be detected too. If it isn't that will be handled programmically.

Example Utterance - https://i.sstatic.net/yjXbM.png

My issue is that when batch testing LUIS it labels True Positive results for the builtin.number as False Positives. I don't know if this is a bug with LUIS or something wrong with my model or batch files. I've labeled screen shots below of the batch training results and will add a snippet from the batch test file.

I've attempted retraining the model and combining intents but no such luck. I've added up to 60 utterances for each intent but I end up with an over trained model with more problems than when I started. Additionally, I've added patterns but again this does not help.

Batch Results Graph - https://i.sstatic.net/S5ONN.png

False Positive Utterances - https://i.sstatic.net/JQ30N.png

[
  {
    "text": "change dob to 09/16/2013 for Charles Patterson in family 53183",
    "intent": "ChangeDOB",
    "entities": [
      {
        "entity": "datetimeV2",
        "startPos": 14,
        "endPos": 24
      },
      {
        "entity": "FirstName",
        "startPos": 29,
        "endPos": 36
      },
      {
        "entity": "LastName",
        "startPos": 37,
        "endPos": 46
      },
      {
        "entity": "number:FamilyID",
        "startPos": 57,
        "endPos": 62
      }
    ]
  },
  {
    "text": "change birth dt for 47224 to 10/22/2015",
    "intent": "ChangeDOB",
    "entities": [
      {
        "entity": "number:MemberID",
        "startPos": 20,
        "endPos": 25
      },
      {
        "entity": "datetimeV2",
        "startPos": 29,
        "endPos": 39
      }
    ]
  },
  {
    "text": "repair birthday to 04/30/2019 in family 84842",
    "intent": "ChangeDOB",
    "entities": [
      {
        "entity": "datetimeV2",
        "startPos": 19,
        "endPos": 29
      },
      {
        "entity": "number:FamliyID",
        "startPos": 40,
        "endPos": 45
      }
    ]
  },
  {
    "text": "alter birthday to 01/18/2004 for Harry Taylor in family 31902",
    "intent": "ChangeDOB",
    "entities": [
      {
        "entity": "datetimeV2",
        "startPos": 18,
        "endPos": 28
      },
      {
        "entity": "FirstName",
        "startPos": 33,
        "endPos": 38
      },
      {
        "entity": "LastName",
        "startPos": 39,
        "endPos": 45
      },
      {
        "entity": "number:FamilyID",
        "startPos": 56,
        "endPos": 61
      }
    ]
  },
  {
    "text": "change date of birth to 11/15/2003 for 87546",
    "intent": "ChangeDOB",
    "entities": [
      {
        "entity": "datetimeV2",
        "startPos": 24,
        "endPos": 34
      },
      {
        "entity": "number:MemberID",
        "startPos": 39,
        "endPos": 44
      }
    ]
  }
]

Ultimately, the batch results should be displaying True Positives for the utterances that it is currently labeled False Positives because they do contain a number:MemberID which it detects correctly.

Answer 1

Actually, you're using the entity role feature to label number:MemberID and number:FamilyID into subtypes, correct? If this is the case, then this is the reason why your batch tests are not working.

Specifically in the LUIS docs on batch testing, it says this:

So it appears you cannot use batch testing to test utterances with these types of entities with roles.

You could request that this feature be worked on via UserVoice--forum that helps our team get feedback from the community and perhaps help prioritize what features we should work on first.