Search code examples
apache-nifi

Boolean condition evaluation in apache nifi using expression language doesnt make sense

I am new to apache nifi, expression languages. I have a requirement to validate the flow file attribute value and route based on that.

Here is what I have as an requirement.

UPDATE: Detailed requirement:

  • The Nifi workflow will read a json/csv data and process each record at a time. At the moment, we are working with considering the input data in json format.
  • Each json object will have attrs as (i have stripped down few other attrs which are not necessary for my question/this topic)
{
    "REC_YEAR": 2020,
    "DESCRIPTION": "test",
    "CURRENCY_CD": "USD",
    "CALCULATED_FLAG": "N",
    "CUR_IND": "Y",
    "START_DT": "2020-12-19 17:33:35",
    "END_DT": "9999-12-31 00:00:00"
}

  • Using "EvaluateJsonPath" processor, I read the json attrs into the flowfile attribute.

  • One of the attribute called CURRENCY_CD which can be either NULL/empty or if its not empty, then it should be of length 3 chars --> this is one of the key requirement (for validation of the incoming data).

  • So VALID data use cases are CURRENCY_CD can be null, empty or any 3 char string. Any other value for that attr is considered as INVALID input data and route to logging processor flow.

  • I am using "UpdateAttribute" processor to evaluate some logic using expression language query and store them under the flowfile attributes. enter image description here

  • I came up with the below logic to validate CURRENCY_CD requirement as:

    ${ ${CURRENCY_CD:isEmpty()} :or(${CURRENCY_CD:isEmpty():not():and(${CURRENCY_CD:length():equals(3)})}) }

  • Then i use "RouteOnAttribute" processor to check the value of this attribute and route accordingly to invalid flow (for logging) or pass the incoming data for further processing workflow stages.

Input data: Now, when I supply the input data as empty string in json, I'd expect it to be TRUE. but it ends up in FALSE always. what am I doing wrong here?

Input Sample #1

{
    "CURRENCY_CD": ""
}

Input Sample #2

{
    "CURRENCY_CD": null
}

For either of these values, empty string or null, nifi marks those respective attributes with values as "Empty string set". enter image description here

If I try to break this logic into multiple attributes to check the expression boolean conditions separately, it does evaluates correctly. The Or() condition to combine them is where I expect to be true but it evaluates to false.

enter image description here

enter image description here

Then, i tried using like this as well, enter image description here

${
    ${Is-CURRENCY_CD-Empty:equals(true)}
    :or(${Is-CURRENCY_CD-NotEmptyAndLen3:equals(true)})
}

and store it in an attr "Is_CurrencyCode_Valid" and expect it to be true, but in turn, it comes with false. enter image description here

Solution

  • It's a bit diffcult to understand your post - could do with being a bit more clear.

    RouteOnAttribute takes one or more Dynamic Properties with some Expression Language that must evaluate to True or False.

    If the Dynamic Property's Expression Language evaluates to True, the FlowFile is directed to a Relationship with the name of the Dynamic Property.

    If the Dynamic Property's Expression Language evaluates to False, it will try the next Dynamic Property (if there is one) - if there are no more Dynamic Properties left, and all have returned False, then the FlowFile is sent to the unmatched Relationship.

    If your data is always either null/empty (Negative) or contains 3 chars (Positive) then there is no need to check for both conditions - simply check for either the Negative or Positive condition.

    E.g.

    One Dynamic Property called valid with the expression ${CURRENCY_CD:length():equals(3)}

    With this configuration, all FlowFiles where CURRENCY_CD has 3 characters will be routed to the valid Relationship. All FlowFiles that do not have 3 characters, including null/empty, will be routed to the unmatched Relationship.

    With that, the following would happen:

    CURRENCY_CD = TRI goes to valid

    CURRENCY_CD = null goes to unmatched

    CURRENCY_CD = testing goes to unmatched (we are assuming here that this won't be present in your data)

    If you can't make that guarantee and some data might fall outside of those 2 conditions, you could add a second Dynamic Property named empty with the expression ${CURRENCY_CD:isEmpty()}. This would route all null/empty FlowFiles to the Relationship empty.

    With that, the following would happen:

    CURRENCY_CD = TRI goes to valid

    CURRENCY_CD = null goes to empty

    CURRENCY_CD = testing goes to unmatched

    It's hard to tell from your post, but if both empty/null and 3 characters are the Postive condition and you want to validate for both in one Expression then you can simplify your Expression to this:

    ${CURRENCY_CD:isEmpty():or(${CURRENCY_CD:length():equals(3)})}

    Which would return True for both empty/null or 3 characters and forward them to the same Relationship.

    With that, the following would happen:

    CURRENCY_CD = TRI goes to valid

    CURRENCY_CD = null goes to valid

    CURRENCY_CD = testing goes to unmatched