Search code examples
marklogicmarklogic-dhf

How to check mandatory fields are present in the CSV while data ingestion in marklogic mlcp

I want to check few mandatory fields are available in CSV while ingestion of data through MLCP in marklogic. If those fields are not available, I need to ignore those records has to be ingested in marklogic

Solution

  • There are many different ways to approach this, here are a few:

    Since you are using DHF, I would first look at using validateEntity in a flow step. This would prevent those records from making it to the MLCP step in the first place.

    Alternatively, If those fields are going to be searched regularly, you could configure a Range Index Range Indexes, which can be configured to reject invalid values, so those documents would fail to ingest. The upside is that this will improve search performance for those searches, the downside is it increases memory

    Another option would be to use a Schema, which can also be used to validate documents, rejecting the ones that don't meet the validation rules.