Can Luigi file targets be properly managed using pandas file i/o functions instead of open/close

Question

I have a script that relies heavily on pandas file i/o functions like read_csv(), to_csv(), etc. with the many custom parameters they offer.

Can these be used in a luigi pipeline for local file targets?

I want to convert it to a luigi pipeline but every example I have seen uses open/close python i/o with custom parsing.

I hear that the luigi target object functionality relies specifically on target.close() or 'with' for the dependency tracking.

Answer 1

Yes. LocalTarget can be used with the output of arbitrary code. The file tracking relies on the exists() method. open() is just a convenience method.