Import and parse a file to fill the form

Currently, I'm developing a custom app. So far I got the DocType ready to be filled in manually. We got files (SQLite3) that I'd like to upload, parse, extract the necessary fields of it and fill in the form. Basically like the import data tool. In my case, no bulk operation is needed and if possible do the extraction part server-side.

What I tried so far

I added a Server Action to call a whitelisted method of my app. I can get the current doc with:

@frappe.whitelist()
def upload_data_and_extract(doc: str):
    """
        Uploads and processes an existing file and extracts data from it
    """
    doc_dict = json.loads(doc)
    custom_dt = frappe.get_doc('CustomDT', doc_dict['name'])
    # parse data here
    custom_dt.custom_field = "new value from parsed data"
    custom_dt.save()
    return doc # How do I return a JSON back to the website from the updated doc?

With this approach, I only can do the parsing when the document has been saved before. I'd rather update the fields of the form when the attach field gets modified. Thus, I tried the Server Side Script approach:

frappe.ui.form.on('CustomDT', {
    original_data: function(frm, cdt, cdn) {
        if(original_data) {
            frappe.call({
                method: "customapp.customapp.doctype.customdt.customdt.parse_file",
                args: {
                    "doc": frm.doc
                },
                callback: function(r) {
                    // code snippet
                }
            });
        }
    }
});

Here are my questions:

What's the best approach to upload a file that needs to be parsed to fill the form?
How to access the uploaded file (attachment) the easiest way. (Is there something like frappe.get_attachment()?)
How to refresh the form fields in the callback easily?

I appreciate any help on these topics.

Simon

Solution

I have developed the same tool but that was for CSV upload. I am going to share that so it will help you to achieve your result.

JS File.

// Copyright (c) 2020, Bhavesh and contributors
// For license information, please see license.txt

    frappe.ui.form.on('Car Upload Tool', {
        upload: function(frm) {
            frm.call({
                doc: frm.doc,
                method:"upload_data",
                freeze:true,
                freeze_message:"Data Uploading ...",
                callback:function(r){
                    console.log(r)
                }
            })
        }
    });

Python Code

# -*- coding: utf-8 -*-
# Copyright (c) 2020, Bhavesh and contributors
# For license information, please see license.txt

from __future__ import unicode_literals
import frappe
from frappe.model.document import Document
from carrental.carrental.doctype.car_upload_tool.csvtojson import csvtojson
import csv 
import json

class CarUploadTool(Document):
    def upload_data(self):
        _file = frappe.get_doc("File", {"file_url": self.attach_file})
        filename = _file.get_full_path()
        csv_json = csv_to_json(filename)
        make_car(csv_json)
        


def csv_to_json(csvFilePath):
    jsonArray = []
    #read csv file
    with open(csvFilePath, encoding='latin-1') as csvf:
        #load csv file data using csv library's dictionary reader
        csvReader = csv.DictReader(csvf,delimiter=";")

        #convert each csv row into python dict
        for row in csvReader:
            frappe.errprint(row)
            #add this python dict to json array
            jsonArray.append(row)  
    #convert python jsonArray to JSON String and write to file
    return jsonArray

def make_car(car_details):
    for row in car_details:
        create_brand(row.get('Marke'))
        create_car_type(row.get('Fahrzeugkategorie'))
        if not frappe.db.exists("Car",row.get('Fahrgestellnr.')):
            car_doc = frappe.get_doc(dict(
                doctype = "Car",
                brand = row.get('Marke'),
                model_and_description = row.get('Bezeichnung'),
                type_of_fuel = row.get('Motorart'),
                color = row.get('Farbe'),
                transmission = row.get('Getriebeart'),
                horsepower = row.get('Leistung (PS)'),
                car_type = row.get('Fahrzeugkategorie'),
                car_vin_id = row.get('Fahrgestellnr.'),
                licence_plate = row.get('Kennzeichen'),
                location_code = row.get('Standort')
            ))
            car_doc.model = car_doc.model_and_description.split(' ')[0] or ''
            car_doc.insert(ignore_permissions = True)
        else:
            car_doc = frappe.get_doc("Car",row.get('Fahrgestellnr.'))
            car_doc.brand = row.get('Marke')
            car_doc.model_and_description = row.get('Bezeichnung')
            car_doc.model = car_doc.model_and_description.split(' ')[0] or ''
            car_doc.type_of_fuel = row.get('Motorart')
            car_doc.color = row.get('Farbe')
            car_doc.transmission = row.get('Getriebeart')
            car_doc.horsepower = row.get('Leistung (PS)')
            car_doc.car_type = row.get('Fahrzeugkategorie')
            car_doc.car_vin_id = row.get('Fahrgestellnr.')
            car_doc.licence_plate = row.get('Kennzeichen')
            car_doc.location_code = row.get('Standort')
            car_doc.save(ignore_permissions = True)
    frappe.msgprint("Car Uploaded Successfully")


def create_brand(brand):
    if not frappe.db.exists("Brand",brand):
        frappe.get_doc(dict(
            doctype = "Brand",
            brand = brand
        )).insert(ignore_permissions = True)

def create_car_type(car_type):
    if not frappe.db.exists("Vehicle Type",car_type):
        frappe.get_doc(dict(
            doctype = "Vehicle Type",
            vehicle_type = car_type
        )).insert(ignore_permissions = True)

So for this upload tool, I created one single doctype with the below field:

Attach File(Field Type = Attach)
Button (Field Type = Button)