Search code examples
pythondjangodojo

Django Defect-dojo: aws-prowler upload failed

I've implemented django-DefectDojo and trying to upload This sample-scan file but it throws me the following error in the logs.

Everything is perfectly installed and other files are being uploaded(whichever I require) but this file.
Error Thrown
Server Error (500)
Logs

2/engagement/1/import_scan_results" "Mozilla/5.0 (X11; Linux x86_64)  
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87  
Safari/537.36" "-" uwsgi_1 | Internal Server Error:          
/engagement/1/import_scan_results uwsgi_1 | Traceback (most recent call         last): uwsgi_1 | File "/usr/local/lib/python3.5/site-    packages/pandas/core/indexes/base.py", line 2897, in get_loc uwsgi_1 |  
return self._engine.get_loc(key) uwsgi_1 | File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc uwsgi_1 | File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc uwsgi_1 | File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in  
pandas._libs.hashtable.PyObjectHashTable.get_item uwsgi_1 | File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item uwsgi_1 | KeyError: 'PROFILE' uwsgi_1 | uwsgi_1 | During handling of the above exception,  
another exception occurred: uwsgi_1 | uwsgi_1 | Traceback (most recent call last): uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner uwsgi_1 | response = get_response(request) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response uwsgi_1 | response = self.process_exception_by_middleware(e, request) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response uwsgi_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view uwsgi_1 | return view_func(request, *args, **kwargs) uwsgi_1 | File "./dojo/engagement/views.py", line 523, in  
import_scan_results uwsgi_1 | parser = import_parser_factory(file, t, active, verified) uwsgi_1 | File "./dojo/tools/factory.py", line 152, in import_parser_factory uwsgi_1 | parser = AWSProwlerParser(file, test) uwsgi_1 | File "./dojo/tools/aws_prowler/parser.py", line 21, in __init__ uwsgi_1 | profile = df.ix[i, 'PROFILE'] uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/indexing.py", line 142, in __getitem__ uwsgi_1 | return self._getitem_tuple(key) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/indexing.py", line 805, in _getitem_tuple uwsgi_1 | return self._getitem_lowerdim(tup) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/indexing.py", line 961, in _getitem_lowerdim uwsgi_1 | return getattr(section, self.name)[new_key] uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/indexing.py", line 148, in __getitem__ uwsgi_1 | return self._getitem_axis(key, axis=axis) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/indexing.py", line 1040, in _getitem_axis uwsgi_1 | return self._get_label(key, axis=axis) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/indexing.py", line 156, in _get_label uwsgi_1 | return self.obj._xs(label, axis=axis) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/pandas/core/generic.py", line 3737, in xs uwsgi_1 | loc = self.index.get_loc(key) uwsgi_1 | File  
"/usr/local/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc uwsgi_1 | return self._engine.get_loc(self._maybe_cast_indexer(key)) uwsgi_1 | File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc uwsgi_1 | File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc uwsgi_1 | File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in  
  pandas._libs.hashtable.PyObjectHashTable.get_item uwsgi_1 | File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in  
pandas._libs.hashtable.PyObjectHashTable.get_item uwsgi_1 | KeyError: 'PROFILE' uwsgi_1 | [pid: 1|app: 0|req: 4448/4448] 182.185.173.118 () {54 vars in 1263 bytes} [Sat Feb 8 06:29:30 2020] POST /engagement/1/import_scan_results => generated 27 bytes in 486 msecs (HTTP/1.1 500) 4 headers in 126 bytes (1 switches on core 0) nginx_1 | 182.185.173.118 - - [08/Feb/2020:06:29:30 +0000] "POST /engagement/1/import_scan_results HTTP/1.1" 500 27 "http://luminsec.synology.me:9092/engagement/1/import_scan_results" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" "-"

CSV I tried CSV I tried

CSV was tried for both of the below-given JSON formats but none of it worked.

JSON Variant 1(Current Error Log)

 {
    "Profile": "default",
    "Account Number": "302350317129",
    "Control": "[check11] Avoid the use of the root account (Scored)",
    "Message": "Root account last accessed (password key_1 key_2): 2018-07-17T20:14:33+00:00 N/A N/A",
    "Status": "Info",
    "Scored": "Scored",
    "Level": "Level 1",
    "Control ID": "1.1",
    "Region": "us-east-2",
    "Timestamp": "2018-10-25T17:21:25Z"
}
{
    "Profile": "default",
    "Account Number": "302350317129",
    "Control": "[check12] Ensure multi-factor authentication (MFA) is enabled for all IAM users that have a console password (Scored)",
    "Message": "User bevans has Password enabled but MFA disabled",
    "Status": "Fail",
    "Scored": "Scored",
    "Level": "Level 1",
    "Control ID": "1.2",
    "Region": "us-east-2",
    "Timestamp": "2018-10-25T17:21:27Z"
}

JSON Variant 2(Different Error Log, which says 1 parameter value was expected and provided are 2)

 [{
    "Profile": "default",
    "Account Number": "302350317129",
    "Control": "[check11] Avoid the use of the root account (Scored)",
    "Message": "Root account last accessed (password key_1 key_2): 2018-07-17T20:14:33+00:00 N/A N/A",
    "Status": "Info",
    "Scored": "Scored",
    "Level": "Level 1",
    "Control ID": "1.1",
    "Region": "us-east-2",
    "Timestamp": "2018-10-25T17:21:25Z"
},
{
    "Profile": "default",
    "Account Number": "302350317129",
    "Control": "[check12] Ensure multi-factor authentication (MFA) is enabled for all IAM users that have a console password (Scored)",
    "Message": "User bevans has Password enabled but MFA disabled",
    "Status": "Fail",
    "Scored": "Scored",
    "Level": "Level 1",
    "Control ID": "1.2",
    "Region": "us-east-2",
    "Timestamp": "2018-10-25T17:21:27Z"
}]
Solution

  • This is file parsing error, and DefectDoJo AWS Prowler Parser expects input file to be in .csv format with header row as in default prowler csv output - all capital letters.

    Example .csv file (based on your .json file) to be imported (although not tested) - data in the same order as was in .json, timestamp removed, header row added with names as expected by DoJo and matching data types:

    Accepted CSV format that matches (default) json data order, as some fields have different names in .json and .csv. Actual order does not matter for DoJo import.

    PROFILE,ACCOUNT_NUM,REGION,TITLE_ID,RESULT,SCORED,LEVEL,TITLE_TEXT,NOTES
default,302350317129,us-east-2,your title id,your result here,Scored,Level 1,your title text here1,your notes
default,302350317129,us-east-2,your title id,your result here,Scored,Level 1,your title text here,your notes