I am attempting to parse Shodan query results and write a new JSON file with only the results that match the criteria I have set.
Example JSON entries:
{
"matches": [
{
"product": "Microsoft IIS httpd",
"hostnames": [],
"hash": -1722221328,
"ip": 1261462342,
"isp": "AT&T Internet Services",
"transport": "tcp",
"cpe": [
"cpe:/a:microsoft:iis:7.5",
"cpe:/o:microsoft:windows"
],
"data": "",
"asn": "AS7018",
"port": 631,
"version": "7.5",
"link": "Ethernet or modem",
"location": {
"city": null,
"region_code": null,
"area_code": null,
"longitude": -97.822,
"country_code3": "USA",
"latitude": 37.751000000000005,
"postal_code": null,
"dma_code": null,
"country_code": "US",
"country_name": "United States"
},
"timestamp": "2017-02-28T23:55:24.306344",
"domains": [],
"org": "AT&T Internet Services",
"os": null,
"_shodan": {
"crawler": "122dd688b363c3b45b0e7582622da1e725444808",
"id": null,
"module": "http-simple-new",
"options": {}
},
"ip_str": "75.48.99.70"
},
{
"hash": 605323305,
"ip": 1757819678,
"isp": "Google Cloud",
"transport": "tcp",
"data": "",
"asn": "AS15169",
"port": 9000,
"hostnames": [
"30.51.198.104.bc.googleusercontent.com"
],
"location": {
"city": "Mountain View",
"region_code": "CA",
"area_code": 650,
"longitude": -122.0574,
"country_code3": "USA",
"latitude": 37.41919999999999,
"postal_code": "94043",
"dma_code": 807,
"country_code": "US",
"country_name": "United States"
},
"timestamp": "2017-02-28T23:51:35.997036",
"domains": [
"googleusercontent.com"
],
"org": "Google Cloud",
"os": null,
"_shodan": {
"crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
"id": null,
"module": "https-simple-new",
"options": {}
},
"ip_str": "104.198.51.30"
}
],
"total": 2
}
My hope is to load the JSON file and iterate over the set of elements, removing an element if it does not match the criteria of having a location country_code of "US".
The code I have (courtesy of https://gist.github.com/madonnelly and Iterate over JsonObject properties) is as follows:
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class ParseJSON {
public static void main(String[] args) {
JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");
Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();
for (Map.Entry<String, JsonElement> queryResult : queryResults) {
JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
for (JsonElement locData : locArray) {
if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
System.out.println(locData.getAsString());
}
}
}
}
public static JsonObject convertFileToJSON(String fileName) {
// Read from File to String
JsonObject jsonObject = new JsonObject();
try {
JsonParser parser = new JsonParser();
JsonElement jsonElement = parser.parse(new FileReader(fileName));
jsonObject = jsonElement.getAsJsonObject();
} catch (FileNotFoundException e) {
}
return jsonObject;
}
}
When I run my code I am receiving the error
Exception in thread "main" java.lang.IllegalStateException: Not a JSON Object: [{"product":"Microsoft IIS httpd","hostnames":[],"hash":-1722221328,"ip":1261462342,"isp":"AT&T Internet Services","transport":...}] at com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:90) at com.cti.shodan.ParseJSON.main(ParseJSON.java:22)
I am sure I am making a ton of mistakes and am hoping someone can point out the error I am making. Thanks in advance!
You have some assumptions about the concrete JSON document parsing that do not meet its real structure. I'm assuming you're going to display the matched results ($.matches) filtering out on their subproperty values ($.matches.*.location.country_code).
for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
final JsonElement value = queryResult.getValue();
// This is necessary to skip <"total": 2>
if ( value.isJsonArray() ) {
// Here comes an array, and should be iterated, rather than taken as an object
for ( final JsonElement match : value.getAsJsonArray() ) {
// This was the root cause, not an array
final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
// Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
// Previously getAsString() -- it requires a JSON string literal, just remove it
System.out.println(match);
}
}
}
}
With Java 8 it may be somewhat more simple:
shodanJsonObject.entrySet()
.stream()
.map(Entry::getValue)
.filter(JsonElement::isJsonArray)
.map(JsonElement::getAsJsonArray)
.flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
.peek(System.out::println)
.map(JsonElement::getAsJsonObject)
.map(jsonObject -> jsonObject.getAsJsonObject("location"))
.filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
.forEach(jsonObject -> {
}); // forEach is a terminal operation and it "pushes" the entire chain above
And probably the most expressive way if it's possible to use querying libraries like JsonPath:
final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(@.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
System.out.println(match);
}