Search code examples
jsondnsipv6avro

Avro tool fails to convert from json due to IPv6

I am trying to convert a simple json file to avro using avro tools (1.7.7).

The command I've been running

java -jar ~/Downloads/avro-tools-1.7.7.jar fromjson 
--schema-file src/main/avro/twitter.avsc tweet.json > tweet.avro

on this schema

{
"type": "record",
"name": "tweet",
"namespace": "co.feeb.avro",
"fields": [
    {
        "name": "username",
        "type": "string",
        "doc": "screen name of the user on twitter.com"
    },
    {
        "name": "text",
        "type": "string",
        "doc": "the content of the user's message"
    },
    {
        "name": "timestamp",
        "type": "long",
        "doc": "unix epoch time in seconds"
    }
],
"doc": "Schema for twitter messages"
}

I see this exception after running this command:

Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1436)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1337)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.avro.tool.Util.openFromFS(Util.java:88)
at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:82)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)

Caused by: java.lang.NumberFormatException: For input string: "810d:340:1770::1"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at com.sun.jndi.dns.DnsClient.<init>(DnsClient.java:127)
at com.sun.jndi.dns.Resolver.<init>(Resolver.java:61)
at com.sun.jndi.dns.DnsContext.getResolver(DnsContext.java:573)
at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:434)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141)
at com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
at sun.security.krb5.KrbServiceLocator.getKerberosService(KrbServiceLocator.java:85)
at sun.security.krb5.Config.checkRealm(Config.java:1120)
at sun.security.krb5.Config.getRealmFromDNS(Config.java:1093)
at sun.security.krb5.Config.getDefaultRealm(Config.java:987)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:81)

Trying to prefer IPv4 over IPv6 using -Djava.net.preferIPv4Stack=true didn't help. (I am running Mac OSX 10.10.3 and Java 1.8.0_25-b17).

Solution

  • Oh snap ... I've solved it myself right after posting this. My local router added an IPv6 nameserver to my local machine.

    Changing the assigned nameserver manually to Google's 8.8.8.8 fixed the issue.