Search code examples
load-balancingooziehaproxycloudera-manager

Oozie HA configuration with HAProxy load balancer


I am trying to enable Oozie HA using Cloudera Manager 5. The process is pretty straightforward. All I have to do is provide a hostname:port for the configured load balancer and Cloudera Manager creates an additional Oozie Server. I already have one Oozie Server on host XXX.XXX.XXX.246 and my additional Oozie Server should be on host XXX.XXX.XXX.247. HAProxy should be configured on host XXX.XXX.XXX.248. For hostname:port of the load balancer I chose XXX.XXX.XXX.248:11000. haproxy.cfg file:

timeout connect         10s
timeout client          1m
timeout server          1m
timeout http-keep-alive 10s
timeout check           10s
maxconn                 3000

frontend  main *:80

   acl url_static       path_beg       -i /static /images /javascript       /stylesheets
acl url_static       path_end       -i .jpg .gif .png .css .js

use_backend static          if url_static
default_backend             app

backend static
balance     roundrobin
server      static 127.0.0.1:4331 check

backend app
balance     roundrobin
mode http
stats enable
stats uri /haproxy?stats
option httpclose
option forwardfor
server  oozie1 XXX.XXX.XXX.246:80 check
server  oozie2 XXX.XXX.XXX.247:80 check

When I try to start Oozie Servers, I get the following exception:

java.lang.NoSuchFieldError: EXTERNAL_PROPERTY
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:777)
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:370)
at org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:772)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:586)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:430)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:343)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:287)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2566)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2056)
at org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:201)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:186)
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217)
at org.apache.oozie.util.ZKUtils.<init>(ZKUtils.java:141)
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154)
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

I am not sure which part of the configuration is wrong and what causes the problem. I would appreciate any help. Thanks!


Solution

  • I just want to give an update. The problem was a bug not so common for CDH 5.x. Due to an incompatibility with the version of Jackson used by Oozie and Hive, and depending on the order that jars are loaded into Oozie's classpath, Oozie may fail to start with the previously mentioned exception. Solution, if you are using parcels, is the following:

    1. Delete or move /opt/cloudera/parcels/CDH/lib/oozie/libserver/hive-exec.jar and /opt/cloudera/parcels/CDH/lib/oozie/libtools/hive-exec.jar.
    2. Download hive-exec--core.jar from the Cloudera repo and put it in /opt/cloudera/parcels/CDH/lib/oozie/libserver/ and /opt/cloudera/parcels/CDH/lib/oozie/libtools/.
    3. Download kryo-2.22.jar from the maven repo and put it in /opt/cloudera/parcels/CDH/lib/oozie/libserver/ and /opt/cloudera/parcels/CDH/lib/oozie/libtools/.

    The haproxy configuration and Oozie HA enabling should be straightforward now.