Search code examples
javahttpyoutube-apijavafx-webengine

How can I work around YouTube API embed restrictions like other websites?

I am building a java program that has the option to play YouTube videos in an embedded player. The problem is that most of the music videos won't play and I get the following error: "This video contains content from (Media Corporation Name). It is restricted from playback on certain sites."

enter image description here

I tried loading the same URL in Chrome and got the same results. https://www.youtube.com/embed/TMZi25Pq3T8

However, after some research, I quickly got it fixed by installing a Chrome Extension that allows me to add HTTP Request Headers and added a Referer header that follows this structure "https://www..com" and got it working.

enter image description here

So I thought that must be it. I added the following code in order to add request headers to my JavaFX WebView / WebEngine:

URI uri = URI.create("https://www.youtube.com/embed/TMZi25Pq3T8");
List<String> cookies = new ArrayList<>();
cookies.add("User-Agent=BDM/v0.92");
cookies.add("Referer=https://www.youtube.com");
Map<String, List<String>> headers = new LinkedHashMap<String, List<String>>();
headers.put("Set-Cookie", cookies);
try {
    CookieHandler.getDefault().put(uri, headers);
} catch (IOException ex) {
    ex.printStackTrace();
}
System.out.println(webView.getEngine().getUserAgent());
webView.getEngine().load(uri.toString());

Still, no success, the same error message.

The website that I'm using to extract data about releases through their API, Discogs, is able to play "restricted" videos as well. What am I missing here?

LATER EDIT: Further clarifications:

I would like to apologize for the mistakes I made:

  1. The line System.out.println(webView.getEngine().getUserAgent()); doesn't print "BDM/v0.92" as I first stated, it prints the default JavaFX user agent, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/538.19 (KHTML, like Gecko) JavaFX/8.0 Safari/538.19". And this leads to number 2
  2. As Roman Nazarenko pointed out, I was confusing cookies with request headers.

This leads to the real question, how can I send HTTP Request headers for JavaFX WebEngine? The only option is to set the user agent by calling webView.getEngine().setUserAgent("myUserAgent");

I found a hack here but this didin't work for me: https://twitter.com/codingfabian/status/524942996748652544

Thanks!

Solution

  • I managed to solve the issue by using javassist and this tutorial on how to instrument Java code.

    As I stated in my question, the YouTube player needs a Referer header to play some videos (like music videos owned by VEVO, Sony Music Enternatinment, etc.).

    What I did is I intercepted prepareConnection method from the URLLoader class that is used by JavaFX's WebEngine and inserted my instruction at the top of the method body:

    c.setRequestProperty("Referer", "https://www.discogs.com");

    Code from the JDK source files

    (Again, please follow the tutorial for all the instructions)

    (Note: Even though the tutorial above is explains very well the concepts, it doesn't really touch much on the role and structure of a MANIFEST.MF file, so please check this link for more info about this aspect)

    These are my two classes:

    MyJavaAgent.java

    package com.busytrack.discographymanager.headerfixagent;
import java.lang.instrument.Instrumentation;
public class MyJavaAgent {
public static void premain(String agentArgument, Instrumentation instrumentation) {
    ClassTransformer transformer = new ClassTransformer();
    instrumentation.addTransformer(transformer);
    }
}

    ClassTransformer.java

    package com.busytrack.discographymanager.headerfixagent;
import java.io.ByteArrayInputStream;
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.security.ProtectionDomain;
import javassist.ClassPool;
import javassist.CtClass;
import javassist.CtMethod;

public class ClassTransformer implements ClassFileTransformer {
    public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {
        byte[] byteCode = classfileBuffer;
        if (className.equals("com/sun/webkit/network/URLLoader")) {
            try {
                ClassPool classPool = new ClassPool(true);
                CtClass ctClass = classPool.makeClass(new ByteArrayInputStream(classfileBuffer));
                CtMethod method = ctClass.getDeclaredMethod("prepareConnection");
                String src = "$1.setRequestProperty(\"Referer\", \"https://www.discogs.com\");"; // Confused about there being "$1" instead of "c"? Please read below
                method.insertBefore(src);
                byteCode = ctClass.toBytecode();
                ctClass.detach();
            }  catch (Exception e) {
                e.printStackTrace();
            }
        }   
        return byteCode;
    }
}

    This is why I used "$1" to access the method parameter, instead of "c":

    The statement and the block can refer to fields and methods. They can also refer to the parameters to the method that they are inserted into if that method was compiled with the -g option (to include a local variable attribute in the class file). Otherwise, they must access the method parameters through the special variables $0, $1, $2, ... described below. Accessing local variables declared in the method is not allowed although declaring a new local variable in the block is allowed.

    The entire javassist tutorial can be found here.

    After packing the two classes and the MANIFEST.MF file in a separate JAR, import it in your IDE (I used Eclipse) and add the following VM argument:

    -javaagent:./(your-jar-name).jar

    In Eclipse, you can add VM arguments like this:

    right click on your project -> Run As -> Run Configurations... -> open the Arguments tab -> insert your VM argument -> Apply

    I hope this helps someone out there. I know I spent a few days on this issue. I don't know if this is the best approach but it does the job for me. Still, it makes me wonder why isn't there a straightforward way of setting Request Headers for JavaFX's WebEngine...

    Later edit:

    I found a much cleaner and easier approach for loading Java Agents, dynamically, without the need to create a separate JAR, manifest file, importing them, passing the -javaagent VM parameter at startup, etc.

    I used the ea-agent-loader (JAR download link).

    Import the JAR in your IDE and change the MyJavaAgent class (the one that had the premain method) to this:

    package com.busytrack.discographymanager.headerfixagent;
import java.lang.instrument.Instrumentation;
public class MyJavaAgent {
    public static void agentmain(String agentArgument, Instrumentation instrumentation) {
        ClassTransformer transformer = new ClassTransformer();
        instrumentation.addTransformer(transformer);
    }
}

    My main method from the MainClass looks like this:

    public static void main(String[] args) {
    AgentLoader.loadAgentClass(MyJavaAgent.class.getName(), null); // Load the MyJavaAgent class
    launch(args); // Start the JavaFX application
}

    I wanted to be able to load the Agent dynamically because, using the static method required me to create separate launchers for all platforms and pass the -javaagent parameter on startup. Now, I can export a runnable JAR from eclipse like I usually do and the agent will load automatically (no VM parameters required). Thanks, BioWare for this tool! :D