Search code examples
javahtmljsoup

How to get video src with jsoup (twitch)?

I am getting null pointer, is possible to get this link?

Element element = document.select("div.tw-absolute.tw-bottom-0.tw-left-0.tw-overflow-hidden.tw-right-0.tw-top-0.video-player__container").first();
System.out.println(element.absUrl("src"));

Tried this too

nullpointer as well

Element video = document.select("video").first();
String absSrc = video.absUrl("src");
System.out.println(absSrc);

html part

<div class = "tw-absolute tw-bottom-0 tw-left-0 tw-overflow-hidden tw-right-0 tw-top-0 video-player__container" data-test-selector="video-player__video-container">
               <video playsinline="" webkit-playsinline="" src="https://clips-media-assets2.twitch.tv/40487770748-offset-9048.mp4?token=%7B%22authorization%22:%7B%22forbidden%22:false,%22reason%22:%22%22%7D,%22chansub%22:%7B%22restricted_bitrates%22:%5B%5D%7D,%22device_id%22:%226518a1542e035018%22,%22expires%22:1609419047,%22https_required%22:true,%22privileged%22:false,%22user_id%22:500437676,%22version%22:2,%22vod_id%22:850278065%7D&amp;sig=5e17731db577b99e535c4aad3eacc70c0cc34521"></video>

link: https://www.twitch.tv/scream/clip/BrightOilyAppleMcaT

Solution

  • Looks like this one will require again, a lot of work to unpick.

    Here's what I can tell you just from a quick look:

    when you make the initial request, it does not contain the result you're looking for in the HTML. Therefore it must be coming from a subsequent HTTP request that is fired off once the page is loaded... i.e. there's javascript communicating with back end servers to get JSON payloads. In one of those payloads you'll find ".mp4".

    If you use Chrome developer tools, you can flick over to the "Network" tab, click on each request following the first one, and check the "Preview" tab. You will find some requests contain JSON responses, others are just .css, .png, etc. ignore these. In the JSON responses, check the results for the occurrence of some generic value you're interested in like ".mp4". Once you've found it:

    enter image description here

    .. you then need to try to recreate the headers, the request body (as its not empty), the type of HTTP request (POST), and pass any relevant cookies (in the headers).

    You're going to have to make anywhere between 1 and 5 HTTP requests to get what you need to get this JSON payload. Once you have it you can then parse it back.

    This is another one of those jobs that's so big I'm not going to begin to try to do it for you.

    If it were me doing the job, I'd check the Twitch API docs https://dev.twitch.tv/docs/api/ to see if there's a better/easier way that's just 1-2 requests.