Parsing HTML page containing JS in Java

Question

I am trying to parse a web page, which contains some JS. Till now I am using Jsoup to parse html in Java, which is working as expected. But I am unable to parse the JavaScript. Below is the snippet of the HTML page-

<script type="text/javascript"> 
var element = document.createElement("input"); 
element.setAttribute("type", "hidden");
element.setAttribute("value", "");
element.setAttribute("name", "AzPwXPs");
element.setAttribute("id", "AzPwXPs");
var foo = document.getElementById("dnipb"); 
foo.appendChild(element);
var element1 = document.createElement("input"); 
element1.setAttribute("type", "hidden");
element1.setAttribute("value", "6D6AB8AECC9B28235F1DE39D879537E1");
element1.setAttribute("name", "ZLZWNK");
element1.setAttribute("id", "ZLZWNK");
foo.appendChild(element1);
</script>

I want to read both the values with their name/id. So that after parsing I can get following results-

AzPwXPs=
ZLZWNK=6D6AB8AECC9B28235F1DE39D879537E1

How to parse in this situation?

@nhahtdh: Ya, I know that. That is why I am stuck in between... :( But there must some other way around — ravi
– ravi, Commented May 1, 2013 at 10:49
Run it through a JS parser? Or get a JS engine? (I actually also have the same problem on a side project, but I never got my hand around it...) — nhahtdh
– nhahtdh, Commented May 1, 2013 at 10:52

Mihkel Selgal · Accepted Answer · 2017-03-02 08:25:37Z

6

I have stumbled upon this question few times when searching for the solution to parse pages with JavaScript but the solution provided is not perfect. I have found pure Java solution to the problem by using JBrowserDriver and JSoup to parse JavaScript manipulated page.

Simple example:

    // JBrowserDriver part
    JBrowserDriver driver = new JBrowserDriver(Settings
            .builder().
            timezone(Timezone.EUROPE_ATHENS).build());
    driver.get(FETCH_URL);
    String loadedPage = driver.getPageSource();

    // JSoup parsing part
    Document document = Jsoup.parse(loadedPage);
    Elements elements = document.select("#nav-console span.data");

    log.info("Found element count: {}", elements.size());

    driver.quit();

answered Mar 2, 2017 at 8:25

Mihkel Selgal

5086 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

gramm Over a year ago

Works perfectly

Hugo Pedrosa · Accepted Answer · 2018-06-28 09:16:30Z

2

I already had the same situation to find url's in css files.

Put the javascript in a string and a apply Regular expressions

Pattern p = Pattern.compile("url\\(\\s*(['" + '"' + "]?+)(.*?)\\1\\s*\\)"); //expression
Matcher m = p.matcher(content);
while (m.find()) {
String urlFound = m.group(); 
}

Regards, Hugo Pedrosa

edited Jun 28, 2018 at 9:16

answered May 2, 2013 at 16:07

Hugo Pedrosa

3472 silver badges12 bronze badges

1 Comment

ravi Over a year ago

Although I got the logic but how this can be modified to serve my purpose?

Community · Accepted Answer · 2017-05-23 12:00:35Z

1

Selenium's Webdriver is fantastic: http://docs.seleniumhq.org/docs/03_webdriver.jsp

See this answer for an example of what you are trying to do: Using Selenium Web Driver to retrieve value of a HTML input

edited May 23, 2017 at 12:00

CommunityBot

11 silver badge

answered May 2, 2013 at 16:11

Tom Carchrae

6,4962 gold badges39 silver badges36 bronze badges

4 Comments

ravi Over a year ago

What you mean by jQuery is being used? Can you explain me a little bit more please?

Tom Carchrae Over a year ago

the javascript return $('#AzPwXPs')[0] uses a jquery selector ( $('#AzPwXPs')) to find the element.

ravi Over a year ago

So, finally in my java parser project there will be Selenium's Webdriver with jquery. Is it?

Tom Carchrae Over a year ago

no, you don't need to use jquery - i'm just going to link to an answer that shows you a better example

Java Israel · Accepted Answer · 2013-05-02 15:54:00Z

1

You can try using query library. Its much more easier with it.

answered May 2, 2013 at 15:54

Java Israel

111 bronze badge

1 Comment

ravi Over a year ago

Can you please suggest any one of them? Is it available in Java?

Community · Accepted Answer · 2017-05-23 12:00:35Z

1

Once you've got the text content of the <script> element from JSoup, you can parse the JS using the Caja JS parser and then walk the parse tree to find what you're looking for.

edited May 23, 2017 at 12:00

CommunityBot

11 silver badge

answered May 2, 2013 at 16:15

Mike Samuel

121k30 gold badges230 silver badges255 bronze badges

2 Comments

ravi Over a year ago

How to get <script> element from JSoup?

Mike Samuel Over a year ago

@Ravi, If you're doing something like Document doc = Jsoup.parse(...), then doc.getElementsByTag("script").first() should get you the first script in the page.

Collectives™ on Stack Overflow

Parsing HTML page containing JS in Java

5 Answers 5

1 Comment

1 Comment

4 Comments

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

1 Comment

1 Comment

4 Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related