PHP - Scraping javascript websites

Question

I am trying to check the content of websites using php curl for any blacklisted keywords. But with curl I will not get the content generated by JS scripts. I have to scan thousands of websites and efficiency is the main point here. I need to get js content. So far I came across Phantomjs-php. Are there any other js scripts which will take less resources that will work with php. I just need to get the html content. Any insights on this is much appreciated as I am new to getting JS generated content.

Thank you Lynn

I'm not sure about PHP libraries, but for python, I've been using js2py to do something similar. It's quite good, and much faster than selenium. — Loïc
– Loïc, Commented Nov 1, 2019 at 15:37
Thanks Loïc . But does it do something like how an headless browser will do? Execute js scripts and give me the content of the website? Python is new to me and I need to explore over there. — Lynn Rey
– Lynn Rey, Commented Nov 1, 2019 at 16:31
Kind of, except it's much faster than a headless browser. But you don't really have a browser view, you can just run any javascript script, so look into the variables, and things like that... github.com/PiotrDabkowski/Js2Py — Loïc
– Loïc, Commented Nov 1, 2019 at 16:59

Lucas Meine · Accepted Answer · 2019-11-01 15:38:48Z

1

I'm pretty sure that Codeception would do the trick for you.

You can configure it to work with a headless browser, just like phantom.js and puppeteer, and see your js generated content. A sample acceptance testing, which is what you want to do, would look like this:

$I->amOnPage('/login');
$I->fillField('username', 'davert');
$I->fillField('password', 'qwerty');
$I->click('LOGIN');
$I->see('Welcome, Davert!');

taken from: https://codeception.com/docs/03-AcceptanceTests

answered Nov 1, 2019 at 15:38

Lucas Meine

1,6175 gold badges25 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Lynn Rey Over a year ago

Thanks Lucas . will check it out.

Collectives™ on Stack Overflow

PHP - Scraping javascript websites

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related