Hi All,
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
Many thanks, -T
On 4/17/19 10:09 PM, ToddAndMargo via users wrote:
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
I don't think any of the command line browsers will run javascript.
On 4/18/19 12:11 AM, Samuel Sieb wrote:
On 4/17/19 10:09 PM, ToddAndMargo via users wrote:
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
I don't think any of the command line browsers will run javascript.
Rats!
On Thu, Apr 18, 2019 at 12:09 AM ToddAndMargo via users users@lists.fedoraproject.org wrote:
Hi All,
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
You can probably use chrome in headless mode for this. There are some examples here: https://developers.google.com/web/updates/2017/04/headless-chrome
On 4/18/19 2:53 AM, Jordan Metzmeier wrote:
On Thu, Apr 18, 2019 at 12:09 AM ToddAndMargo via users users@lists.fedoraproject.org wrote:
Hi All,
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
You can probably use chrome in headless mode for this. There are some examples here: https://developers.google.com/web/updates/2017/04/headless-chrome
Thank you!
The elinks browser does at least some javascript. Im not sure, off the top of my head, whether it has a -dump feature the way lynx does, but you could mess around with it.
On Fri, 19 Apr 2019, 09:31 ToddAndMargo via users, < users@lists.fedoraproject.org> wrote:
On 4/18/19 2:53 AM, Jordan Metzmeier wrote:
On Thu, Apr 18, 2019 at 12:09 AM ToddAndMargo via users users@lists.fedoraproject.org wrote:
Hi All,
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
You can probably use chrome in headless mode for this. There are some examples here:
https://developers.google.com/web/updates/2017/04/headless-chrome
Thank you! _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Wed, 2019-04-17 at 22:09 -0700, ToddAndMargo via users wrote:
Hi All,
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
Maybe try one of the 'web scraping' libraries e.g.:
https://www.scraperapi.com/blog/the-10-best-web-scraping-tools
They usually seem to return a complete DOM (Document Object Model) but also provide functions to parse the contents.
poc
On 4/18/19 3:31 AM, Patrick O'Callaghan wrote:
On Wed, 2019-04-17 at 22:09 -0700, ToddAndMargo via users wrote:
Hi All,
Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)
Maybe try one of the 'web scraping' libraries e.g.:
https://www.scraperapi.com/blog/the-10-best-web-scraping-tools
They usually seem to return a complete DOM (Document Object Model) but also provide functions to parse the contents.
poc
Thank you!
-
Jordan Metzmeier -
Patrick O'Callaghan -
Samuel Sieb -
Seth Kenlon -
ToddAndMargo