Is there a way to download the text showing on a web page? - users - Fedora Mailing-Lists

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

1989

1988

1987

1986

1985

1984

1983

1982

1981

1980

Is there a way to download the text showing on a web page?

Running vdagent with QEMU/KVM +...

how to get stoken?

ToddAndMargo

Thursday, 18 April 2019 Thu, 18 Apr '19

12:09 a.m.

Hi All, Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.) Many thanks, -T

Reply

Show replies by thread

Samuel Sieb

Thursday, 18 April Thu, 18 Apr

2:11 a.m.

On 4/17/19 10:09 PM, ToddAndMargo via users wrote:

Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)

I don't think any of the command line browsers will run javascript.

Reply

ToddAndMargo

2:17 a.m.

On 4/18/19 12:11 AM, Samuel Sieb wrote:

On 4/17/19 10:09 PM, ToddAndMargo via users wrote: > Curl can download the source of a web page. Is there a way to > download what shows on a web page after all the javascritps > have done their thing? (From the command line, not my hand > and my mouse.) I don't think any of the command line browsers will run javascript.

Rats!

Reply

Jordan Metzmeier

4:53 a.m.

On Thu, Apr 18, 2019 at 12:09 AM ToddAndMargo via users <users(a)lists.fedoraproject.org> wrote:

Hi All, Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)

You can probably use chrome in headless mode for this. There are some examples here: https://developers.google.com/web/updates/2017/04/headless-chrome

Reply

Patrick O'Callaghan

5:31 a.m.

On Wed, 2019-04-17 at 22:09 -0700, ToddAndMargo via users wrote:

Hi All, Curl can download the source of a web page. Is there a way to download what shows on a web page after all the javascritps have done their thing? (From the command line, not my hand and my mouse.)

Maybe try one of the 'web scraping' libraries e.g.: https://www.scraperapi.com/blog/the-10-best-web-scraping-tools They usually seem to return a complete DOM (Document Object Model) but also provide functions to parse the contents. poc

Reply

ToddAndMargo

4:30 p.m.

On 4/18/19 3:31 AM, Patrick O'Callaghan wrote:

On Wed, 2019-04-17 at 22:09 -0700, ToddAndMargo via users wrote: > Hi All, > > Curl can download the source of a web page. Is there a way to > download what shows on a web page after all the javascritps > have done their thing? (From the command line, not my hand > and my mouse.) Maybe try one of the 'web scraping' libraries e.g.: https://www.scraperapi.com/blog/the-10-best-web-scraping-tools They usually seem to return a complete DOM (Document Object Model) but also provide functions to parse the contents. poc

Thank you!

Reply

ToddAndMargo

4:30 p.m.

On 4/18/19 2:53 AM, Jordan Metzmeier wrote:

On Thu, Apr 18, 2019 at 12:09 AM ToddAndMargo via users <users(a)lists.fedoraproject.org> wrote: > > Hi All, > > Curl can download the source of a web page. Is there a way to > download what shows on a web page after all the javascritps > have done their thing? (From the command line, not my hand > and my mouse.) > You can probably use chrome in headless mode for this. There are some examples here: https://developers.google.com/web/updates/2017/04/headless-chrome

Thank you!

Reply

Seth Kenlon

5:34 p.m.

The elinks browser does at least some javascript. Im not sure, off the top of my head, whether it has a -dump feature the way lynx does, but you could mess around with it. On Fri, 19 Apr 2019, 09:31 ToddAndMargo via users, < users(a)lists.fedoraproject.org> wrote:

On 4/18/19 2:53 AM, Jordan Metzmeier wrote: > On Thu, Apr 18, 2019 at 12:09 AM ToddAndMargo via users > <users(a)lists.fedoraproject.org> wrote: >> >> Hi All, >> >> Curl can download the source of a web page. Is there a way to >> download what shows on a web page after all the javascritps >> have done their thing? (From the command line, not my hand >> and my mouse.) >> > > You can probably use chrome in headless mode for this. There are some > examples here: https://developers.google.com/web/updates/2017/04/headless-chrome Thank you! _______________________________________________ users mailing list -- users(a)lists.fedoraproject.org To unsubscribe send an email to users-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Reply

1837

days inactive

1837

days old

users@lists.fedoraproject.org

Manage subscription

7 comments

5 participants

tags (0)

participants (5)

Jordan Metzmeier
Patrick O'Callaghan
Samuel Sieb
Seth Kenlon
ToddAndMargo