Re: Using Netsurf cookies with wget

On 21 Jan 2022 as I do recall,
Jeremy Nicoll - ml netsurf wrote:

> On 2022-01-21 00:55, Harriet Bazley wrote:
>
> > I also tried using --post-data 'user-Login=USER&user_password=PASSWORD'
> > with no result,
>
> That /may/ be because you weren't careful enough coding that. According
> to the form code, the login variable isn't called "user-Login" but
> instead (2 differences) "user_login".

No - unfortunately it looks as if that was just sloppy transcription.
The actual data in my test script is "user_login". :-(


>
> Also, the form when used on a webpage, sets variable "user_remember_me"
> and (I'm not completely sure) maybe also the submit button part sets
> something - I don't know why it defines a name and a value - the latter
> is the text on the button but what's "name" for?

That's just a 'Remember Me' (and don't ask for log-in again but redirect
to the stats page) button.
I'm afraid I don't know enough about HTML forms to understand exactly
what the Submit button is doing, but the page only prompts for two user
inputs in this form.

(There are other forms on that page, e.g. a search box to search the
site, but I'm assuming that provided I supply the correct id/input pairs
the server will act on the data from the correct form - the trouble is
that I understand very little of what I am actually doing here and am
basically poking around at random.)

>
>
> I haven't used Netsurf for ages, but in Firefox one can see in its logs
> what URLs are built and sent back to a server - indeed in developer
> tools one can see the "curl" equivalent command to each communication
> with a server. Really the simplest way to recreate something is to
> look at what the browser actually does.

Yes, that was the recommendation on the Web page that suggested using
the --post-data method: simply 'turn on development tools'.
Unfortunately so far as I know Netsurf doesn't provide access to sniff
around at that level.

You can't read the log in !Scrap while the browser is actually running,
which makes looking at that a little tricky, and it gives a *lot* of
data that is nothing to do with curl fetches. But all I'm seeing is
HTTP status codes, e.g.

(16.330000) [INFO netsurf] content/fetchers/curl.c:1200 fetch_curl_process_headers: HTTP status code 200

And then way, way down the page

(17.460000) [INFO netsurf] content/handlers/html/html.c:126 fire_generic_dom_event: Dispatching 'click' against 0x55ef7118
(18.690000) [INFO netsurf] content/handlers/html/html.c:203 fire_dom_keyboard_event: Dispatching 'keydown' against 0x55d2d690
(18.960000) [INFO netsurf] content/handlers/html/html.c:203 fire_dom_keyboard_event: Dispatching 'keydown' against 0x55d2d690

etc., which is me typing passwords.

Lots of things being removed on the submission of the form:

(20.030000) [INFO netsurf] content/handlers/html/html.c:1192 html_destroy: content 0x55a955c0
(20.030000) [INFO netsurf] content/handlers/html/form.c:1460 form_free_control: Control:0x55e20698 name:0x55e1ca48 value:0x55b1e9f0 initial:0x0
(20.030000) [INFO netsurf] content/handlers/html/form.c:1460 form_free_control: Control:0x55e6f2e0 name:0x55e6f2c8 value:0x55e6f348 initial:0x55e6ea88
(20.030000) [INFO netsurf] content/handlers/html/form.c:1460 form_free_control: Control:0x55e42780 name:0x0 value:0x55e42000 initial:0x0
(20.030000) [INFO netsurf] content/content.c:695 content_remove_user: content file:///NetSurf:/Resources/CSS (0x55b62098), user 0x16c498 0x55a92c28

Then the browser history getting updated with the new page and a
FETCH_REDIRECT from the login page to the user home page. No record of
what data was sent to the server, that I can see.

But as I said, I understand very little of what is going on.

--
Harriet Bazley == Loyaulte me lie ==

Death is nature's way of telling you to slow down.
_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org

Re: Using Netsurf cookies with wget

On 2022-01-21 00:55, Harriet Bazley wrote:

> I also tried using --post-data 'user-Login=USER&user_password=PASSWORD'
> with no result,

That /may/ be because you weren't careful enough coding that. According
to the form code, the login variable isn't called "user-Login" but
instead (2 differences) "user_login".

Also, the form when used on a webpage, sets variable "user_remember_me"
and (I'm not completely sure) maybe also the submit button part sets
something - I don't know why it defines a name and a value - the latter
is the text on the button but what's "name" for?


I haven't used Netsurf for ages, but in Firefox one can see in its logs
what URLs are built and sent back to a server - indeed in developer
tools
one can see the "curl" equivalent command to each communication with a
server. Really the simplest way to recreate something is to look at
what
the browser actually does.


--
Jeremy Nicoll - my opinions are my own
_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org

Re: Using Netsurf cookies with wget

On 20 Jan 2022 as I do recall,
Harriet Bazley wrote:

> On 20 Jan 2022 as I do recall,
> simon_smith@zen.co.uk wrote:
>
> [snip]
>
> >
> > The cookie part is probably a red herring. The conventional approach
> > would be to use the wget tools to fetch the login page and send
> > username and password using the features wget has built-in. Hey
> > presto, now you're logged in, via wget, and you should be able to get
> > the rest of the stuff you're after.
>
> I've tried "wget --ask-password URL", which prompts me for the password
> but then redirects to fetch the login page as if I were not logged in,
> and "wget --user=USER --password=PASSWORD URL", which also redirects to
> the login page instead of retrieving the one I asked for.
>
> I've tried fetching the login page directly using --user and --pass, but
> it just fetches the 'please log in' prompt instead of the 'you are
> already signed in' prompt.
>

I also tried using --post-data 'user-Login=USER&user_password=PASSWORD'
with no result, where the form in the log-in page is as follows:


<dt><label for="user_login">User name or email:</label></dt>
<dd><input type="text" name="user[login]" id="user_login"/></dd>
<dt><label for="user_password">Password:</label></dt>
<dd><input type="password" name="user[password]" id="user_password"/></dd>
<dt><label for="user_remember_me">Remember me</label></dt>
<dd><input name="user[remember_me]" type="hidden" value="0"/><input type="checkbox" value="1" name="user[remember_me]" id="user_remember_me"/></dd>
<dt class="landmark">Submit</dt>
<dd class="submit actions">
<input type="submit" name="commit" value="Log in" class="submit"/>
</dd>


--
Harriet Bazley == Loyaulte me lie ==

The fact that you're paranoid.... doesn't mean they're NOT out to get you.
_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org

Re: Using Netsurf cookies with wget

On 20 Jan 2022 as I do recall,
simon_smith@zen.co.uk wrote:

[snip]

>
> The cookie part is probably a red herring. The conventional approach
> would be to use the wget tools to fetch the login page and send
> username and password using the features wget has built-in. Hey
> presto, now you're logged in, via wget, and you should be able to get
> the rest of the stuff you're after.

I've tried "wget --ask-password URL", which prompts me for the password
but then redirects to fetch the login page as if I were not logged in,
and "wget --user=USER --password=PASSWORD URL", which also redirects to
the login page instead of retrieving the one I asked for.

I've tried fetching the login page directly using --user and --pass, but
it just fetches the 'please log in' prompt instead of the 'you are
already signed in' prompt.

I've tried using the --save-cookies option to save any cookies generated
by 'logging in', and it just saves a blank "generated by Wget" file with
no data in it.

I've tried --load-cookies=SCSI::SSD.$.!BOOT.Choices.WWW.NetSurf.Cookies
on the offchance, but that didn't work either....


--
Harriet Bazley == Loyaulte me lie ==

My opinions may have changed, but not the fact that I am right.
_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org

Re: Using Netsurf cookies with wget

On 2022-01-20 06:46 PM, "Harriet Bazley" <lists@bazleyfamily.co.uk> wrote:
> On 20 Jan 2022 as I do recall,
> cj wrote:
>
> > In article <88ea7bad59.harriet@bazleyfamily.co.uk>,
> > Harriet Bazley <lists@bazleyfamily.co.uk> wrote:
> > > How can I use Netsurf's cookie file with wget to retrieve a web page
> > > that is only accessible to logged-in users
> >
> > wget --help shows there are commands to send user names and passwords
> > when downloading, ftping etc. Have you tried that?
> >
> I'm not sure how that would work, unless I fetched the log-in page
> first? The stats page doesn't request a password - so far as I can
> tell it just doesn't let you fetch it if the relevant browser cookie
> isn't present.

The cookie part is probably a red herring. The conventional approach would
be to use the wget tools to fetch the login page and send username and
password using the features wget has built-in. Hey presto, now you're logged
in, via wget, and you should be able to get the rest of the stuff you're after.

If this web site really was using a cookie for 'authentication', then in theory you should be able to send the cookie to someone else, they could put it on
their computer, and then access the 'protected' area without logging in at all.
That makes no sense.

Try the tools wget already provides - unless it genuinely is a highly eccentric
website, wget should be sufficient.

--
Simon Smith [via webmail]




_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org

Re: Using Netsurf cookies with wget

On 20 Jan 2022 as I do recall,
cj wrote:

> In article <88ea7bad59.harriet@bazleyfamily.co.uk>,
> Harriet Bazley <lists@bazleyfamily.co.uk> wrote:
> > How can I use Netsurf's cookie file with wget to retrieve a web page
> > that is only accessible to logged-in users
>
> wget --help shows there are commands to send user names and passwords
> when downloading, ftping etc. Have you tried that?
>
I'm not sure how that would work, unless I fetched the log-in page
first? The stats page doesn't request a password - so far as I can
tell it just doesn't let you fetch it if the relevant browser cookie
isn't present.

--
Harriet Bazley == Loyaulte me lie ==

Cole's Law: Thinly sliced cabbage.
_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org

Re: Using Netsurf cookies with wget

In article <88ea7bad59.harriet@bazleyfamily.co.uk>,
Harriet Bazley <lists@bazleyfamily.co.uk> wrote:
> How can I use Netsurf's cookie file with wget to retrieve a web page
> that is only accessible to logged-in users

wget --help shows there are commands to send user names and passwords
when downloading, ftping etc. Have you tried that?

--
Chris Johnson
Edinburgh
_______________________________________________
netsurf-users mailing list -- netsurf-users@netsurf-browser.org
To unsubscribe send an email to netsurf-users-leave@netsurf-browser.org