Jump to content











Photo
- - - - -

Alternate ways to download files via batch.?


  • Please log in to reply
12 replies to this topic

#1 Zharif

Zharif

    Frequent Member

  • .script developer
  • 172 posts
  • Location:Germany
  •  
    Germany

Posted 11 September 2016 - 12:16 AM

While trying to create some new plugins for Win10PESE I stumbled about some downloading issues.
Using provided tools such as wget or webget I wasn't able to download files from the EF Software Site via command line.
Site: http://www.efsoftware.com/dw/e.htm.
x64 portable version of EFCommander: http://www.efsoftwar...m/dw/wz6.cgi?cw

I see that this site uses java script and it seems that wget is not able to follow them correctly (honestly, I'm guessing only).
Example code line for wget I used before:
SET "_DestDir=%~dp0" 2>NUL
IF "%DestDir:~-1%"=="\" SET "DestDir=%DestDIR:~0,-1%" 2>NUL
SET "_URL=http://www.efsoftwar.../dw/wz6.cgi?cw"
SET "LANG=EN"
SET "LANGUAGE=EN"
SET "TSTAMP=-N"
SET "VERBOSENESS=-v"
SET "TRIES=--tries=3"
SET "WAITRETRY=--waitretry=2"
SET "CACHING=--cache=off"
%_WGET% %VERBOSENESS% %TSTAMP% %CACHING% %TRIES% %WAITRETRY% "%_URL%" -P"%_DESTDIR%"

In the meantime I tried to "bypass" this issue by calling powershell and bitsadmin.
The code below is what I have so far. It might be of help somehow but some code lines are really clumsy in my eyes.

Could anybody guide me here?
Am I guessing right that wget isn't able to resolve links from sites written with java script?
If No, is this a syntax problem I didn't solve here?
If Yes, are there any other tools (freeware) that are able to download files from this/such site(s)?

Regarding bitsadmin.exe - is it legal to provide this file inside a plugin script encoded via Winbuilder?
Maybe my coding attempts below are reliable alternatives (in general) to external tools that must be downloaded for this specific task?
If so, maybe someone could help to improve code?

 

EDIT: FileExists section removed - it's completely unneccessary.

Note: Please do not copy the following code to notepad.exe. Use a real text editor instead.

 

@echo off
SETLOCAL ENABLEEXTENSIONS
SET "_LDir=%~dp0"
IF "%_LDIR:~-1%"=="\" SET "_LDir=%_LDIR:~0,-1%" 2>NUL

SET _LocalFile=%_LDIR%\EFCommander.zip
SET _URL=http://www.efsoftwar...m/dw/wz6.cgi?cw

DIR /B "%_LOCALFILE%" >NUL 2>&1 && ECHO File to download already exists... skipping download.&&GOTO:_End

:: check for existance of powershell.exe and set _PS variable

powershell "EXIT ECHO Works"  2>NUL && SET "_PS=1"
:: get PSVersion Major and Minor
CALL:_PSVers _PSVersMaj _PSVersMin

ECHO Powershell engine %_PSVersMaj%.%_PSVersMin% or higher detected.
ECHO Press any key to to download the desired file...& PAUSE>NUL
:: try to download file via Powershell
CALL:_PSDownload "%_LOCALFILE%" "%_URL%"
DIR /B "%_LOCALFILE%" >NUL 2>&1 && ECHO Download successful.&&GOTO:_End

:: check for existance of bitsadmin.exe

:: on success - evaluate if bitsadmin.exe supports the /TRANSFER switch

bitsadmin >NUL 2>&1 && SET "_BITS=1" && bitsadmin.exe /?|FIND /I "/transfer " >NUL 2>&1 && SET _BITS=2
:: try to download file via bitsadmin.exe
CALL:_BitsDownload "%_LOCALFILE%" "%_URL%" 15
DIR /B "%_LOCALFILE%" >NUL 2>&1 && ECHO Download successful.||ECHO Download failed

:_End
ECHO This is the END
PAUSE>NUL&EXIT
:: _____________________________________________________________________

:: This code is tested under XP SP3, Win7,Win8.1 and Win10.
:: The PS exitcode command helps to capture integer values via returned
:: errorlevel. PSVersion.Major must be 1 or higher - use errorlevel 1.
:: PSVersion.Minor must be 0 or higher, so use errorlevel 0.
:: Ref:https://blog.jourdan...with-powershell
:: _____________________________________________________________________

:_PSVers  VersionMajorOUT  VersionMinorOUT
IF NOT DEFINED _PS GOTO:EOF
powershell "Exit $PSVersionTable.PSVersion.Major"
IF ERRORLEVEL 1 SET /A %~1=%ERRORLEVEL%
powershell "Exit $PSVersionTable.PSVersion.Minor"
IF ERRORLEVEL 0 SET /A %~2=%ERRORLEVEL%
GOTO:EOF

:_PSDownload TargetFileIN  UrlIN
IF NOT DEFINED _PS GOTO:EOF
SET /P "=Downloading via PS BitsTransfer..." <NUL
powershell "Import-Module BitsTransfer;EXIT Start-BitsTransfer -Source '%~2' -Destination '%~1'" >NUL

:: NET2.0
IF ERRORLEVEL 1 (Echo Errors occured.
    SET ERRORLEVEL=
    SET /P "=Downloading via PS Net.WebClient..." <NUL
    powershell "EXIT (New-Object System.Net.WebClient).DownloadFile('%~2', '%~1')" >NUL
)

:: PS 3.0 or higher
IF ERRORLEVEL 1 (Echo Errors occured.
    SET ERRORLEVEL=
    IF %_PSVersMaj% GTR 3 (
        SET /P "=Downloading via PS Invoke-WebRequest cmdlet..." <NUL
        powershell "Invoke-WebRequest '%~2abcd' -OutFile '%~1'" >NUL
    )
)
IF ERRORLEVEL 1 Echo Errors occured.
GOTO:EOF
:: __________________________________________________________________________

:: If bitsadmin.exe supports the /transfer switch then everything goes well.
:: Otherwise fall back to commandline syntax that is supported for very early
:: versions [e.g. v1.0.5.1.2600.0 [XPPro SP3] in support\tools\support.cab].
:: In regard to the version mentioned above, I couldn't figure out how to get
:: feedback if a download is completed before using the /COMPLETE switch.
:: Therefore a very clumpsy wait-like command was placed here. Anybody know
:: how to improve/overcome this?
:: __________________________________________________________________________

:_BitsDownload  TargetFileIN  UrlIN TimeoutIN
SETLOCAL
IF NOT DEFINED _BITS GOTO:EOF
SET "_B=bitsadmin.exe"
SET "_Job=MyJob"
SET "_T=%~3"

%_B% /RESET
IF %_BITS%.==2. (
    %_B% /transfer %JOB% /download /priority normal "%~2" "%~1"
) ELSE (
    %_B% /CREATE %_JOB%
    %_B% /ADDFILE %_JOB% "%~2" "%~1"
    %_B% /RESUME %_JOB%
    REM %_B% /WRAP /INFO %_JOB% /VERBOSE
    REM %_B% /GETCOMPLETIONTIME %_JOB%
    ECHO.&ECHO Waiting %_T% seconds to execute /COMPLETE switch...
    ping -n %_T% localhost >NUL
    %_B% /COMPLETE %_JOB%
)
ENDLOCAL
GOTO:EOF



#2 Guest_AnonVendetta_*

Guest_AnonVendetta_*
  • Guests

Posted 11 September 2016 - 03:44 AM

You say you've tried wget. But have you tried curl?



#3 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 11 September 2016 - 09:27 AM

I would say that w32.cgi is a cgi script. (i.e. something like php executed on servers side, not javascript, it is easy to check go there with a browser with javascript disabled and the file will download the same).

CURL should be able to manage cgi, but then also wget should, maybe the script needs a referrer or a given user agent.

Or maybe you need the -O somefile.extension parameter?

http://superuser.com...when-redirected

It has to be seen if a windows Wget port has that option, though. :unsure:

 

:duff:

Wonko



#4 Zharif

Zharif

    Frequent Member

  • .script developer
  • 172 posts
  • Location:Germany
  •  
    Germany

Posted 11 September 2016 - 11:09 AM

Thanks for info and link Wonko.
Regarding the provided switches of this site, usage of curl with any combination of these results at least in a text file indicating that the download page moved to another url.

Command: curl  "http://www.efsoftwar.../dw/wz6.cgi?cw"-o "EFCommander.txt"
Resulting file:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://www.efsoftwar...">here</a>.</p>
<hr>
<address>Apache Server at www.efsoftware.com Port 80</address>
</body></html>

 

in combination with sed it's relatively easy to get the changed url only and to save it as variable.
Sample Command: sed -n "/.*href=.*/p; s/.*\d34\(.*\)\d34.*/\1/p" efcommander.txt

 

Then use curl with the new url:
Command: curl -o %CD%\EFCommander.zip http://www.efsoftwar...ommander_64.zip

 

Okay, this works. But its coding nightmare I don't want.

BTW, wget is still not able to resolve the new retrieved url too.

 

I'm very sure that there must be a possibility around without need of downloading another command line tool (curl here).? Any help is greatly appreciated. Till now I suppose that using powershell or bitsadmin is a better choice.

 

BTW, I altered the code of my first post and removed the FileExists function. It appears to be completely unneccessary.



#5 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 11 September 2016 - 11:54 AM

What happens with CURL -L or --location ?

(i.e. is that html page sent correctly as a 302 ?) :unsure:

https://curl.haxx.se/docs/manpage.html

 

-L, --location

(HTTP/HTTPS) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place. If used together with -i, --include or -I, --head, headers from all requested pages will be shown. When authentication is used, curl only sends its credentials to the initial host. If a redirect takes curl to a different host, it won't be able to intercept the user+password. See also --location-trusted on how to change this. You can limit the amount of redirects to follow by using the --max-redirs option.

When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.

You can tell curl to not change the non-GET request method to GET after a 30x response by using the dedicated options for that: --post301, --post302 and --post303.

 

 

 

 

:duff:

Wonko



#6 Zharif

Zharif

    Frequent Member

  • .script developer
  • 172 posts
  • Location:Germany
  •  
    Germany

Posted 11 September 2016 - 12:13 PM

Good news, the -L switch works.

Using -L in conjunction with -l and/or -i works too, but the resulting zip file seems to be corrupted and can be only opened by WinRAR :hyper:

 

Working command:

curl  -L "http://www.efsoftwar.../dw/wz6.cgi?cw"-o "%CD%\EFCommander.zip"

...downloads the file to the specified filepath.

 

Must have a look if wget provides a similar switch.



#7 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 11 September 2016 - 01:46 PM

Wget should have following 302 redirections active by default, this is why I asked, as said it may be a user agent or http referrer causing the issue with wget. :unsure:

http://stackoverflow...ollow-redirects

As simply put in the above, if you don't want redirection, use CURL WITHOUT -L (two negations make an affirmative ;)).

 

See if this helps:
https://bbs.archlinu...c.php?id=176806

it is possible that without a proper user agent or http referrer the 302 link changes ...  :dubbio:

 

Or it is possible that for *whatever* reasons the CURL -L manages to parse the 302 page and wget doesn't. :(

 

:duff:

Wonko



#8 Zharif

Zharif

    Frequent Member

  • .script developer
  • 172 posts
  • Location:Germany
  •  
    Germany

Posted 11 September 2016 - 05:08 PM

I'm very much afraid, that you last guessing is right.

 

I followed your links and whatever I try wgets' output looks always like this:

 

--2016-09-11 18:14:43--  http://www.efsoftwar...m/dw/wz6.cgi?cw
Resolving www.efsoftware.com... 109.237.132.30
Connecting to www.efsoftware.com|109.237.132.30|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.efsoftwar...ommander_64.zip [following]
--2016-09-11 18:14:43--  http://www.efsoftwar...ommander_64.zip
Reusing existing connection to www.efsoftware.com:80.
HTTP request sent, awaiting response... 302 Found
Location: http://www.efsoftwar.com/3.e [following]
--2016-09-11 18:14:43--  http://www.efsoftwar.com/3.e
Resolving www.efsoftwar.com... failed: Unknown host.
wget: unable to resolve host address `www.efsoftwar.com'

 

It looks that wget indeed is able to follow the redirection to the changed download link.

But note the missing "e" at http://www.efsofware.comwhen it tries to follow.

 

I tried:

wget --mirror --domains=http://www.efsoftware.comhttp://www.efsoftwar...m/dw/wz6.cgi?cw

wget --mirror http://www.efsoftwar...m/dw/wz6.cgi?cw

wget http://www.efsoftwar...m/dw/wz6.cgi?cw--referer=http://www.efsoftware.com

wget http://www.efsoftwar...ommander_64.zip--referer=http://www.efsoftware.com

Any of these produce a more or less similar output than the one posted above.

However, inside my available browsers (Firefox,IExplore, Edge) if you paste any of the two download links into the address line then it works.



#9 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 11 September 2016 - 06:01 PM

Well, if it is not referrer, then it is user agent.

 

This works fine here:

wget --user-agent="Mozilla/5.0" http://www.efsoftware.com/dw/wz6.cgi?cw
--19:59:37--  http://www.efsoftware.com:80/dw/wz6.cgi?cw
           => `wz6.cgi@cw'
Connecting to www.efsoftware.com:80... connected!
HTTP request sent, awaiting response... 302 Found
Location: http://www.efsoftware.com/016090/ef_commander_64.zip [following]
--19:59:38--  http://www.efsoftware.com:80/016090/ef_commander_64.zip
           => `ef_commander_64.zip'
Connecting to www.efsoftware.com:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 6,655,586 [application/zip]

    0K -> .......... .......... .......... .......... .......... [  0%]
   50K -> .......... .......... .......... .......... .......... [  1%]
  100K -> .......... .......... .......... .......... .......... [  2%]
  150K -> .......... .......... .......... .......... .......... [  3%]
.....
 6150K -> .......... .......... .......... .......... .......... [ 95%]
 6200K -> .......... .......... .......... .......... .......... [ 96%]
 6250K -> .......... .......... .......... .......... .......... [ 96%]
 6300K -> .......... .......... .......... .......... .......... [ 97%]
 6350K -> .......... .......... .......... .......... .......... [ 98%]
 6400K -> .......... .......... .......... .......... .......... [ 99%]
 6450K -> .......... .......... .......... .......... .........  [100%]

19:59:53 (433.31 KB/s) - `ef_commander_64.zip' saved [6655586/6655586]

:duff:

Wonko



#10 Zharif

Zharif

    Frequent Member

  • .script developer
  • 172 posts
  • Location:Germany
  •  
    Germany

Posted 11 September 2016 - 06:13 PM

Wow, Works for me as well.

I learned something here.

I'm really happy now to stay at wget (no need  to use a second tool for downloading tasks).

Thanks much Wonko for your help.



#11 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 11 September 2016 - 06:47 PM

Good :).

 

:duff:

Wonko



#12 Zharif

Zharif

    Frequent Member

  • .script developer
  • 172 posts
  • Location:Germany
  •  
    Germany

Posted 11 September 2016 - 07:08 PM

Wonko, after reading the wget man site I went on search for more possible values of the --user-agent switch.

After some search and findings I composed my new command line like this:


wget -N --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" --referer="http://www.efsoftware.com" "http://www.efsoftware.com/dw/wz6.cgi?cw"

wget -N --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" --referer="http://www.weihenstephan.de" "http://www.weihenstephan.de/~syring/win32/UnxUpdates.zip"

Am I right that usage of this command line should give a more general solution (for as most cases as possible) while trying to download something via http without changing the wget command line syntax

(whereas user agent vesions may be adapted of course)?



#13 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 September 2016 - 08:40 AM

I personally disagree :w00t:, but of course as long as it works it is ok :).

 

If this works:

 

wget --user-agent="Mozilla/5.0" http://www.efsoftwar...m/dw/wz6.cgi?cw

 

There is no reason (that I can see) to make it more complex than what actually works and that works exactly the same:

wget -N --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" --referer="http://www.efsoftware.com" "http://www.efsoftware.com/dw/wz6.cgi?cw"

 

the above may be a "template" or a "reference" from which you REMOVE unneeded parameters or simplify them, I would use it as a reference to ADD one by one the parameters in those rare cases where the plainer syntax does not work.

 

It is not like adding unneeded parameters and having stupidly long user agent strings you get "better" or "faster" bytes (if you do then by all means use the more complex command).

 

Namely "Mozilla5/0" without any other detail should be accepted by most servers, no need normally to tell the server which OS you are on, which browser you are using, etc. whilst on some site you might need another speciific user agent.

 

Otherwise you are simply "wasting" (very few, of course)  bytes BUT apart from this you introduce more possibilities of a typo or of a conflicting setting, if you are lucky the server will ignore those added parameters, but what tells you that one of them - besides the ones actually needed - do not actually prevent the download on another server?

 

In practice it won't make that much a difference, but philosophically I see it as "wrong".

 

:duff:

Wonko






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users