#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2013
    Posts
    19
    Rep Power
    0

    Wget not working


    hey folks,

    i am trying to download a file using wget for the following url.

    http://pubmlst.org/perl/bigsdb/bigsdb.pl?db=pubmlst_bordetella_genome_isolates&page=plugin&name=Contigs&format=text&isolate_id=6069 7&match=1&pc_untagged=0&min_length=&header=1

    OR

    http://pubmlst.org/perl/bigsdb/bigsdb.pl?db=pubmlst_bordetella_genome_isolates&page=downloadSeqbin&isolate_id=60697

    I have tried various options with wget and curl but everytime it fails to download the file.
    i cant find any other way how to get wget/curl working.
    i intend to change the isolate_id=$$$ in the url to download various file.

    can anyone help me in solving this issue.
  2. #2
  3. Come play with me!
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,757
    Rep Power
    9397
    Did you quote the URL?
    Code:
    wget 'http://pubmlst.org/perl/bigsdb/bigsdb.pl?db=pubmlst_bordetella_genome_isolates&page=plugin&name=Contigs&format=text&isolate_id=60697&match=1&pc_untagged=0&min_length=&header=1' -O bigsdb.pl
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2013
    Posts
    19
    Rep Power
    0
    Originally Posted by requinix
    Did you quote the URL?
    Code:
    wget 'http://pubmlst.org/perl/bigsdb/bigsdb.pl?db=pubmlst_bordetella_genome_isolates&page=plugin&name=Contigs&format=text&isolate_id=60697&match=1&pc_untagged=0&min_length=&header=1' -O bigsdb.pl
    Yes i have tried to quote it. i dont want the perl code file but the actual data that is displayed on webpage
  6. #4
  7. Come play with me!
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,757
    Rep Power
    9397
    Originally Posted by ap88
    i dont the perl code file but the actual data that is displayed on webpage
    You don't what? Is wget downloading something? If not then what is the exact command you're executing and what happens when you try? Actual copy/paste from your terminal window would be great.
    Once it is downloading, is it downloading what you want it to get? If not then what is it getting and what did you want it to get?
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2013
    Posts
    19
    Rep Power
    0
    Originally Posted by requinix
    You don't what? Is wget downloading something? If not then what is the exact command you're executing and what happens when you try? Actual copy/paste from your terminal window would be great.
    Once it is downloading, is it downloading what you want it to get? If not then what is it getting and what did you want it to get?
    ok here is the command i am using:

    wget --user=apirani --password=MYPASSWORD 'http://pubmlst.org/perl/bigsdb/bigsdb.pl?db=pubmlst_bordetella_genome_isolates&page=downloadSeqbin&isolate_id=60697' -O test.fas

    The output of below command contains:

    !DOCTYPE html>
    <html>
    <head>
    <title>Log in - Bordetella genomes</title>
    <meta name="viewport" content="width=device-width" />
    <link rel="stylesheet" type="text/css" href="/bigsdb.css?v=20131202" media="Screen"/>
    <link rel="stylesheet" type="text/css" href="/jquery-ui.css?v=20131202" media="Screen"/>
    <style type="text/css">
    <!--/* <![CDATA[ */
    .tooltip{display:none} ....................................

    I need this output:

    >14157965|SRR932188_A93513_NODE_1_COV_17.919388
    CCGCGCCGCTGCTGATTTCCATCCCCCATCTGGGCAGCCTGATCCCCGAGGCGCAAAAGC
    CGATCGCCAGCGTCATTCCGCGCCTGTTCGAAGGCAAGCTGCCCGACCTGAACATCGGCA
    CCTCCAGCGGCGCCAGCGCCGCGCCCGCCATCCAGGACGCCATCCAGCGCGAGCTGCAAG
    GGCAGTCGGCCTATACCTGGGCGGTCAATGGCCGTTTCAAGGGCGG
  10. #6
  11. Come play with me!
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,757
    Rep Power
    9397
    Their login mechanism uses cookies, not HTTP authentication. You have to submit a log in request, remember the cookies it sends back, then submit your second request using those cookies.
    Here's a example of how to do that.
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2013
    Posts
    19
    Rep Power
    0
    Originally Posted by requinix
    Their login mechanism uses cookies, not HTTP authentication. You have to submit a log in request, remember the cookies it sends back, then submit your second request using those cookies.
    Here's a example of how to do that.
    Thanx requinix. It helped.
    I downloaded my cookies file and used the following command.

    wget --load-cookies cookies.txt --keep-session-cookies -O filename 'URL'

    But i have another problem to deal with. The URL which now i am using to download does not have any query string(hidden).

    Any idea how to resolve this?
  14. #8
  15. Wiser? Not exactly.
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    May 2001
    Location
    Bonita Springs, FL
    Posts
    5,906
    Rep Power
    3969
    If it's using POST and sending data then you use the --post-data option of wget to send that same data.
    Recycle your old CD's, don't just trash them



    If I helped you out, show some love with some reputation, or tip with Bitcoins to 1N645HfYf63UbcvxajLKiSKpYHAq2Zxud

IMN logo majestic logo threadwatch logo seochat tools logo