Shell Scripting

I'm trying to write a shell script to query a webpage for some info. how i can do is this, given the following:

telnet somewhere.com 80

how do i enter more commands once i have entered the telnet session? i have tried EVERYTHING i can think of from piping to echoing what i want to execute. the problem is that as soon as i get into telnet i cant enter more commands. i have been reading and reading the man telnet, and i think i can send arguments? possible variables that contain my commands? but i just can't figure this out!!! anyone help me plz b4 i burn my O'Reilly fish book and dance around the flames??

Check out Expect for scripting interactive sessions of stuff.

cool thnx. i have d/l and installed most recent versions of tcl and expect. tried toying around with them but i am dumbfounded. from what i have read about them they seem pretty useful. heh, i am now learning java, perl, html, and tcl all at the same time! LOL, i wonder if my brain will explode soon. now if only i could find a way to get credit for all this stuff at school i would be set

edit: found some nice tutorials this is actually pretty easy so far, thanks again.

Again I'll try to upload my sysadmin instructor's script for fetching the IBM stock quote off of Yahoo. In terms of inputting commands to telnet, he wrote a sub-shell that sends a GET command and pipes it to telnet, then redirects the output from telnet to a variable which he then searches for the desired data.

I had tried to upload the file before, but munged it when I tried to tar it. This time I've gotten it right, though the tar file is somewhat larger than the original (nearly 10 times larger).

Another possible source is an out-of-print O'Reilly book, "Web Client Programming with Perl" by Clinton Wong. It's available on-line at http://www.oreilly.com/openbook/webclient/ . Perl has several special-purpose modules. This book makes use of the LWP module which handles HTTP.

cool thanks i have d/l that and will be checkin it out. i always seem to learn best from examples. one of the things that makes all this stuff a little easier is the similarity between the languages. perl, tcl, and shell scripting are all so closely related it seems... makes it much easier on my brain, though i have been semiconfused when i go back to C++ for doing my schoolwork

My advice when somebody wants to know which language to learn first is for them to choose one of the C-ish ones, because that also gives them a head start to learning the others.

My understanding of the history (greatly oversimplified, I'm sure) is that UNIX was originally written in C in order to make it more portable between different hardware platforms -- only the device drivers close to the hardware would have to be rewritten.

So it looks to me like the different languages in UNIX/Linux derived most of their syntax and function libraries from those of C. Sure makes it easier to learn the languages and to figure out their code.

=yea i definitly agree with that. i've been studying C++ for about 3 years now. i just completed my first multithreaded program using semaphores, so feel semi-knowledgable about the language now. but what really interests me the most in Network Programming/Security. I have been reading ahead of my class in my Win32 book about sockets... and hmm, to the say the least: this **** looks incredibly confusing!!! Sooo many complex functions/parameters, i mean, my teacher doesnt even know what half the parameters are when dealing with threaded programs functions! He even told us he has no idea how to even start debugging a multi-threaded program, ! The reason i got started with perl and shell scripting is b/c i heard they were much simpler for accomplishing the kind of stuff i was interested,(network). and damn, they are incredibly simple. Expect is one of the coolest programs i've ever used! In the last day i've written a little menu driven script with Expect to do bunch of things in telnet that i couldnt even fathom writing pseudocode for in C++!

Sockets programming (AKA "network programming") is a lot easier than it looks. I myself spent almost a year in analysis paralysis trying to figure it all out and afraid to try anything until I had it all figured out. But then I read the right book that got me started coding (see below).

Visit my page on the subject, "DWise1's Sockets Programming Pages " at http://members.aol.com/DSC30574/sockets/index.html . It's not finished yet -- actually I'm in the middle of a rewrite of what I had. But it should help you get started. In particular, check out the resources, which will point you towards other sites and to books on the subject.

I recommend the book that got me started writing networking code, "The Pocket Guide to TCP/IP Sockets: C Version" by Michael J. Donahoo and Kenneth L. Calvert. The second edition of the book is titled "TCP/IP Sockets in C: Practical Guide for Programmers". Both books are $15 . They are clear and concise and give you short programs (echo clients and servers) to use as a basis for your own programs. ISBNs and the books' web sites are on my page, including a link for converting UNIX sockets programs to Winsock.

Here's the skinny: network programs under Windows use the Winsock DLL and API. In addition to scads of special and powerful functions (which are most likely the ones confusing you right now), the API also contains and supports the same Berkeley sockets functions and data structures that UNIX uses. You don't need all those fancy WSA_ functions and MFC socket classes and the like to get started; just Berkeley sockets will do the job for you. So you can write a network program for UNIX and convert it to Winsock , and vice versa, with just a few minor changes, all of which are covered in a PDF file I provide a link to (well, it was Donahoo and Calvert's link to begin with). Donahoo and Calvert's sites also provide source code from their books and the same source code converted to Winsock.

Another point: with sockets programming, you're just programming at the transport and IP layers, just working with IP addresses (with some cool built-in functions to do DNS for you) and sockets and with the tcp and udp protocols. You make the connections and you send and receive the packets. That's actually the easy part. Above that are the application protocols, like telnet, FTP, HTTP, NTP, SNMP. That's where it can get interesting.

I hope that made sense and will help you make sense of what you're reading.

Actually, my first network programs were from Lincoln Stein's "Networking with Perl". However, when I started writing sockets programs in C, I found that my C programs could not talk with my Perl programs (eg, a Perl echo client trying to talk with a C server, or a C client trying to talk with a Perl server). I think that Perl was doing something extra that the C programs didn't know about (probably something about the print or <>). I never did track the problem down, but instead just do all my network programs in C where I know what it's doing and can control it better.

BTW, Donahoo and Calvert also wrote a version of their book for Java. In that case, you do have to use Java's networking package instead of straight sockets.

I haven't quite made the plunge yet into tcl. Right now, I'm busy both with work and with a Linux systems programming class using C++ -- actually, the programming is in C, but he wants us to use iostreams for console I/O. That and allocating with new are the only things C++ about our assignments.


The strength of a language is in how it not only enables you to express your ideas, but also in how it leads you to new and useful kinds of ideas. I've always believed that the languages we use help to structure how we think (I started out as a foreign language major).

heh, foreign language -> computer languages ... interesting, it seems that just knowing English is not satisfying enough for great minds like ours
i briefly browsed over your site, seems interesting so far, i will have to add it to my links. yes, most of what i was reading above was making sense, im sure once i actually start diving deeper into socket programming it will come as easily as the rest has, but it does indeed look confusing at a first glance. I am definitly glad that i have learned C before such languages as perl/expect. They almost seem like "cheats", like letting u get around all of the sometimes annoying strict guidelines/constraints of C. (how many times have u thought of new variables u needed on the fly and had to go page up to your declarations sections to add them?? -1000's of times here). But even though C++ can be annoying, i am fully aware of the immense importance of such things as requiring variable declarations, etc. perl and expect are just more "fun" i guess u could say. LIke u said, " The strength of a language is in how it not only enables you to express your ideas, but also in how it leads you to new and useful kinds of ideas." - and C certainly seems to be an contain an endless number of new ways to accomplish things, at least from what i have done so far. It's interesting that i will learn one way of solving a problem, and then a few months later a much easier way will come.... and then AGAIN we will come across yet another way and another.... heh u get my point

It's not so much that a new language offers us new ways to solve problems, which they do, but also that they get us to look at problems differently.

For instance, with C we almost have to think about how the variables are stored in memory; we have to think closer to the metal than with perl, for instance. One of the two primary problems I see the beginners ask about here seem to stem from their not understanding the relationship between character strings and character pointers, which is second-nature for more experienced C'ers (the other primary problem seems to be with scope). And once we understand it, then that structures how we approach a new problem.

BTW, the first programming class we had to take was Fortran. Coming from a foreign language background, I just approached it like yet another language (I'm more a grammatical type of guy than aural). So while everybody else was trying to debug their programs with random changes, I was reading my code in terms of what I was telling the computer to do.

A quick note on the most common mistakes in sockets programming. For multiple-byte values, there's network order and host order. The address and port values need to be in network order, which is the reverse of host order on an Intel machine. There are built-in functions to handle this conversion (htonl, htons, ntohl, and ntohs), but you need to remember to use them. I've caught myself a couple times forgetting to use htons to convert the port number and wondering why it wouldn't work.

One question arises here about the data payload of a packet: Do we also need to put that in network order? So far, I've only been connecting Intel machines, so it hasn't come up. I think that needs to be researched further.

The other common mistake is mixing tcp and udp clients and servers. tcp clients and servers will talk to each other, and udp clients and servers will talk to each other, but tcp doesn't talk to udp. So if you set a server to use, for example, udp port 37 and tell a client to connect to it using tcp, it won't work. I think the first question I answered on this forum was due to that.

I just answered my own question: it does matter.

Because we work with Network Time Protocol (NTP) servers, one of my projects was an NTP client. Six of the fields in the NTP packet are long integers. When I wrote the client, I just assumed that those longs needed to be sent in network byte order, so I used the htonl() and ntohl() functions to stuff and extract those fields. Since that part worked first-time, I didn't think any more of it and forgot about it.

I ran a test in which I left those byte-order conversions out. It still ran, but now the times and dates were way off (2008-02-21 instead of 2003-04-14).

The insidious aspect of this error is that if you write your own server and client apps and test them on the same architecture (eg, intel, be it Windows and/or Linux), then you won't catch the error. Then when somebody tries to use either one on an architecture that has a different byte-order, you won't be able to figure out why it works on some computers and not on others.

So that's a caveat for us. Byte order does matter, even in the data.