#1
  1. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2002
    Posts
    32
    Rep Power
    13

    need help with output redirection


    Hi all

    I've been trying to figure this out for days but still to no avail and I'm fairly frustrated. If you can help or have any suggestions, please post and help me out :)

    I've been trying to get the output of each command in a pipeline logged into a log file. For example, for the pipeline ls -l|grep sh|wc, I'd need to log the output of ls -l, that of grep sh and also that of wc in the output log file. I've managed to stitch up the commands so that piping works fine but as soon as I try to add the code for redirection to the output log, everything breaks down.

    So what I wanna know is: how do you redirect the output of intermediate commands to a file without breaking the pipe that's already there?

    Here's my code at the moment:
    Code:
        /* duplicate stdin as a copy of read_fd, so that input is redirected from read_fd
           read_fd is the file descriptor of the end of the pipe (of prev command) to get input from */
        if (read_fd)
        {
    	dup2(read_fd, STDIN_FILENO);
    	close(read_fd);
        }
    
        /* duplicate stdout as a copy of write_fd, so that output is redirected to write_fd
           write_fd is the file descriptor of the end of the pipe to write to*/
        if (write_fd)
        {
    	dup2(write_fd, STDOUT_FILENO);
    	close(write_fd);
        }
    
        if ( logging == TRUE )
        {
    	/* open output log, which gets file descriptor of 1, effectively replacing stdout */
    	if ( (outlog_fd = open(OUTPUTLOG, O_WRONLY | O_CREAT | O_APPEND)) == -1 )
    	    perror("open");
    
    	dup2(outlog_fd, STDOUT_FILENO);
        }
    
        /* execute command, searching for the executable file in the environment PATH variable if necessary */
        if ( execvp(command_argv[0], command_argv) == -1 )
        {
    	printf("%s: command not found\n", command_argv[0]);
    	exit(1);
        }
    thanks! :)
  2. #2
  3. *bounce*
    Devshed Novice (500 - 999 posts)

    Join Date
    Jan 2002
    Location
    Delft, The Netherlands
    Posts
    514
    Rep Power
    42
    Ok, here we go again :)

    The *NIX kernel in general has no provisions for duplicating streams. It can handle streams flowing from, say, A to B, but not from A to both B and C. That's just not how the kernel works.

    If you want to send data from one end to more than one destination, you have to do it yourself, explicitly copying every byte to all the write fds:

    Code:
    /* returns 1 on success, 0 on EOF, -1 on error */
    int
    split_stream (int src_fd, int dest_fd1, int dest_fd2)
    {
            char c;
            ssize_t retval;
    
            if ( (retval = read(src_fd, &c, sizeof(c))) < 0) {
                    perror("read");
                    return (-1);
            }
    
            if (retval == 0) {
                    /* EOF; caller should handle closing of fds etc */
                    return (0);
            }
    
            while ( (retval = write(dest_fd1, &c, sizeof(c))) == 0) {
                    ; /* do nothing */
            }
    
            if (retval < 0) {
                    perror("write");
                    return (-1);
            }
    
            while ( (retval = write(dest_fd2, &c, sizeof(c))) == 0) {
                    ; /* do nothing */
            }
    
            if (retval < 0) {
                    perror("write");
                    return (-1);
            }
    
            return (1);
    }
    Your next question then probably is, where to put this code? Time for another diagram:

    Code:
    -------------          -------------
    | program 1 |  (pipe)  | program 2 |
    |           +--------->|           +---> stdout (screen)
    |  (ls -l)  |          |    (wc)   |
    -------------          -------------
    That's basically what things look like now. Output from ls goes through the pipe to wc (or grep, or whatever). What you really want though, is this:

    Code:
    -------------        ----------        -------------
    | program 1 | (pipe) | logger | (pipe) | program 2 |
    |           +------->|........+------->|           +---> stdout
    |  (ls -l)  |        |    :   |        |    (wc)   |
    -------------        -----+----        -------------
                              |
                              v
                        (to logfile)
    That means you'll have to create a logger process for every process you want logged. It is very well possible to use one logger process that reads from multiple pipes and copies them all to even more pipes, but I think things are complicated enough for you already :)

    Hopefully this will give you some ideas on how to proceed. If not, I'm sure you'll let us know ;)
    "A poor programmer is he who blames his tools."
    http://analyser.oli.tudelft.nl/
  4. #3
  5. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2002
    Posts
    32
    Rep Power
    13
    Heh thanks for helping again Analyser :)

    Now, I've been trying to make sense of what you're saying and I think I've got most of it but I've no idea how to implement it in code. I've tried to, and for a few frustrating hours, but again no results. I'm not good at pipes and using dup2 I guess.

    How do I create a logger process to place in between the original 2 processes of a pipe? Do I have to create a new pipe and redirect the input of ls -l (from the above example) to the pipe and tie the pipe to the logger process? How do I do that? Forking the process?

    I'm really lost and I really need your advice (you did say to let you know :D)

    - Joel
  6. #4
  7. *bounce*
    Devshed Novice (500 - 999 posts)

    Join Date
    Jan 2002
    Location
    Delft, The Netherlands
    Posts
    514
    Rep Power
    42
    How do I create a logger process to place in between the original 2 processes of a pipe? Do I have to create a new pipe and redirect the input of ls -l (from the above example) to the pipe and tie the pipe to the logger process? How do I do that? Forking the process?
    The short version: yes, you'll need an extra pipe, and yes, it'll include another fork() for every logger process.

    The long version:

    You'll need an additional pipe for every logger process. Just look at the second scheme; The pipe from ls goes into the logger process instead of to wc, and there's a new pipe, going from the logger to wc. You have to set all that up in the parent process.

    The thing is, you might not want to rely on a version of tee being available, but the mechanics that it implements (reading from one fd and writing to many) is what you need. So you'll have to roll your own version of tee, basically.

    Now, you don't have to write it as a seperate program. You can just stuff the same functionality into a function, and call it from a child process, thus creating a logger.

    One note to the scheme: the down-arrow to the logfile is (obviously) not a pipe; the logger just has to open a file to write to.

    As always, good luck ;)
    "A poor programmer is he who blames his tools."
    http://analyser.oli.tudelft.nl/

IMN logo majestic logo threadwatch logo seochat tools logo