#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2011
    Posts
    6
    Rep Power
    0

    Interacting with another program


    I am trying to write a python program to interact with a separate program (UCI chess engine). To do so I need to read and write via stdin and stdout. Thus far I have had limited success and thought I would reach out for some help.

    I have been trying to use the SUBPROCESS module and have been able to capture some limited output using the CHECK_OUTPUT command. The main issue with this is that it closes the instance of the program I am interacting with as soon as the output is returned. Ideally I would like to keep the program running in the background rather than opening and closing many times.

    At any rate, I seem to be making this harder than it should be. I think it should be a fairly simple task to do but it is not turning out that way.

    To summarize, I am trying to:
    1. Open chessengine.exe
    2. Send command string
    3. Read output
    4. Return to step 2, etc.

    Thanks in advance for any suggestions.
  2. #2
  3. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481

    2 way pipe.


    time to sleep and I haven't gotten my adorable example to work.

    You can get assistance at freechess.org
    or read the instructions for connecting a chess engine to xboard

    good night.
  4. #3
  5. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481

    my example, which doesn't work as I expect.


    Code:
    #python version 3
    import subprocess as S
    
    with S.Popen(('cat','-n',),shell=False,stdin=S.PIPE,stdout=S.PIPE,bufsize=1,universal_newlines=True) as p:
        p.stdin.write('these\n')
        print(p.stdout.readline())
    I tried a few different values of bufsize. Sorry.
    I'll report the __exit__ undefined in python2.7 error as python bug.
  6. #4
  7. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2011
    Posts
    6
    Rep Power
    0
    Thank you very much for the suggestion. The universal_newlines option was a big help. I was unaware of it and was trying to encode / decode the output so that makes things a bit cleaner.

    It is quite close, but the p.stdout.readline() does not capture all of the text. For example, sending the command UCI to the engine receives several lines of text in return but I am only capturing the first line. I tried using p.stdout.read() and p.stdout.readlines() and they both hang.

    I also tried using a loop to iterate through the lines but that is hanging as well. I tried:
    Code:
    for lines in p.stdout:
    	print(p.stdout.readline())
    Any other suggestions would be greatly appreciated.
  8. #5
  9. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481

    Trouble in River City


    Your code will print alternate lines from the input as does this example. python3 simplifies the demonstration, I don't need an external file.

    Code:
    #! python3
    import io
    p = io.StringIO('line 0\nLINE B\nLine c\nlInE 3\n')
    for lines in p:
        print(p.readline(),end='')
    
    '''
              output from this program:
    LINE B
    lInE 3
    '''

    You might have meant
    Code:
    for line in p.stdout.readlines():
        print(line)
  10. #6
  11. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2011
    Posts
    6
    Rep Power
    0
    Thanks for the comments but I am still having issues retrieving all of the slave program's output. Commands have been highlighted for reference and the output of the slave program is in the normal text.

    Here is an example command line session I am trying to duplicate.

    Stockfish 2.1.1 JA by Tord Romstad, Marco Costalba and Joona Kiiski
    uci
    id name Stockfish 2.1.1 JA
    id author Tord Romstad, Marco Costalba and Joona Kiiski

    option name Use Search Log type check default false
    option name Search Log Filename type string default SearchLog.txt
    option name Book File type string default book.bin
    option name Best Book Move type check default false
    option name Mobility (Middle Game) type spin default 100 min 0 max 200
    option name Mobility (Endgame) type spin default 100 min 0 max 200
    option name Passed Pawns (Middle Game) type spin default 100 min 0 max 200
    option name Passed Pawns (Endgame) type spin default 100 min 0 max 200
    option name Space type spin default 100 min 0 max 200
    option name Aggressiveness type spin default 100 min 0 max 200
    option name Cowardice type spin default 100 min 0 max 200
    option name Minimum Split Depth type spin default 4 min 4 max 7
    option name Maximum Number of Threads per Split Point type spin default 5 min 4
    max 8
    option name Threads type spin default 2 min 1 max 32
    option name Use Sleeping Threads type check default false
    option name Hash type spin default 32 min 4 max 8192
    option name Clear Hash type button
    option name Ponder type check default true
    option name OwnBook type check default true
    option name MultiPV type spin default 1 min 1 max 500
    option name Skill Level type spin default 20 min 0 max 20
    option name Emergency Move Horizon type spin default 40 min 0 max 50
    option name Emergency Base Time type spin default 200 min 0 max 30000
    option name Emergency Move Time type spin default 70 min 0 max 5000
    option name Minimum Thinking Time type spin default 20 min 0 max 5000
    option name UCI_Chess960 type check default false
    option name UCI_AnalyseMode type check default false
    uciok
    isready
    readyok
    position fen 8/4K3/4NN2/p3p3/rnp1p3/1pk5/bp1n4/qrb1N3 w
    go
    info depth 1
    info depth 1 seldepth 1 multipv 1 score cp -4129 nodes 106 nps 1139 time 93 pv e
    7d6
    info nodes 106 nps 1139 time 93
    bestmove e7d6

    Here is the program that I am trying, which is based on your first example.

    Code:
    import subprocess as S
    commands = ['uci', 'isready', 'position fen 8/4K3/4NN2/p3p3/rnp1p3/1pk5/bp1n4/qrb1N3 w', 'go']
    proc = S.Popen('c:\stockfish.exe', stdin=S.PIPE, stdout=S.PIPE, bufsize=1, universal_newlines = True)
    for i in commands:
        proc.stdin.write(i + '\n')
        print(proc.stdout.readline())
    and here is the program output.

    >>>
    Stockfish 2.1.1 JA by Tord Romstad, Marco Costalba and Joona Kiiski

    id name Stockfish 2.1.1 JA

    id author Tord Romstad, Marco Costalba and Joona Kiiski



    >>>

    And now the question - why does my program output not match the command line output?
  12. #7
  13. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481

    incremental idea


    (a decade ago?) I got this sort of thing to work with fluent as the program I wanted to control.

    The idea is to identify the interactive prompt returned by the program. Maybe this will work on your system? It doesn't work on mine. I took the isready command out of your command list. The code issues two commands for each command you actually care about: the one you want followed by isready. I presume the output from stockfish following isready is always 'readyok'. The program reads output from stockfish until it finds 'readyok' then you can issue the next command paired with isready.

    Good luck.
    Code:
    import sys
    
    chess = r'c:\stockfish.exe /usr/games/stockfish'.split()['linux' in sys.platform]
    
    import subprocess as S
    getprompt = 'isready'
    commands = ('uci', 'position fen 8/4K3/4NN2/p3p3/rnp1p3/1pk5/bp1n4/qrb1N3 w', 'go',)
    proc = S.Popen(chess, stdin=S.PIPE, stdout=S.PIPE, bufsize=1, universal_newlines = True)
    
    for i in commands:
        proc.stdin.write(i + '\n')
        proc.stdin.write(getprompt + '\n')
        for line in proc.stdout.readline():
            print(line)
            if 'readyok' == line:
                break
    I suppose my trouble is that I don't understand how buffering works on unix.
    Last edited by b49P23TIvg; October 19th, 2011 at 09:09 PM. Reason: completion
  14. #8
  15. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481

    Fixed!


    This works for me. The other post was bad code---Good idea, Wrong implementation. Thank you for pointing out that I was missing new lines from my tests and struggles!!!

    Code:
    import sys
    
    chess = r'c:\stockfish.exe /usr/games/stockfish'.split()['linux' in sys.platform]
    
    import subprocess as S
    getprompt = 'isready'
    done= 'readyok'
    
    commands = ('uci', 'position fen 8/4K3/4NN2/p3p3/rnp1p3/1pk5/bp1n4/qrb1N3 w', 'go',)
    proc= S.Popen(chess, stdin=S.PIPE, stdout=S.PIPE, bufsize=1, universal_newlines=True)
    
    for i in commands:
        proc.stdin.write(i + '\n')
        proc.stdin.write(getprompt + '\n')
        while True:
            line= proc.stdout.readline()
            print(line)
            if done == line[:len(done)]:
                break
    Last edited by b49P23TIvg; October 19th, 2011 at 11:08 PM. Reason: too
  16. #9
  17. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2011
    Posts
    6
    Rep Power
    0
    I just wanted to follow up and say that I got this to work as well. Thanks so much for all of your help.
  18. #10
  19. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2010
    Posts
    1
    Rep Power
    0

    Problems with this approach


    I am trying to do something very similar.

    I was able to get this approach to work too, but I find that it doesn't work with some other chess engines. I think it has something to do with buffering but I'm not certain. I get a lot of blank lines with some engines.

    I am noticing that Crafty has an annotate feature where you can tell it to annotate an entire PGN input file, and an input feature where you can tell it to read commands from an input file. So this might be the solution for chess.

    But for the more general question of how to interact with an external terminal program, I am thinking maybe telnet would work better. I am working in Linux so I can make the assumption that console logins are OK.
  20. #11
  21. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481

    thoughts


    My approach absolutely depends on knowing what the next "ready" prompt will be. I've also used regular expressions to match a set of distinguishable possible "ready" prompts.

    If chess is your interest, the people at
    http://www.freechess.org
    FICS are helpful. They host discussion a group devoted to interfacing chess engines to other programs. If you write your own chess program it can play at FICS, if you're nice and agree to the rules. They accept variants on available codes. For instance, stockfish is already (given correct recollection) available to play. They might however, accept stockfish with custom weights, or self-learning weights.

    (oh boy! There is a program at FICS that "learns" by collecting a database of moves. This is infeasible because the tree of moves is large. I've just cut the number of variables to as low as 6. Cool---stockfish driven by Nelder-Mead.)
  22. #12
  23. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Posts
    2
    Rep Power
    0
    im working on the same project. im making a chess playing robotic arm. so i need to communicate with the chess engine and take the output generated by the engine to my program. i tried the codes provided by u guys but its only printing what's already there in the command window "Stockfish 2.2.2 JA by Tord Romstad, Marco Costalba and Joona Kiiski"
    that means im not able to write the command strings to the engine
    pls help me
  24. #13
  25. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481
    I've attached the description of the interface, although I'd think engine-interface.txt comes with stockfish installation.

    The code of my post on
    October 19th, 2011 11:05 PM
    still works. Upon review, the only change I'd make is to change
    print(line)
    to
    sys.stdout.write(line)
    And the code work with both python2 and python3.

    I'm using current Ubuntu linux distribution.

    Do you have a recent installation of python with a current subprocess module?
    Attached Files
    [code]Code tags[/code] are essential for python code and Makefiles!
  26. #14
  27. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Posts
    2
    Rep Power
    0
    im using python 2.7 on windows..... can that be the problem. ???
    import subprocess as S
    getprompt = 'isready'
    done= 'readyok'

    commands = ('uci', 'position fen 8/4K3/4NN2/p3p3/rnp1p3/1pk5/bp1n4/qrb1N3 w', 'go',)
    proc= S.Popen('c:\stockfish.exe', stdin=S.PIPE, stdout=S.PIPE, bufsize=1, universal_newlines=True)

    for i in commands:
    proc.stdin.write(i + '\n')
    proc.stdin.write(getprompt + '\n')
    while True:
    line= proc.stdout.readline()
    print(line)
    if done == line[:len(done)]:
    break

    this is what i used.....
  28. #15
  29. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481
    Beats me. Sure looks like similar code. And you claimed to have started stockfish. I don't do windows but thought that the subprocess module was supposed to be operating system independent. Maybe the line endings differ? Maybe you have CR LF to contend with? I don't think that's the trouble though. I'm quite clueless.
    [code]Code tags[/code] are essential for python code and Makefiles!

IMN logo majestic logo threadwatch logo seochat tools logo