Script Suggestions

Code junkies hangout here

Moderators: ChrisThornett, LXF moderators

Script Suggestions

Postby LeeNukes » Fri Jan 21, 2011 10:10 pm

Hi guys,

Written several bash scripts to run through a process, stuck on one due to it launching a program which it becomes stuck in.

Essentially the program runs, outputs strings into a file, once those strings have ran through, another program needs to be ran which then compares the strings which ran through a program with the strings which are in another file.

It won't skip to the next step unless the program is quit, normally Ctrl+C will do this as normal.

There is no real set time at how long the program takes, anything from 10 minutes to 30 minutes probably, but ideally I want the next step to run straight after the program finishes.

I was thinking of something with wc maybe? To compare the two files line counts, but how do I get a wc going whilst its running the program?
User avatar
LeeNukes
LXF regular
 
Posts: 954
Joined: Sun Jun 21, 2009 8:11 pm
Location: At the bar

Postby MartyBartfast » Fri Jan 21, 2011 10:30 pm

You could put the program into the background, and then loop round waiting for it to complete, checking it's status with jobs e.g:





Code: Select all
$ sleep 60s &
[1] 18685
$ jobs -l     # do this in a timed loop
[1]+ 18685 Running                 sleep 60s & 
$ jobs -l
[1]+ 18685 Running                 sleep 60s &
$ jobs -l
[1]+ 18685 Running                 sleep 60s &
$ jobs -l
[1]+ 18685 Running                 sleep 60s &
$ jobs -l
[1]+ 18685 Done                    sleep 60s
$  # now go on and do the next bit
I have been touched by his noodly appendage.
User avatar
MartyBartfast
LXF regular
 
Posts: 815
Joined: Mon Aug 22, 2005 7:25 am
Location: Hants, UK

Postby nelz » Fri Jan 21, 2011 11:19 pm

Could you make the file it writes to a FIFO? Then your script can read the data as it is being written.
"Insanity: doing the same thing over and over again and expecting different results." (Albert Einstein)
User avatar
nelz
Site admin
 
Posts: 8489
Joined: Mon Apr 04, 2005 11:52 am
Location: Warrington, UK

Postby LeeNukes » Sat Jan 22, 2011 2:43 pm

The program by default writes to STDOUT but I'm redirecting it into a file.

Its essentially translation software, so it will take in the strings in one language and out put them to another.

It's a service though, so once its processed the file, it sits there. Waiting for input from STDIN.

As it waits for input from STDIN I don't think the loop method will work as it won't ever be "done".
User avatar
LeeNukes
LXF regular
 
Posts: 954
Joined: Sun Jun 21, 2009 8:11 pm
Location: At the bar

Postby nelz » Sat Jan 22, 2011 10:01 pm

Can you control the data it is fed, such as sending an EOF signal so the program knows there is no more data and it is time to exit.

Otherwise, you'll need to run the program in the background and read data from its file as and when it appears, possibly with a while read loop.
"Insanity: doing the same thing over and over again and expecting different results." (Albert Einstein)
User avatar
nelz
Site admin
 
Posts: 8489
Joined: Mon Apr 04, 2005 11:52 am
Location: Warrington, UK

Postby LeeNukes » Sat Jan 22, 2011 11:35 pm

The data can be added to I guess, just as long as it doesn't try to translate it.

How would I append an end to it? Would it depend on the program supporting an EOF? Or can a ctrl+c be put in somehow?
User avatar
LeeNukes
LXF regular
 
Posts: 954
Joined: Sun Jun 21, 2009 8:11 pm
Location: At the bar

Postby nelz » Sun Jan 23, 2011 12:08 pm

That depends on the program doing the reading. Do you have any control over that?
"Insanity: doing the same thing over and over again and expecting different results." (Albert Einstein)
User avatar
nelz
Site admin
 
Posts: 8489
Joined: Mon Apr 04, 2005 11:52 am
Location: Warrington, UK

Postby LeeNukes » Sun Jan 23, 2011 3:23 pm

nelz wrote:That depends on the program doing the reading. Do you have any control over that?


I have control over everything, this is the line for feeding in the data at which point I get caught in the program:

Code: Select all
# Translate scoring data using moses engine
/opt/moses/bin/moses -f $basedir/binarized/moses.ini < $scoringdir/$3.strue.$4 > $scoringdir/$3.mosesscore.$5


the -f is the config file for the particular engine, the $scoringdir/$3.strue.$4 is the input data and obviously the $scoringdir/$3.mosesscore.$5 is the output.

Here is the help for the program: http://www.leenukes.co.uk/documents/mosesinfo
User avatar
LeeNukes
LXF regular
 
Posts: 954
Joined: Sun Jun 21, 2009 8:11 pm
Location: At the bar

Postby nelz » Sun Jan 23, 2011 6:55 pm

Is the problem that moses keeps waiting for more data from $scoringdir/$3.strue.$4 even when there is no more? If the program writing to that file closed it when it had finished, I would expect moses to exit as it would see an EOF.

Or is the hangup elsewhere in the workflow?
"Insanity: doing the same thing over and over again and expecting different results." (Albert Einstein)
User avatar
nelz
Site admin
 
Posts: 8489
Joined: Mon Apr 04, 2005 11:52 am
Location: Warrington, UK

Postby LeeNukes » Mon Jan 24, 2011 9:54 am

it just sits there waiting for more input unless you kill it, even at the end of the file.

The process is it loads its engine, then takes in the data, then when its finished, just sits there.

I'll try and craft an end of the file, see if it works.
User avatar
LeeNukes
LXF regular
 
Posts: 954
Joined: Sun Jun 21, 2009 8:11 pm
Location: At the bar

Postby LeeNukes » Tue Jan 25, 2011 3:11 pm

Scratch this, it appears that after some time, it kills itself. :oops:
User avatar
LeeNukes
LXF regular
 
Posts: 954
Joined: Sun Jun 21, 2009 8:11 pm
Location: At the bar

Postby nelz » Tue Jan 25, 2011 3:36 pm

Suicidal software? :shock:
"Insanity: doing the same thing over and over again and expecting different results." (Albert Einstein)
User avatar
nelz
Site admin
 
Posts: 8489
Joined: Mon Apr 04, 2005 11:52 am
Location: Warrington, UK


Return to Programming

Who is online

Users browsing this forum: No registered users and 0 guests