Email thread with Xspectrum on 4/17/2022
Lambda 2m IOC stability (subject of thread)
From Suresh: Another problem that happens across all the Lambda’s: during repeated acquire calls after a previous “Done Acquire”,
we are able to get the IOC to hang. Tejas found that dc script has 1 sec gap between repeats. We have been using the same 1 sec.
The frequency of these hangs is proportional (not necessarily linear) to the gap between “Done and next Acquire”. If we go
below 1 sec, then we can make this happen very soon.
It seems like the Done signal is not necessarily telling us that it is Done.
Would you be able to reproduce and possibly see what is the cause?
Note the new features on the lower right of att. 1.
[4/13 12:43 PM] Guruswamy, Tejas
Lambda and chiller are now on. So Dufresne, Eric would you prefer if I left the software config as is, or upgrade
to Keenan's latest IOC so that if it does get stuck we might get a more useful error message?
[4/13 12:44 PM] Guruswamy, Tejas
I would test acquisition after upgrading of course
[4/13 1:59 PM] Narayanan, Suresh
Guruswamy, Tejas is it just ioc update or lib update as well? I would vote for it if only
ioc but it is Dufresne, Eric call
[4/13 2:02 PM] Dufresne, Eric
Guruswamy, Tejas I also agree with Suresh, would be helpful to report error so we can learn.
As long as you are around Tejas next week, let's go for it.
[4/13 2:03 PM] Guruswamy, Tejas
Ok, thanks Eric. Will keep a backup and undo if my testing suggests any problems.
[4/13 2:04 PM] Narayanan, Suresh
his most recent update also updates mode/threshold etc only if changed rather than
everytime which is the right thing to do
[4/14 11:18 AM] Guruswamy, Tejas
I think I'm satisfied that the IOC upgrade on the 250k (and 750k) has resolved several issues and not introduced any new problems
(what remains is in my email). If any of you would like to test, please go ahead. Just note that the 250k will always mark the
first frame of every acquisition as bad in its current state -- I am not restarting it yet in case X-Spectrum would like to
take a look since it seems a firmware issue.
Dufresne, Eric do I guess correctly that your SPEC script is relying on an IOC status PV returning 'Done' to start the next acquisition?
[4/14 11:26 AM] Narayanan, Suresh
thanks Guruswamy, Tejas for the thorough tests, very exhaustive tests.
[4/14 11:26 AM] Narayanan, Suresh
it is correct that we are checking for 'Done' to start next acq
[4/14 11:28 AM] Narayanan, Suresh
in usual acq, where we acq 100s of frames in a single acq and repeated over, there are plenty of overheads
in spec and so should not be an issue. When people do detector scans is when at each point in the scan,
they take 1-2 images and that usually has been problematic
[4/18/22 4:18 PM] Guruswamy, Tejas
Sure. As far as I can tell, the 250K has two outstanding issues: '1 bad frame' (acquisitions finish ok but the first frame
always is marked as incomplete; the IOC may crash on the first occurrence of this); and also the same issue as other Lambdas
with getting stuck starting an acquisition if it is too soon since the previous one ("done" is not reliable).
Currently the detector is on but the firmware is in the 1 bad frame state. You may want to power cycle it to reset that.
And based on our findings if it's not already there you probably want to add a delay between acquisitions in your SPEC
scripts to reduce the chance of it getting stuck.
[4:28 PM] Guruswamy, Tejas
If it does happen to crash, the system is set up to automatically save a core dump now
[4/19/2022 5:01 PM] Narayanan, Suresh
Lang, Keenan C. i heard from JoAnne and Mark that Mark pointed some issues in the lambda driver and that
they are now fixed by you. Is it related to the 2m problem? Should we update 2m and 250k ioc, keeping in
mind that we are getting ready to take data soon.
[7:04 PM] Dufresne, Eric
Guruswamy, Tejas We are taking data now on 250K. We started alignment and spec scans where we take one image
and move a motor. This is now worse than before the upgrade... The IOC doesn't crash but the scan hangs.
We managed to do what we need for now, so we started a long time series, 6 hours. This is done with multiple
image and is typically stable so I am hoping this will remain so. Let's cross our finger and hope for
the best.
[7:05 PM] Narayanan, Suresh
maybe we should add more sleep to the scan
[7:05 PM] Narayanan, Suresh
there is a global variable in spec that one can set
[7:05 PM] Dufresne, Eric
Good idea. This was also done quickly, maybe 0.5 s per step. You are probably right.
[7:06 PM] Narayanan, Suresh
just making 0.5 sec longer wont help
[7:06 PM] Narayanan, Suresh
we need to add a sleep after each point
[7:07 PM] Narayanan, Suresh
the neat way to do this would be to set all the images at once and then trigger each
frame with hardware, is a bit of work to do that
[7:09 PM] Narayanan, Suresh
Dufresne, Eric set sleep, it is a global var that is not set, default is zero, set that to
1 sec and then run the scan, at the end of the scan, make it 0. In either case, it should not affect padtimescan
[7:09 PM] Narayanan, Suresh
sorry, its "_sleep", the _ made all italics above
[7:09 PM] Narayanan, Suresh
set underscore_sleep=1
4/20/2022 setscans command
https://certif.com/spec_print/setscans.html
The setscan command allows one to sleep for a set number of seconds before counting.
| Quote: | Setting the sleep time to a nonzero value causes the scan macros to delay for that many
seconds before counting at each point. The value entered here is assigned to the _sleep global variable,
which can also be assigned to directly. |
slowing down diffractometer scans
Dufresne, Eric
Wed 4/20/2022 1:48 PM
To: 8ID-E user
Hellow,
we should be able to reduce the hanging problem in a scan by sleeping a bit before
taking an image.
You can type:
fourc8IDE> p _sleep=1
1
and this will set a sleep time of 1s before starting to count.
This is described in the setscans command:
https://certif.com/spec_print/setscans.html
Suresh thinks it will not affect on time series data collection.
If you don't want to take a chance, set it to 0 after the alignment is completed.
Eric |