SPEC data file is in wrong directory now #408

prjemian · 2020-09-16T00:38:35Z

SPEC data is in the wrong directory. Scan log says /share1/USAXS_data/2020-09/09_15_Qu/09_15_Qu.dat but file is not there. It's in the parent directory (the current working directory). Need to fix that ASAP.

The text was updated successfully, but these errors were encountered:

jilavsky · 2020-09-16T02:07:25Z

I do not think this is fixed yet, last data set on screen is SAXS S16 Qu09, we collected 100 SAXS scans in the mean time (I was fixing my computer).
Checked we are writing in 2020-09/09_15_Qu/09_15_Qu.dat while our intention was to write in 2020-09/09_15_Qu.dat. ???

Anyway, something to look on Thursday if you have time.

prjemian · 2020-09-16T02:07:52Z

scan logs are not updating, last entries:

index	title	type	scan	file	started	ended
1237	Qu9	SAXS	16	/share1/USAXS_data/2020-09/09_15_Qu/09_15_Qu.dat	2020-09-15 20:14:28	2020-09-15 20:14:44
1236	AirBlank	SAXS	15	/share1/USAXS_data/2020-09/09_15_Qu/09_15_Qu.dat	2020-09-15 20:13:56	2020-09-15 20:14:13
1235	Qu8	SAXS	14	/share1/USAXS_data/2020-09/09_15_Qu/09_15_Qu.dat	2020-09-15 20:13:24	2020-09-15 20:13:40

but livedata page says

file	scan	date time
/share1/USAXS_data/2020-09/09_15_Qu/09_15_Qu.dat	scan #107	2020-09-15 21:04:57.177599

prjemian · 2020-09-16T02:10:27Z

Our intention is this table:

purpose	folder	PV
user data folder base	CWD/MM_DD_USER	9idcLAX:userDir
SPEC data file	CWD/MM_DD_USER/MM_DD_USER.dat	9idcLAX:USAXS:specFile
AD folder - SAXS	CWD/MM_DD_USER/MM_DD_USER_saxs/
folder - USAXS	CWD/MM_DD_USER/MM_DD_USER_usaxs/
AD folder - WAXS	CWD/MM_DD_USER/MM_DD_USER_waxs/

jilavsky · 2020-09-16T02:40:37Z

Confusing, Did you change anything? WAXS graphs are coming out.

prjemian · 2020-09-16T02:46:59Z

newUser?

…

On Tue, Sep 15, 2020, 9:40 PM Jan Ilavsky ***@***.***> wrote: Confusing, Did you change anything? WAXS graphs are coming out. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMHPJS23VB5FQ3Q7PWLSGAQTDANCNFSM4RN7QW4Q> .

jilavsky · 2020-09-16T15:27:34Z

Live data page shows only SAXS and WAXS when running from spec, which is what we need to do for now for more complicated experiments which talk directly to motors.

Note, that in spec we will have different location of spec file than in BS, I am not sure how to move file in spec. I will ask jeff if he can help me to make a change to follow BS logic. For now, this is what spec collects:

purpose	folder	PV
user data folder base	CWD/MM_DD_USER	9idcLAX:userDir
SPEC data file	CWD/MM_DD_USER.dat	9idcLAX:USAXS:specFile
AD folder - SAXS	CWD/MM_DD_USER/MM_DD_USER_saxs/
folder - USAXS	CWD/MM_DD_USER/MM_DD_USER_usaxs/
AD folder - WAXS	CWD/MM_DD_USER/MM_DD_USER_waxs/

jilavsky · 2020-09-16T15:43:39Z

Interesting, plots SAXS and WAXS data. But Flyscan data are in the logs.
We really created a mess...

jilavsky · 2020-09-16T17:16:36Z

Looks like Jeff will modify the code in spec to arrange data same as BlueSky. I will test tomorrow.

prjemian · 2020-09-16T17:43:02Z

Meanwhile, the scanlog is logging the Fly scans now and livedata (known problem) is not plotting them yet.

index	title	type	scan	file	started	ended
1638	splA_17_saxs	SAXS	114	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:40:51	scanning
1637	splA_17	FlyScan	113	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:38:14	2020-09-16 12:40:25
1636	splA_16_waxs	WAXS	112	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:35:52	2020-09-16 12:36:53
1635	splA_16_saxs	SAXS	110	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:34:17	2020-09-16 12:35:18
1634	splA_16	FlyScan	109	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:31:39	2020-09-16 12:33:51
1633	splA_15_waxs	WAXS	108	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:29:17	2020-09-16 12:30:18
1632	splA_15_saxs	SAXS	107	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:27:57	2020-09-16 12:28:58
1631	splA_15	FlyScan	106	/share1/USAXS_data/2020-09/09_16_Krzysko.dat	2020-09-16 12:25:20	2020-09-16 12:27:31

prjemian · 2020-09-16T18:10:41Z

In these logs, the SPEC data file 09_16_Krzysko.dat is reported to be stored in directory /share1/USAXS_data/2020-09. (This is the location for SPEC data files under the old plan.) Yes, it is found there now:

usaxs@usaxscontrol .../USAXS_data/2020-09 $ ll /share1/USAXS_data/2020-09/09_16_Krzysko.dat
-rw-rw-r-- 1 usaxs 417690 Sep 16 13:10 /share1/USAXS_data/2020-09/09_16_Krzysko.dat

prjemian · 2020-09-16T18:13:23Z

The SPEC data file reports FlyScan NeXus/HDF5 files are stored in subdirectories according to the new plan:

usaxs@usaxscontrol .../USAXS_data/2020-09 $ grep h5 09_16_Krzysko.dat | tail -6
#C Wed Sep 16 12:38:13 2020.  FlyScan file name = ./09_16_Krzysko/09_16_Krzysko_usaxs/splA_17_0025.h5.
#C Wed Sep 16 12:45:11 2020.  FlyScan file name = ./09_16_Krzysko/09_16_Krzysko_usaxs/splA_18_0026.h5.
#C Wed Sep 16 12:51:30 2020.  FlyScan file name = ./09_16_Krzysko/09_16_Krzysko_usaxs/splA_19_0027.h5.
#C Wed Sep 16 12:58:04 2020.  FlyScan file name = ./09_16_Krzysko/09_16_Krzysko_usaxs/splA_20_0028.h5.
#C Wed Sep 16 13:05:02 2020.  FlyScan file name = ./09_16_Krzysko/09_16_Krzysko_usaxs/splA_21_0029.h5.
#C Wed Sep 16 13:11:20 2020.  FlyScan file name = ./09_16_Krzysko/09_16_Krzysko_usaxs/splA_22_0030.h5.

prjemian · 2020-09-16T18:23:17Z

These are the most recent files in that subdirectory:

usaxs@usaxscontrol .../USAXS_data/2020-09 $ ls -lat 09_16_Krzysko/09_16_Krzysko_usaxs | head -10
total 13656
-rw-rw-r-- 1 usaxs usaxs 446938 Sep 16 13:13 splA_22_0030.h5
drwxrwxr-x 2 usaxs usaxs   4096 Sep 16 13:11 .
-rw-rw-r-- 1 usaxs usaxs  27642 Sep 16 13:11 splA_22_0030.jpg
-rw-rw-r-- 1 usaxs usaxs 446986 Sep 16 13:07 splA_21_0029.h5
-rw-rw-r-- 1 usaxs usaxs  27709 Sep 16 13:05 splA_21_0029.jpg
-rw-rw-r-- 1 usaxs usaxs 446954 Sep 16 13:00 splA_20_0028.h5
-rw-rw-r-- 1 usaxs usaxs  27625 Sep 16 12:58 splA_20_0028.jpg
-rw-rw-r-- 1 usaxs usaxs 446970 Sep 16 12:53 splA_19_0027.h5
-rw-rw-r-- 1 usaxs usaxs  27629 Sep 16 12:51 splA_19_0027.jpg

The file splA_22_0030.h5 is most recent. Good, data is found where it is stated to be.

These files are written when SPEC is used. We can tell that by looking at the first lines of the SPEC data file:

SPEC writes the file:

#F 09_16_Krzysko.dat
#E 1600264797
#D Wed Sep 16 08:59:57 2020
#C Krzysko  User = usaxs

Bluesky writes the file:

#F 09_15_Pete.dat
#E 1600194860
#D Tue Sep 15 13:34:20 2020
#C Bluesky  user = usaxs  host = usaxscontrol.xray.aps.anl.gov

The ONLY thing that SPEC needs now is to write the data file in the CWD/MM_DD_USER/ subdirectory (instead of in CWD/).

jilavsky · 2020-09-16T18:26:43Z

Assuming Jeff makes the code change, I will test this:
"The ONLY thing that SPEC needs now is to write the data file in the CWD/MM_DD_USER/ subdirectory (instead of in CWD/)."
tomorrow. I do not want to do this while AJ is collecting data.

prjemian · 2020-09-16T20:31:13Z

We also need to (re)confirm that Bluesky uses and reports the new directory scheme for step scans and fly scans.

jilavsky · 2020-09-17T15:35:13Z

I have fixed the file location iss spec. There are few wrong attempts. Will send some details using teams.

prjemian · 2020-09-17T15:45:21Z

Much better too document those details right here.

…

On Thu, Sep 17, 2020, 10:35 AM Jan Ilavsky ***@***.***> wrote: I have fixed the file location iss spec. There are few wrong attempts. Will send some details using teams. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMFTGY6HBFGE3EPTZCLSGIUEFANCNFSM4RN7QW4Q> .

jilavsky · 2020-09-17T15:52:59Z

Not if user names are involved.

jilavsky · 2020-09-17T15:58:57Z

I am confused about record in scanlog for live data page. SAXS and WAXS are wrong now. FLyscan is correct. How is the log filled, what does it use? Obviously, I broke something by fixing the paths and do not know what to fix now.
Which PVs are used to create scanlog?

jilavsky · 2020-09-17T16:18:22Z

I think I got it. it is using 9idcLAX:specFile and that needs to contain path to spec file... I can fix that.

prjemian · 2020-09-18T17:47:44Z

Bluesky-directed FlyScan now plots correctly in livedata.

prjemian · 2020-09-18T17:56:40Z

Once we are confident that Bluesky's USAXS step scans will plot properly with livedata, this issue can be closed (again).

prjemian · 2020-09-18T17:58:54Z

I believe 9idcLAX:specFile could be aliased to 9idcLAX:dataFile so that programmers of our various code bases stop getting confused.

prjemian · 2020-09-18T17:59:48Z

Basically, getting spec out of our PV names unless the PV is truly specific to SPEC only.

prjemian · 2020-09-18T18:01:01Z

Last two comments belong in separate issue if you agree.

prjemian · 2020-09-21T15:57:57Z

Scans in these directories have SPEC, Bluesky FlyScan, and Bluesky step scan variations, respectively. Same sequence for each.

usaxs@usaxscontrol .../USAXS_data/2020-09 $ ll 09_17_AniKulkarni*/09_17_AniKulkarni*.dat
-rw-rw-r-- 1 usaxs 113475 Sep 17 13:24 09_17_AniKulkarni/09_17_AniKulkarni.dat
-rw-rw-r-- 1 usaxs  94593 Sep 17 11:38 09_17_AniKulkarniBS/09_17_AniKulkarniBS.dat
-rw-rw-r-- 1 usaxs 478674 Sep 17 12:31 09_17_AniKulkarniSTEP/09_17_AniKulkarniSTEP.dat

prjemian · 2020-09-21T16:09:10Z

I'll look at the .dat files and try to make livedata plots from each. That should satisfy this issue.

jilavsky · 2020-11-12T00:57:45Z

I think this was fixed. We need to test this. I think spec was changed to follow BS template - "spec" file inside user folder.
I will power up tomorrow all detectors and hardware, so we can test this in demo mode.

jilavsky · 2020-11-17T23:33:49Z

Seems to be fixed

prjemian added bug Something isn't working high priority more urgent labels Sep 16, 2020

prjemian added this to the 2020-11 operations milestone Sep 16, 2020

prjemian self-assigned this Sep 16, 2020

prjemian added a commit that referenced this issue Sep 16, 2020

fixes #408

a76277b

prjemian mentioned this issue Sep 16, 2020

specify directory for SPEC data file #410

Merged

prjemian modified the milestones: 2020-11 operations, 2020-10 operations Sep 16, 2020

prjemian mentioned this issue Sep 16, 2020

Did #388 break live data plots? #409

Closed

prjemian closed this as completed in #410 Sep 16, 2020

jilavsky reopened this Sep 16, 2020

prjemian modified the milestones: 2020-10 operations, 2020-11 operations Nov 4, 2020

jilavsky closed this as completed Nov 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPEC data file is in wrong directory now #408

SPEC data file is in wrong directory now #408

prjemian commented Sep 16, 2020

jilavsky commented Sep 16, 2020

prjemian commented Sep 16, 2020

prjemian commented Sep 16, 2020

jilavsky commented Sep 16, 2020 •

edited

Loading

prjemian commented Sep 16, 2020 via email

jilavsky commented Sep 16, 2020

jilavsky commented Sep 16, 2020

jilavsky commented Sep 16, 2020

prjemian commented Sep 16, 2020

prjemian commented Sep 16, 2020 •

edited

Loading

prjemian commented Sep 16, 2020

prjemian commented Sep 16, 2020

jilavsky commented Sep 16, 2020

prjemian commented Sep 16, 2020

jilavsky commented Sep 17, 2020

prjemian commented Sep 17, 2020 via email

jilavsky commented Sep 17, 2020

jilavsky commented Sep 17, 2020 •

edited

Loading

jilavsky commented Sep 17, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 21, 2020

prjemian commented Sep 21, 2020

jilavsky commented Nov 12, 2020

jilavsky commented Nov 17, 2020

SPEC data file is in wrong directory now #408

SPEC data file is in wrong directory now #408

Comments

prjemian commented Sep 16, 2020

jilavsky commented Sep 16, 2020

prjemian commented Sep 16, 2020

prjemian commented Sep 16, 2020

jilavsky commented Sep 16, 2020 • edited Loading

prjemian commented Sep 16, 2020 via email

jilavsky commented Sep 16, 2020

jilavsky commented Sep 16, 2020

jilavsky commented Sep 16, 2020

prjemian commented Sep 16, 2020

prjemian commented Sep 16, 2020 • edited Loading

prjemian commented Sep 16, 2020

prjemian commented Sep 16, 2020

jilavsky commented Sep 16, 2020

prjemian commented Sep 16, 2020

jilavsky commented Sep 17, 2020

prjemian commented Sep 17, 2020 via email

jilavsky commented Sep 17, 2020

jilavsky commented Sep 17, 2020 • edited Loading

jilavsky commented Sep 17, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 18, 2020

prjemian commented Sep 21, 2020

prjemian commented Sep 21, 2020

jilavsky commented Nov 12, 2020

jilavsky commented Nov 17, 2020

jilavsky commented Sep 16, 2020 •

edited

Loading

prjemian commented Sep 16, 2020 •

edited

Loading

jilavsky commented Sep 17, 2020 •

edited

Loading