WUs that slow way down at the end

[bikeaddict]

Occasionally I see WUs that start with PPD estimate around 400-500K and then suddenly drop to 20-30K for the last few percent and it takes many hours to finish.

Project: 12422 (Run 66, Clone 0, Gen 23) is an example that I have currently running.

Here are the time intervals by percent:

Code: Select all

18:03:01:WU00:FS01:0xa8:Steps: first=115000000 total=120000000
18:03:05:WU00:FS01:0xa8:Completed 1 out of 5000000 steps (0%)
18:13:24:WU00:FS01:0xa8:Completed 50000 out of 5000000 steps (1%)
18:23:44:WU00:FS01:0xa8:Completed 100000 out of 5000000 steps (2%)
18:34:07:WU00:FS01:0xa8:Completed 150000 out of 5000000 steps (3%)
18:44:27:WU00:FS01:0xa8:Completed 200000 out of 5000000 steps (4%)
18:54:48:WU00:FS01:0xa8:Completed 250000 out of 5000000 steps (5%)
19:05:10:WU00:FS01:0xa8:Completed 300000 out of 5000000 steps (6%)
19:15:31:WU00:FS01:0xa8:Completed 350000 out of 5000000 steps (7%)
19:25:51:WU00:FS01:0xa8:Completed 400000 out of 5000000 steps (8%)
19:36:12:WU00:FS01:0xa8:Completed 450000 out of 5000000 steps (9%)
19:46:33:WU00:FS01:0xa8:Completed 500000 out of 5000000 steps (10%)
19:56:54:WU00:FS01:0xa8:Completed 550000 out of 5000000 steps (11%)
20:07:15:WU00:FS01:0xa8:Completed 600000 out of 5000000 steps (12%)
20:17:36:WU00:FS01:0xa8:Completed 650000 out of 5000000 steps (13%)
20:27:57:WU00:FS01:0xa8:Completed 700000 out of 5000000 steps (14%)
20:38:18:WU00:FS01:0xa8:Completed 750000 out of 5000000 steps (15%)
20:48:40:WU00:FS01:0xa8:Completed 800000 out of 5000000 steps (16%)
20:59:00:WU00:FS01:0xa8:Completed 850000 out of 5000000 steps (17%)
21:09:15:WU00:FS01:0xa8:Completed 900000 out of 5000000 steps (18%)
21:19:32:WU00:FS01:0xa8:Completed 950000 out of 5000000 steps (19%)
21:29:51:WU00:FS01:0xa8:Completed 1000000 out of 5000000 steps (20%)
21:40:10:WU00:FS01:0xa8:Completed 1050000 out of 5000000 steps (21%)
21:50:33:WU00:FS01:0xa8:Completed 1100000 out of 5000000 steps (22%)
22:00:54:WU00:FS01:0xa8:Completed 1150000 out of 5000000 steps (23%)
22:11:15:WU00:FS01:0xa8:Completed 1200000 out of 5000000 steps (24%)
22:21:36:WU00:FS01:0xa8:Completed 1250000 out of 5000000 steps (25%)
22:31:57:WU00:FS01:0xa8:Completed 1300000 out of 5000000 steps (26%)
22:42:19:WU00:FS01:0xa8:Completed 1350000 out of 5000000 steps (27%)
22:52:34:WU00:FS01:0xa8:Completed 1400000 out of 5000000 steps (28%)
23:02:47:WU00:FS01:0xa8:Completed 1450000 out of 5000000 steps (29%)
23:13:00:WU00:FS01:0xa8:Completed 1500000 out of 5000000 steps (30%)
23:23:16:WU00:FS01:0xa8:Completed 1550000 out of 5000000 steps (31%)
23:33:34:WU00:FS01:0xa8:Completed 1600000 out of 5000000 steps (32%)
23:43:51:WU00:FS01:0xa8:Completed 1650000 out of 5000000 steps (33%)
23:54:12:WU00:FS01:0xa8:Completed 1700000 out of 5000000 steps (34%)
00:04:34:WU00:FS01:0xa8:Completed 1750000 out of 5000000 steps (35%)
00:14:55:WU00:FS01:0xa8:Completed 1800000 out of 5000000 steps (36%)
00:25:17:WU00:FS01:0xa8:Completed 1850000 out of 5000000 steps (37%)
00:35:38:WU00:FS01:0xa8:Completed 1900000 out of 5000000 steps (38%)
00:46:02:WU00:FS01:0xa8:Completed 1950000 out of 5000000 steps (39%)
00:56:29:WU00:FS01:0xa8:Completed 2000000 out of 5000000 steps (40%)
01:06:55:WU00:FS01:0xa8:Completed 2050000 out of 5000000 steps (41%)
01:17:20:WU00:FS01:0xa8:Completed 2100000 out of 5000000 steps (42%)
01:27:45:WU00:FS01:0xa8:Completed 2150000 out of 5000000 steps (43%)
01:38:03:WU00:FS01:0xa8:Completed 2200000 out of 5000000 steps (44%)
01:48:22:WU00:FS01:0xa8:Completed 2250000 out of 5000000 steps (45%)
01:58:43:WU00:FS01:0xa8:Completed 2300000 out of 5000000 steps (46%)
02:09:03:WU00:FS01:0xa8:Completed 2350000 out of 5000000 steps (47%)
02:19:24:WU00:FS01:0xa8:Completed 2400000 out of 5000000 steps (48%)
02:29:45:WU00:FS01:0xa8:Completed 2450000 out of 5000000 steps (49%)
02:40:07:WU00:FS01:0xa8:Completed 2500000 out of 5000000 steps (50%)
02:50:28:WU00:FS01:0xa8:Completed 2550000 out of 5000000 steps (51%)
03:00:49:WU00:FS01:0xa8:Completed 2600000 out of 5000000 steps (52%)
03:11:10:WU00:FS01:0xa8:Completed 2650000 out of 5000000 steps (53%)
03:21:32:WU00:FS01:0xa8:Completed 2700000 out of 5000000 steps (54%)
03:31:52:WU00:FS01:0xa8:Completed 2750000 out of 5000000 steps (55%)
03:42:13:WU00:FS01:0xa8:Completed 2800000 out of 5000000 steps (56%)
03:52:35:WU00:FS01:0xa8:Completed 2850000 out of 5000000 steps (57%)
04:02:56:WU00:FS01:0xa8:Completed 2900000 out of 5000000 steps (58%)
04:13:29:WU00:FS01:0xa8:Completed 2950000 out of 5000000 steps (59%)
04:23:50:WU00:FS01:0xa8:Completed 3000000 out of 5000000 steps (60%)
04:34:11:WU00:FS01:0xa8:Completed 3050000 out of 5000000 steps (61%)
04:44:31:WU00:FS01:0xa8:Completed 3100000 out of 5000000 steps (62%)
04:54:52:WU00:FS01:0xa8:Completed 3150000 out of 5000000 steps (63%)
05:05:14:WU00:FS01:0xa8:Completed 3200000 out of 5000000 steps (64%)
05:15:35:WU00:FS01:0xa8:Completed 3250000 out of 5000000 steps (65%)
05:25:56:WU00:FS01:0xa8:Completed 3300000 out of 5000000 steps (66%)
05:36:18:WU00:FS01:0xa8:Completed 3350000 out of 5000000 steps (67%)
05:46:37:WU00:FS01:0xa8:Completed 3400000 out of 5000000 steps (68%)
05:56:56:WU00:FS01:0xa8:Completed 3450000 out of 5000000 steps (69%)
06:07:17:WU00:FS01:0xa8:Completed 3500000 out of 5000000 steps (70%)
06:17:38:WU00:FS01:0xa8:Completed 3550000 out of 5000000 steps (71%)
06:28:00:WU00:FS01:0xa8:Completed 3600000 out of 5000000 steps (72%)
06:38:21:WU00:FS01:0xa8:Completed 3650000 out of 5000000 steps (73%)
06:48:43:WU00:FS01:0xa8:Completed 3700000 out of 5000000 steps (74%)
06:59:04:WU00:FS01:0xa8:Completed 3750000 out of 5000000 steps (75%)
07:09:25:WU00:FS01:0xa8:Completed 3800000 out of 5000000 steps (76%)
07:19:46:WU00:FS01:0xa8:Completed 3850000 out of 5000000 steps (77%)
07:30:08:WU00:FS01:0xa8:Completed 3900000 out of 5000000 steps (78%)
07:40:29:WU00:FS01:0xa8:Completed 3950000 out of 5000000 steps (79%)
07:50:50:WU00:FS01:0xa8:Completed 4000000 out of 5000000 steps (80%)
08:01:11:WU00:FS01:0xa8:Completed 4050000 out of 5000000 steps (81%)
08:11:34:WU00:FS01:0xa8:Completed 4100000 out of 5000000 steps (82%)
08:21:55:WU00:FS01:0xa8:Completed 4150000 out of 5000000 steps (83%)
08:32:16:WU00:FS01:0xa8:Completed 4200000 out of 5000000 steps (84%)
08:42:37:WU00:FS01:0xa8:Completed 4250000 out of 5000000 steps (85%)
08:52:58:WU00:FS01:0xa8:Completed 4300000 out of 5000000 steps (86%)
09:03:19:WU00:FS01:0xa8:Completed 4350000 out of 5000000 steps (87%)
09:13:35:WU00:FS01:0xa8:Completed 4400000 out of 5000000 steps (88%)
09:23:54:WU00:FS01:0xa8:Completed 4450000 out of 5000000 steps (89%)
09:34:15:WU00:FS01:0xa8:Completed 4500000 out of 5000000 steps (90%)
09:44:36:WU00:FS01:0xa8:Completed 4550000 out of 5000000 steps (91%)
09:55:00:WU00:FS01:0xa8:Completed 4600000 out of 5000000 steps (92%)
11:59:54:WU00:FS01:0xa8:Completed 4650000 out of 5000000 steps (93%)
14:04:52:WU00:FS01:0xa8:Completed 4700000 out of 5000000 steps (94%)
16:09:46:WU00:FS01:0xa8:Completed 4750000 out of 5000000 steps (95%)

About 10 minutes per percent then jumps to a couple hours after 92% and shows completion estimates of 8+ hours. This only happens on my old 14-core or 16-core Xeons but not on a modern 16-core Ryzen.

Just wanted to check if this is intended behavior or if there is a problem with these WUs.

[Joe_H]

I have seen slowdowns in the beginning, middle and end of WUs. Barring a slowdown from the CPU being throttled for thermal or other reasons, usually this indicates a shift in conformation of the protein being modeled that takes longer to calculate interactions.