Over the last 90 min, the Validator backlog has dropped by over 100k. Looks like it's dropping by around 35k per hour (when the Validators were down completely, the rate of increase was roughly 12k per hour).
It's taken 16 hours since the Validators were restarted, but we're starting to get some significant falls in the backlog- and looking at my systems pendings, they've actually started to drop too.
*fingers crossed*

I looked a few hours ago and my 132 pending had dropped to 80 and now I've arrived home it's already further down to just 31.
Backlog down to 370k so it's all looking good now. My fears from yesterday have largely been allayed.

Err... backlog to validate - nil

---

Over the last 90 min, the Validator backlog has dropped by over 100k. Looks like it's dropping by around 35k per hour (when the Validators were down completely, the rate of increase was roughly 12k per hour).
It's taken 16 hours since the Validators were restarted, but we're starting to get some significant falls in the backlog- and looking at my systems pendings, they've actually started to drop too.
*fingers crossed*

I looked a few hours ago and my 132 pending had dropped to 80 and now I've arrived home it's already further down to just 31.
Backlog down to 370k so it's all looking good now. My fears from yesterday have largely been allayed.

Err... backlog to validate - nil

Not quite sure what's happening atm, but the validation backlog is up at 10k, but I don't think it's stopped working - just not quite keeping up for some reason.
The weirdness continues

---

Not quite sure what's happening atm, but the validation backlog is up at 10k, but I don't think it's stopped working - just not quite keeping up for some reason.
The weirdness continues

26k now.
The server has had issues for months now. I'm wondering if this is a symptom of those issues as they progressively worsen?

Someone there really needs to take a close look at the system logs to see just what is going on- WTF does the server keep crashing? And why is it now having so much trouble Validating work?
I'm thinking it's time to it to be replaced- a decade and a half is a very long time in computer hardware development.

---

Grant
Darwin NT

Someone there really needs to take a close look at the system logs to see just what is going on- WTF does the server keep crashing? And why is it now having so much trouble Validating work? I'm thinking it's time to it to be replaced- a decade and a half is a very long time in computer hardware development.

As i said a lot of time ago, we don't know if the server page is updated.
If not, the hw and (above all) the os/sw are very old.
Ubuntu 16....

---

Someone there really needs to take a close look at the system logs to see just what is going on- WTF does the server keep crashing? And why is it now having so much trouble Validating work? I'm thinking it's time to it to be replaced- a decade and a half is a very long time in computer hardware development.

As i said a lot of time ago, we don't know if the server page is updated.
If not, the hw and (above all) the os/sw are very old.
Ubuntu 16....

Just being 'old' isn't the worst thing in the world.
Being old and having failure issues every few weeks is a sign that if you don't fix this stuff, it's going to fail altogether.
Which will inevitably result in someone asking whether they can afford the time and trouble to update it all or whether they should go in another direction entirely.
I'm not sure how convinced I am they'll update the hw & sw to continue here tbh

In the meantime, I think all tasks have just run out, so we'll soon see if the validation backlog (currently 59k) will start to edge back down again

Edit: Just checked and no-one in my team has <any> tasks pending validation. Am I just lucky? Or is the backlog not real?

---

In the meantime, I think all tasks have just run out, so we'll soon see if the validation backlog (currently 59k) will start to edge back down again

Edit: Just checked and no-one in my team has <any> tasks pending validation. Am I just lucky? Or is the backlog not real?

Well, that changed quick.
Validation backlog back down to nil and 700k tasks have popped up
We live to crunch another day

---

Just being 'old' isn't the worst thing in the world.

Not for servers exposed costantly to the internet.
Security fixes, bugfix, support are fundamental (if you care about the project).
There is also the performance factor: do you see the difference of a recente file system (ZFS 2.5) and old one (0.7 - if true)?

Which will inevitably result in someone asking whether they can afford the time and trouble to update it all or whether they should go in another direction entirely.
I'm not sure how convinced I am they'll update the hw & sw to continue here tbh

+1

---

Validation backlog back down to nil and 700k tasks have popped up
We live to crunch another day

And another day with over 46k wus pending validation... :-(

---

Validation backlog back down to nil and 700k tasks have popped up
We live to crunch another day

And another day with over 46k wus pending validation... :-(

Yes, and now 88k
But I just looked through my team's tasks again and it's the same as a few days ago.
A high figure showing on the server status page, but none of my team have <any> tasks awaiting validation.

Is this 2 coincidences in a row? I'm certainly confused.

---

Validation backlog back down to nil and 700k tasks have popped up
We live to crunch another day

And another day with over 46k wus pending validation... :-(

Yes, and now 88k
But I just looked through my team's tasks again and it's the same as a few days ago.
A high figure showing on the server status page, but none of my team have <any> tasks awaiting validation.

Is this 2 coincidences in a row? I'm certainly confused.

You are the lucky one.

The problem appears to have started for me at 02:00 GMT, for the next hour I have about 50% pending and since then I’ve only had 5 validated out of nearly 100 completed.

Validation backlog back down to nil and 700k tasks have popped up
We live to crunch another day

And another day with over 46k wus pending validation... :-(

Yes, and now 88k
But I just looked through my team's tasks again and it's the same as a few days ago.
A high figure showing on the server status page, but none of my team have <any> tasks awaiting validation.

Is this 2 coincidences in a row? I'm certainly confused.

Could it mean that the validator processes for some operating systems correctly, but not for some others?

Validation backlog back down to nil and 700k tasks have popped up
We live to crunch another day

And another day with over 46k wus pending validation... :-(

Yes, and now 88k
But I just looked through my team's tasks again and it's the same as a few days ago.
A high figure showing on the server status page, but none of my team have <any> tasks awaiting validation.

Is this 2 coincidences in a row? I'm certainly confused.

You are the lucky one.

The problem appears to have started for me at 02:00 GMT, for the next hour I have about 50% pending and since then I’ve only had 5 validated out of nearly 100 completed.

I've just looked at your pending tasks and I'm amazed at the backlog.
I just returned 8 tasks and, while they didn't validate immediately, it only took 20-30 minutes, not 20hrs!
I'm almost apologetic about my success. I've done nothing to warrant it, certainly.
Definitely some strange and inexplicable business going on.

---