RNA World (beta)

RNA World (beta) needs donations

The RNA World (beta) project needs donations to cover the BOINC project server (bandwidth and traffic) costs. Please support us and donate!

Run-time filter required


Advanced search

Message boards : News : Run-time filter required

Author Message
Profile Michael H.W. Weber
Project administrator
Project scientist
Send message
Joined: 25 May 09
Posts: 146
Credit: 4,562,976
RAC: 395
Message 130 - Posted: 23 Feb 2010, 5:42:23 UTC

I think we will need to design a filter on the server side that, based on the run time estimates, prevents such small WUs from being sent out to clients in the future. Although we have more than 200,000 WUs left, little is computed due to the extremely heavy MySQL load that is caused by the massive connections. So, another lesson learned, I would say. These small WUs in fact can easily run on some of the (mostly idle) server CPU cores and should also finish within a few days.
Michael.
____________
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.

Profile robertmilesProject donor
Send message
Joined: 22 Feb 10
Posts: 10
Credit: 483,628
RAC: 0
Message 134 - Posted: 24 Feb 2010, 2:01:13 UTC

Have you considered a new type of workunit that is essentially a script to run several (perhaps 10) of what is now the workunit size, one after the other, with code to write some kind of checkpoints in between?

Profile Michael H.W. Weber
Project administrator
Project scientist
Send message
Joined: 25 May 09
Posts: 146
Credit: 4,562,976
RAC: 395
Message 135 - Posted: 24 Feb 2010, 6:24:51 UTC - in response to Message 134.

Have you considered a new type of workunit that is essentially a script to run several (perhaps 10) of what is now the workunit size, one after the other, with code to write some kind of checkpoints in between?

I thought about it. Checkpointing is not required given these short run times, though. ;-)
Michael.
____________
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.

Profile robertmilesProject donor
Send message
Joined: 22 Feb 10
Posts: 10
Credit: 483,628
RAC: 0
Message 136 - Posted: 24 Feb 2010, 6:34:38 UTC - in response to Message 135.
Last modified: 24 Feb 2010, 6:44:25 UTC

Would the checkpoint files need to contain much more than the number of sections that have already been completed and therefore do not need to be repeated? That would allow you to build much longer workunits without doing much more than building a script to run the same program over and over, with a different set of input files each time. Also, if you select which old-style workunits to combine into one new-style workunit primarily by picking those which share the same largest input files, you could reduce the communications load on your server - assuming that there is enough sharing.

Also, I've seen a number of BOINC projects that reduce the communications load by sending most of the workunit files, and getting the output files, in compressed form (ZIP or similar) and having the workunits include steps to decompress the input files and compress the output files.

Profile Michael H.W. Weber
Project administrator
Project scientist
Send message
Joined: 25 May 09
Posts: 146
Credit: 4,562,976
RAC: 395
Message 137 - Posted: 24 Feb 2010, 10:44:07 UTC

First of all, our traffic is always in compressed format to reduce bandwidth. Secondly, as said before, we have thought about combinatorial WU archives. It is not advisable because it would require to significantly change the validator. That in turn makes not much sense as we will not send out archives with so many WUs that are this short again. We have learnt a "MySQL lesson" from this and that's what our current testing phase is meant for - although we are processing scientifically meaningful work throughout our "test" runs. ;-)
Michael.
____________
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.

Profile rilian
Avatar
Send message
Joined: 6 Feb 10
Posts: 3
Credit: 29,431
RAC: 0
Message 138 - Posted: 24 Feb 2010, 14:48:18 UTC - in response to Message 130.

there may be few solutions, do not know if they can be easily implemented in BOINC project, but

use different tables for different WU sizes. You would need to tweak all queries so WU lookup will be done depending on some (precalculated) WU size. With this, you can apply different mysql cache/keys settings to different WU tables

or use BOINC server setting to connect not faster than in X minutes (like in pirates@home - 1 hour, or LHC@home - 15 mins). So clients wont disturb server too often
____________
I crunch for Ukraine

Profile Michael H.W. Weber
Project administrator
Project scientist
Send message
Joined: 25 May 09
Posts: 146
Credit: 4,562,976
RAC: 395
Message 139 - Posted: 25 Feb 2010, 0:36:36 UTC

We actually know the sources of the issue and have solved it.
Michael.
____________
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.


Post to thread

Message boards : News : Run-time filter required