RNA World (beta)

RNA World (beta) needs donations

The RNA World (beta) project needs donations to cover the BOINC project server (bandwidth and traffic) costs. Please support us and donate!

Run-time filter required

Message boards : News : Run-time filter required
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Michael H.W. Weber
Project administrator
Project scientist

Send message
Joined: 25 May 09
Posts: 153
Credit: 4,711,930
RAC: 2
Message 130 - Posted: 23 Feb 2010, 5:42:23 UTC

I think we will need to design a filter on the server side that, based on the run time estimates, prevents such small WUs from being sent out to clients in the future. Although we have more than 200,000 WUs left, little is computed due to the extremely heavy MySQL load that is caused by the massive connections. So, another lesson learned, I would say. These small WUs in fact can easily run on some of the (mostly idle) server CPU cores and should also finish within a few days.
Michael.
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.
ID: 130 · Report as offensive     Reply Quote
Profile robertmiles

Send message
Joined: 22 Feb 10
Posts: 11
Credit: 483,628
RAC: 0
Message 134 - Posted: 24 Feb 2010, 2:01:13 UTC

Have you considered a new type of workunit that is essentially a script to run several (perhaps 10) of what is now the workunit size, one after the other, with code to write some kind of checkpoints in between?
ID: 134 · Report as offensive     Reply Quote
Profile Michael H.W. Weber
Project administrator
Project scientist

Send message
Joined: 25 May 09
Posts: 153
Credit: 4,711,930
RAC: 2
Message 135 - Posted: 24 Feb 2010, 6:24:51 UTC - in response to Message 134.  

Have you considered a new type of workunit that is essentially a script to run several (perhaps 10) of what is now the workunit size, one after the other, with code to write some kind of checkpoints in between?

I thought about it. Checkpointing is not required given these short run times, though. ;-)
Michael.
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.
ID: 135 · Report as offensive     Reply Quote
Profile robertmiles

Send message
Joined: 22 Feb 10
Posts: 11
Credit: 483,628
RAC: 0
Message 136 - Posted: 24 Feb 2010, 6:34:38 UTC - in response to Message 135.  
Last modified: 24 Feb 2010, 6:44:25 UTC

Would the checkpoint files need to contain much more than the number of sections that have already been completed and therefore do not need to be repeated? That would allow you to build much longer workunits without doing much more than building a script to run the same program over and over, with a different set of input files each time. Also, if you select which old-style workunits to combine into one new-style workunit primarily by picking those which share the same largest input files, you could reduce the communications load on your server - assuming that there is enough sharing.

Also, I've seen a number of BOINC projects that reduce the communications load by sending most of the workunit files, and getting the output files, in compressed form (ZIP or similar) and having the workunits include steps to decompress the input files and compress the output files.
ID: 136 · Report as offensive     Reply Quote
Profile Michael H.W. Weber
Project administrator
Project scientist

Send message
Joined: 25 May 09
Posts: 153
Credit: 4,711,930
RAC: 2
Message 137 - Posted: 24 Feb 2010, 10:44:07 UTC

First of all, our traffic is always in compressed format to reduce bandwidth. Secondly, as said before, we have thought about combinatorial WU archives. It is not advisable because it would require to significantly change the validator. That in turn makes not much sense as we will not send out archives with so many WUs that are this short again. We have learnt a "MySQL lesson" from this and that's what our current testing phase is meant for - although we are processing scientifically meaningful work throughout our "test" runs. ;-)
Michael.
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.
ID: 137 · Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 6 Feb 10
Posts: 3
Credit: 29,431
RAC: 0
Message 138 - Posted: 24 Feb 2010, 14:48:18 UTC - in response to Message 130.  

there may be few solutions, do not know if they can be easily implemented in BOINC project, but

use different tables for different WU sizes. You would need to tweak all queries so WU lookup will be done depending on some (precalculated) WU size. With this, you can apply different mysql cache/keys settings to different WU tables

or use BOINC server setting to connect not faster than in X minutes (like in pirates@home - 1 hour, or LHC@home - 15 mins). So clients wont disturb server too often
I crunch for Ukraine
ID: 138 · Report as offensive     Reply Quote
Profile Michael H.W. Weber
Project administrator
Project scientist

Send message
Joined: 25 May 09
Posts: 153
Credit: 4,711,930
RAC: 2
Message 139 - Posted: 25 Feb 2010, 0:36:36 UTC

We actually know the sources of the issue and have solved it.
Michael.
Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.
ID: 139 · Report as offensive     Reply Quote

Message boards : News : Run-time filter required