Search code examples
python-3.xbuildbot

Buildbot: trigger virtual builders problem

I wish to trigger builders dynamically (depending of list of supported platforms of incoming project). There are virtual builders for this purposes. It's really works but there is a critical problem: when I trigger more virtual builds then number of free workers, other builds are skipped.

On image below 4 builds triggered with one active worker enter image description here

On success we can see only two executed builds. Other are lost (not executed at all)enter image description here

Again, for 1 free (active, not paused) worker only 2 (of 4) triggered builds was executed. For N free workers it will be N+1 executed builds. Reproduction 100%.

The code is pretty clear

class MultiplePlatformsBuilder(Trigger):
    def __init__(self, *args, **kwargs):
        ...

    def getSchedulersAndProperties(self):
        sch = self.schedulerNames[0]
        triggered_schedulers = []
        counter = 0
        # trigger scheduler for each requested platform
        platforms_list = self.set_properties.get('buildable_platforms_list')
        for pl in platforms_list:
            counter += 1
            props = dict()
            props["virtual_builder_name"] = "virtual-" + pl
            props["virtual_builder_description"] = "This is dynamic build " + str(counter)
            triggered_schedulers.append([sch, props])

        return triggered_schedulers

# trigger configuration    
c['schedulers'].append(schedulers.Triggerable(name='dynamic_scheduler', builderNames=['factory1']))
factory.addStep(MultiplePlatformsBuilder(
    schedulerNames=['dynamic_scheduler'],
    set_properties={
        'buildable_platforms_list': ["platform1", "platform2", "platform3", "platform4"]
        # platforms_list_var: Property(platforms_list_var),
    },
))
Solution

  • The problem took me several days. It turned out this is not a bug but feature named Collapsing Build Requests

    All build requests with the same codebases will collapse if there are no free workers.

    All I need is just disable this feature:

    c['collapseRequests'] = False

    I'm sure this should be mentioned in context of virtual builders because they all have same codebases but shouldn't be collapsed - this makes no sense