Python - Generator within another generator

Question

I have to use the output of a generator in another generator.

Below is the code -

Here Generator 2 is called within generator 1 and the final output is received from generator 2.

I am trying to use something like below , can anyone suggest a solution ?

def sub_gen(data) : for r in res_gen() : yield each train_datagen(r)

Generator 1

def res_gen (num_threads = 4 ):
    while (True) :
      for i in range(0,len(file_list),num_threads):
        # use multi-process to speed up
        res = []
        p = Pool(num_threads)
        patch = p.map(gen_patches,file_list[i:min(i+num_threads,len(file_list))])
        #patch = p.map(gen_patches,file_list[i:i+num_threads])
        for x in patch:
            res += x
        res1 = np.array(res)
        res1 = res1.reshape((res1.shape[0],res1.shape[1],res1.shape[2],1))
        res1 = res1.astype('float32')/255.0
        yield res1

Generator 2

def train_datagen(res1, batch_size=4):
    indices = list(range(res1.shape[0]))
    while(True):
        np.random.shuffle(indices)    # shuffle
        for i in range(0, len(indices), batch_size):
            ge_batch_y = res1[indices[i:i+batch_size]]
            noise =  np.random.normal(0, sigma/255.0, ge_batch_y.shape)   
            #noise =  K.random_normal(ge_batch_y.shape, mean=0, stddev=sigma/255.0)
            ge_batch_x = ge_batch_y + noise  # input image = clean image + noise
            yield ge_batch_x, ge_batch_y

Answer 1

I'm pretty sure the only issue in your short sub_gen generator is that you've written yield each instead of yield from. The latter expects an iterable value after it (often another generator), and it yields each value just like an explicit for loop

So I think your code should be:

def sub_gen(data) :
  for r in res_gen() :
      yield from train_datagen(r)

Lets test this with much simpler generator functions:

def foo():
    yield [1, 2]
    yield [3, 4]

def bar(iterable):
    for x in iterable:
        yield 10+x
        yield 20+x

def baz():
    for iterable in foo():
        yield from bar(iterable)

for value in baz():   # use the top-level generator!
    print(value)      # prints 11, 21, 12, 22, 13, 23, 14, 24 each on its own line