Concatenate multiple IterableDataset instances end-to-end.
Iteration walks the chain in registration order: every element from
the first child is yielded first, then every element from the second,
and so on. The iterable equivalent of ConcatDataset for
map-style datasets.
Each child must be an IterableDataset; map-style datasets
must be iterated through a sampler-driven DataLoader
instead.
Parameters
datasetslist of IterableDatasetIterable datasets to chain in order. Each element must be an
IterableDataset instance — passing a Dataset
raises TypeError at construction time.Notes
Unlike ConcatDataset, no random access is possible: total
length is unknown a priori and the chain may be infinite if any
child is. Useful for stitching together streaming shards
(e.g. multiple files in a sharded log) into one logical stream.
Examples
>>> from lucid.utils.data import ChainDataset, IterableDataset
>>> class CountUp(IterableDataset):
... def __init__(self, n):
... self.n = n
... def __iter__(self):
... yield from range(self.n)
>>> a = CountUp(3) # yields 0, 1, 2
>>> b = CountUp(2) # yields 0, 1
>>> chain = ChainDataset([a, b])
>>> list(chain)
[0, 1, 2, 0, 1]Used by 1
Constructors
1Build a chain over the given iterable datasets.
Parameters
datasetslist of IterableDatasetIterable datasets to chain in order. Each must be an
IterableDataset instance.Raises
TypeErrorIf any element is not an
IterableDataset.