pyspark.sql.DataFrame.randomSplit#

DataFrame.randomSplit(weights, seed=None)[source]#

Randomly splits this DataFrame with the provided weights.

New in version 1.4.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters
weightslist

list of doubles as weights with which to split the DataFrame. Weights will be normalized if they don’t sum up to 1.0.

seedint, optional

The seed for sampling.

Returns
list

List of DataFrames.

Examples

>>> from pyspark.sql import Row
>>> df = spark.createDataFrame([
...     Row(age=10, height=80, name="Alice"),
...     Row(age=5, height=None, name="Bob"),
...     Row(age=None, height=None, name="Tom"),
...     Row(age=None, height=None, name=None),
... ])
>>> splits = df.randomSplit([1.0, 2.0], 24)
>>> splits[0].count()
2
>>> splits[1].count()
2