array in databricks

Hello Vineet S,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you the challenges you're having is a 2-Dimensional Array in Databricks where each row is a reshuffled version of the previous row, starting from a sorted array.

Solution

To solve this, ensure you have a PySpark session running in Databricks. Then, you can follow these steps:

# Step 1: Create the initial sorted array
initial_array = [1, 2, 3, 4, 5]
# Function to generate the reshuffled arrays
def reshuffle_array(array, steps):
    return array[steps:] + array[:steps]
# Step 2: Generate the reshuffled rows
reshuffled_rows = [reshuffle_array(initial_array, i) for i in range(5)]
# Convert to a PySpark DataFrame
columns = ["col1", "col2", "col3", "col4", "col5"]
df = spark.createDataFrame(reshuffled_rows, schema=columns)
# Step 3: Display the DataFrame
display(df)

The outcome is:

+----+----+----+----+----+
|col1|col2|col3|col4|col5|
+----+----+----+----+----+
|   1|   2|   3|   4|   5|
|   5|   1|   2|   3|   4|
|   4|   5|   1|   2|   3|
|   3|   4|   5|   1|   2|
|   2|   3|   4|   5|   1|
+----+----+----+----+----+

This generate a DataFrame with the reshuffled rows as specified:

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam