Output: Step -10: Create and update New Column in Existing DataFrame Show multiple columns df.select('Film', 'Year').show(5) We are just using the select method to show a single column df.select('Film').show(5) Output: Step -9: How Select Single and Multiple Columns We are just using the lenmethod to get the total number len(df.toPandas(). Output: Step -8: Find the Total Number of Unique Values Available in the Gender column Output: Step -7: Find the unique value in the specific columnįor that, we are using the toPandas method to convert the data frame into the panda’s data frame then add column name then using a unique method df.toPandas(). To display the overall statistic we are using describe method and df.describe(). The number of columns is 8 Step -6: Get Overall Statistics About The Dataset To display several columns we can use the length method on the columns list len(df.columns) To display the number of rows we are using the count method df.count() Output: Step -5: Count the Number of Rows and Columns of the dataset # load datasetĭf = ('/content/movies.csv', header=True, inferSchema=True)
Here we are using the printSchemamethod to show the datatypes of the columns and also added the inferschema parameter in the df. Step -4: Display the Datatype of Each Column It’s fast, it’s easy to use, and it offers a wide range of capabilities. They’ve chosen Spark because, unlike their competitors, Spark just works. Companies, large and small, are turning to it to analyze their data. Recently, Spark has become a major player in the field of big data. Spark is a cluster computing framework built around speed, ease of use, and sophisticated analytics of big data using Python. PySpark is an interface for Apache Spark in Python. 17 Final Words Pre-requisites: Pyspark:.16 Step -15: Sort the rows in ascending and descending Order.14 Step -13: Multiple conditions at the same time.11 Step -10: Create and update New Column in Existing DataFrame.10 Step -9: How Select Single and Multiple Columns.9 Step -8: Find the Total Number of Unique Values Available in the Gender column.8 Step -7: Find the unique value in the specific column.7 Step -6: Get Overall Statistics About The Dataset.6 Step -5: Count the Number of Rows and Columns of the dataset.5 Step -4: Display the Datatype of Each Column.
0 kommentar(er)
