In the lecture on Spark Structured API, we did not specify the schema of our dataset. We relied on the inference of Spark engine which may not always be accurate. We can create a schema by using an object of a class called StructType consisting of an array of StructFields.
More details on Spark Schemas can be found at this link https://sparkbyexamples.com/spark/spark-schema-exp… The code to load the youtube dataset used in the lectures with a schema has been provided as a guide. Once you are familiar with how to create schemas, load the stocks dataset into Spark.
Note that dates in Spark are only recognized if they have a special format. You can treat dates as strings for simplicity. Once you have loaded the stocks datasets with the correct schema in Spark, answer ONE of the following query questions:
Find the top 5 stocks with the maximum average trading volume
Find the top 5 stocks with the maximum closing price
Find the top 5 stocks with the highest price change during any trading day
DELIVERABLES Submit your code (creating the schema, loading of data as a DataFrame, and the corresponding query) as text file. Along with your code, in a separate file, submit the screenshots of your code being executed.
PLACE THE ORDER WITH US TODAY AND GET A PERFECT SCORE!!!