Cloudera CDP-3002 問題集

質問 1：
Explain the concept of lineage tracking in Spark and its benefits for fault tolerance and debugging.
A. It creates a log of all operations performed on data, aiding in debugging issues.
B. Lineage allows for caching intermediate results, improving performance.
C. Both A and B.
D. Lineage tracks the dependencies between data transformations, enabling efficient re-execution on failures.
正解：C
解説: (Topexam メンバーにのみ表示されます)

質問 2：
You're building an Airflow DAG to automate data quality checks on the output of your ETL pipeline. The checks involve performing various data validation tasks like checking for missing values, ensuring data type consistency, and verifying data integrity based on specific business rules. How can you implement these checks within Airflow?
A. Utilize Python libraries like Pandas or Spark for data manipulation and validation within the PythonOperator.
B. Use the PythonOperator to write custom Python scripts for each individual check and chain them together in the DAG.
C. Leverage dedicated Airflow operators like BigQueryCheckOperator or S3KeySensor (these operators are specific to certain data sources and not generally applicable for all data quality checks).
D. All of the above
正解：D

質問 3：
You have an Airflow DAG that includes tasks for data extraction, transformation, and loading. You notice that the transformation tasks are computationally intensive and are causing delays in the DAG's execution. To optimize performance, you decide to offload these tasks to a cloud-based service that can scale dynamically. Which approach ensures minimal changes to the DAG structure while integrating this optimization?
A. Use the ExternalTaskSensor to wait for the transformation to complete on the cloud service before proceeding.
B. Implement the transformation tasks as DockerOperator tasks, with each task running in a containerized environment on the cloud service.
C. Replace the transformation tasks with HttpSensor tasks that trigger the cloud service and poll for completion.
D. Modify the transformation tasks to use the PythonOperator to make API calls to the cloud service, handling the transformation.
正解：D
解説: (Topexam メンバーにのみ表示されます)

質問 4：
How can you utilize Spark SQL for complex data analysis involving joins and aggregations on large datasets?
A. By using Spark ML libraries for machine learning tasks on data.
B. By directly writing SQL queries against distributed data in Spark.
C. Both A and B.
D. By implementing custom logic in RDDs for data manipulation.
正解：B
解説: (Topexam メンバーにのみ表示されます)

質問 5：
A data engineer needs to query a table stored in Apache Hive using SparkSQL. Which of the following commands correctly retrieves data from a Hive table named 'sales data'?
A.

B.

C.

D.

正解：C
解説: (Topexam メンバーにのみ表示されます)

質問 6：
When deploying a packaged PySpark application using 'spark-submit', which option is used to include the packaged dependencies?
A. --jars
B. --py-files
C. --packages
D. -files
正解：B
解説: (Topexam メンバーにのみ表示されます)

質問 7：
What is the difference between RDDs and DataFrames in Spark?
A. RDDs are mutable, while DataFrames are immutable
B. Both B and C
C. RDDs are lower-level abstractions, while DataFrames offer higher-level functionalities
D. RDDs represent distributed datasets, while DataFrames work with structured data
正解：B
解説: (Topexam メンバーにのみ表示されます)

質問 8：
You encounter an error during a data quality check within your Airflow DAG. How can you access detailed information about the error to aid in troubleshooting?
A. The Airflow logs (scheduler, worker) might contain relevant error messages from the failing task.
B. You can directly inspect the custom Python script used for the check to identify potential issues.
C. The Airflow web UI task details page might display error messages related to the failed check.
D. All of the above
正解：D
解説: (Topexam メンバーにのみ表示されます)