Databricks Databricks-Certified-Professional-Data-Engineer 問題集

質問 1：
Incorporating unit tests into a PySpark application requires upfront attention to the design of your jobs, or a potentially significant refactoring of existing code.
Which statement describes a main benefit that offset this additional effort?
A. Yields faster deployment and execution times
B. Troubleshooting is easier since all steps are isolated and tested individually
C. Improves the quality of your data
D. Ensures that all steps interact correctly to achieve the desired end result
E. Validates a complete use case of your application
正解：B

質問 2：
A data engineer is designing a pipeline in Databricks that processes records from a Kafka stream where late- arriving data is common.
Which approach should the data engineer use?
A. Use batch processing and overwrite the entire output table each time to ensure late data is incorporated correctly.
B. Use a watermark to specify the allowed lateness to accommodate records that arrive after their expected window, ensuring correct aggregation and state management.
C. Use an Auto CDC pipeline with batch tables to simplify late data handling.
D. Implement a custom solution using Databricks Jobs to periodically reprocess all historical data.
正解：B
解説: (Topexam メンバーにのみ表示されます)

質問 3：
Which approach demonstrates a modular and testable way to use DataFrame.transform for ETL code in PySpark?
A. def transform_data(input_df):
# transformation logic here
return output_df
test_input = spark.createDataFrame([(1, " a " )], [ " id " , " value " ]) assertDataFrameEqual(transform_data(test_input), expected)
B. def upper_transform(df):
return df.withColumn( " value_upper " , upper(col( " value " )))
actual = test_input.transform(upper_transform)
assertDataFrameEqual(actual, expected)
C. def upper_value(df):
return df.withColumn( " value_upper " , upper(col( " value " )))
def filter_positive(df):
return df.filter(df[ " id " ] > 0)
pipeline_df = df.transform(upper_value).transform(filter_positive)
D. class Pipeline:
def transform(self, df):
return df.withColumn( " value_upper " , upper(col( " value " )))
pipeline = Pipeline()
assertDataFrameEqual(pipeline.transform(test_input), expected)
正解：C
解説: (Topexam メンバーにのみ表示されます)

質問 4：
A new data engineer notices that a critical field was omitted from an application that writes its Kafka source to Delta Lake. This happened even though the critical field was in the Kafka source. That field was further missing from data written to dependent, long-term storage. The retention threshold on the Kafka service is seven days. The pipeline has been in production for three months.
Which describes how Delta Lake can help to avoid data loss of this nature in the future?
A. The Delta log and Structured Streaming checkpoints record the full history of the Kafka producer.
B. Data can never be permanently dropped or deleted from Delta Lake, so data loss is not possible under any circumstance.
C. Ingestine all raw data and metadata from Kafka to a bronze Delta table creates a permanent, replayable history of the data state.
D. Delta Lake automatically checks that all fields present in the source data are included in the ingestion layer.
E. Delta Lake schema evolution can retroactively calculate the correct value for newly added fields, as long as the data was in the original source.
正解：C
解説: (Topexam メンバーにのみ表示されます)

質問 5：
A Delta Lake table was created with the below query:

Realizing that the original query had a typographical error, the below code was executed:
ALTER TABLE prod.sales_by_stor RENAME TO prod.sales_by_store
Which result will occur after running the second command?
A. All related files and metadata are dropped and recreated in a single ACID transaction.
B. The table reference in the metastore is updated and no data is changed.
C. The table name change is recorded in the Delta transaction log.
D. The table reference in the metastore is updated and all data files are moved.
E. A new Delta transaction log Is created for the renamed table.
正解：B
解説: (Topexam メンバーにのみ表示されます)