Databricks Databricks-Certified-Data-Engineer-Professional 問題集

質問 1：
A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.
Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?
A. Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.
B. Decrease the trigger interval to 5 seconds; triggering batches more frequently allows idle executors to begin processing the next batch while longer running tasks from previous batches finish.
C. Use the trigger once option and configure a Databricks job to execute the query every 10 seconds; this ensures all backlogged records are processed with each batch.
D. Increase the trigger interval to 30 seconds; setting the trigger interval near the maximum execution time observed for each batch is always best practice to ensure no records are dropped.
E. The trigger interval cannot be modified without modifying the checkpoint directory; to maintain the current stream state, increase the number of shuffle partitions to maximize parallelism.
正解：A
解説: (Topexam メンバーにのみ表示されます)

質問 2：
Spill occurs as a result of executing various wide transformations. However, diagnosing spill requires one to proactively look for key indicators.
Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?
A. Stage's detail screen and Executor's log files
B. Driver's and Executor's log files
C. Stage's detail screen and Query's detail screen
D. Query's detail screen and Job's detail screen
E. Executor's detail screen and Executor's log files
正解：A
解説: (Topexam メンバーにのみ表示されます)

質問 3：
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Incremental state information should be maintained for 10 minutes for late-arriving data.
Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:

Choose the response that correctly fills in the blank within the code block to complete this task.
A. withWatermark("event_time", "10 minutes")
B. delayWrite("event_time", "10 minutes")
C. slidingWindow("event_time", "10 minutes")
D. awaitArrival("event_time", "10 minutes")
E. await("event_time + `10 minutes'")
正解：A
解説: (Topexam メンバーにのみ表示されます)

質問 4：
A Data engineer wants to run unit's tests using common Python testing frameworks on python functions defined across several Databricks notebooks currently used in production. How can the data engineer run unit tests against function that work with data in production?
A. Define and unit test functions using Files in Repos
B. Run unit tests against non-production data that closely mirrors production
C. Define units test and functions within the same notebook
D. Define and import unit test functions from a separate Databricks notebook
正解：B
解説: (Topexam メンバーにのみ表示されます)

質問 5：
A team of data engineer are adding tables to a DLT pipeline that contain repetitive expectations for many of the same data quality checks.
One member of the team suggests reusing these data quality rules across all tables defined for this pipeline.
What approach would allow them to do this?
A. Use global Python variables to make expectations visible across DLT notebooks included in the same pipeline.
B. Maintain data quality rules in a separate Databricks notebook that each DLT notebook of file.
C. Add data quality constraints to tables in this pipeline using an external job with access to pipeline configuration files.
D. Maintain data quality rules in a Delta table outside of this pipeline's target schema, providing the schema name as a pipeline parameter.
正解：D
解説: (Topexam メンバーにのみ表示されます)

質問 6：
An upstream system has been configured to pass the date for a given batch of data to the Databricks Jobs API as a parameter. The notebook to be scheduled will use this parameter to load data with the following code:
df = spark.read.format("parquet").load(f"/mnt/source/(date)")
Which code block should be used to create the date Python variable used in the above code block?
A. import sys
date = sys.argv[1]
B. date = spark.conf.get("date")
C. input_dict = input()
date= input_dict["date"]
D. date = dbutils.notebooks.getParam("date")
E. dbutils.widgets.text("date", "null")
date = dbutils.widgets.get("date")
正解：E
解説: (Topexam メンバーにのみ表示されます)

質問 7：
The downstream consumers of a Delta Lake table have been complaining about data quality issues impacting performance in their applications. Specifically, they have complained that invalid latitude and longitude values in the activity_details table have been breaking their ability to use other geolocation processes.
A junior engineer has written the following code to add CHECK constraints to the Delta Lake table:

A senior engineer has confirmed the above logic is correct and the valid ranges for latitude and longitude are provided, but the code fails when executed.
Which statement explains the cause of this failure?
A. The activity details table already contains records; CHECK constraints can only be added prior to inserting values into a table.
B. The activity details table already contains records that violate the constraints; all existing data must pass CHECK constraints in order to add them to an existing table.
C. The current table schema does not contain the field valid coordinates; schema evolution will need Get Latest & Actual Certified-Data-Engineer-Professional Exam's Question and Answers from to be enabled before altering the table to add a constraint.
D. The activity details table already exists; CHECK constraints can only be added during initial table creation.
E. Because another team uses this table to support a frequently running application, two-phase locking is preventing the operation from committing.
正解：B
解説: (Topexam メンバーにのみ表示されます)