crp-demand-service - Details for Query 51

Submitted Time: 2026/01/27 08:08:10
Duration: 0.6 s
Succeeded Jobs: 140 141 142 143 144 145 146

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (6)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 64.2 MiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 3 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 251.0 B number of coalesced partitions: 1"]; 4 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 244.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 240.0 B local merged bytes read: 0.0 B number of partitions: 37 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 244.0 B"]; subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 12 ms"; 6 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 4 ms peak memory: 64.2 MiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 1"]; 7 [labelType="html" label=" Project "]; 8 [labelType="html" label="BroadcastHashJoin number of output rows: 1"]; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="Filter number of output rows: 1"]; 17 [labelType="html" label="Generate number of output rows: 1"]; 18 [labelType="html" label=" Project "]; 19 [labelType="html" label="BroadcastHashJoin number of output rows: 1"]; 26 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 3 ms peak memory: 256.0 KiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; } 9 [labelType="html" label="BroadcastExchange time to broadcast: 0 ms time to build: 6 ms time to collect: 372 ms number of output rows: 10 data size: 64.0 MiB"]; 10 [labelType="html" label=" Project "]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n1.5 s (189 ms, 214 ms, 258 ms (stage 200.0: task 1599))"; 12 [labelType="html" label="Filter number of output rows: 10"]; 13 [labelType="html" label="ColumnarToRow number of output rows: 10 number of input batches: 7"]; } 14 [labelType="html" label="Scan parquet number of files read: 7 scan time total (min, med, max (stageId: taskId)) 1.5 s (187 ms, 212 ms, 256 ms (stage 200.0: task 1599)) metadata time: 0 ms size of files read: 49.6 KiB number of output rows: 10"]; 20 [labelType="html" label="BroadcastExchange time to broadcast: 0 ms time to build: 6 ms time to collect: 20 ms number of output rows: 10 data size: 64.0 MiB"]; 21 [labelType="html" label="AQEShuffleRead number of partitions: 1"]; 22 [labelType="html" label="Exchange shuffle records written: 10 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 5 ms (0 ms, 0 ms, 0 ms (stage 198.0: task 1596)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 10 local bytes read: 520.0 B fetch wait time: 0 ms remote bytes read: 788.0 B merged fetch fallback count: 0 local blocks read: 4 remote merged chunks fetched: 0 remote blocks read: 6 data size total (min, med, max (stageId: taskId)) 920.0 B (80.0 B, 96.0 B, 104.0 B (stage 198.0: task 1587)) local merged bytes read: 0.0 B number of partitions: 37 remote reqs duration: 2 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 1308.0 B (123.0 B, 132.0 B, 136.0 B (stage 198.0: task 1591))"]; subgraph cluster23 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n1.2 s (103 ms, 119 ms, 141 ms (stage 198.0: task 1588))"; 24 [labelType="html" label="Filter number of output rows: 10"]; 25 [labelType="html" label="Scan ExistingRDD number of output rows: 10"]; } 27 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 2.4 KiB number of coalesced partitions: 1"]; 28 [labelType="html" label="Exchange shuffle records written: 25 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 14 ms (0 ms, 0 ms, 0 ms (stage 203.0: task 1617)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 25 local bytes read: 644.0 B fetch wait time: 0 ms remote bytes read: 1656.0 B merged fetch fallback count: 0 local blocks read: 7 remote merged chunks fetched: 0 remote blocks read: 18 data size total (min, med, max (stageId: taskId)) 800.0 B (32.0 B, 32.0 B, 32.0 B (stage 203.0: task 1609)) local merged bytes read: 0.0 B number of partitions: 37 remote reqs duration: 2 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 2.2 KiB (92.0 B, 92.0 B, 92.0 B (stage 203.0: task 1609))"]; subgraph cluster29 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n2.5 s (81 ms, 100 ms, 158 ms (stage 203.0: task 1622))"; 30 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 2.4 s (79 ms, 94 ms, 155 ms (stage 203.0: task 1622)) peak memory total (min, med, max (stageId: taskId)) 6.3 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 203.0: task 1609)) number of output rows: 25 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 31 [labelType="html" label=" Project "]; 32 [labelType="html" label="BroadcastHashJoin number of output rows: 1,174"]; 33 [labelType="html" label="ColumnarToRow number of output rows: 1,196 number of input batches: 25"]; } 34 [labelType="html" label="Scan parquet number of files read: 25 scan time total (min, med, max (stageId: taskId)) 2.4 s (79 ms, 94 ms, 154 ms (stage 203.0: task 1622)) metadata time: 0 ms size of files read: 83.2 KiB number of output rows: 1,196"]; 35 [labelType="html" label="BroadcastExchange time to broadcast: 1 ms time to build: 7 ms time to collect: 184 ms number of output rows: 1,174 data size: 64.1 MiB"]; subgraph cluster36 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 159 ms"; 37 [labelType="html" label="Filter number of output rows: 1,174"]; 38 [labelType="html" label="ColumnarToRow number of output rows: 1,196 number of input batches: 1"]; } 39 [labelType="html" label="Scan parquet number of files read: 1 scan time: 158 ms metadata time: 0 ms size of files read: 8.1 KiB number of output rows: 1,196"]; 2->0; 3->2; 4->3; 6->4; 7->6; 8->7; 9->8; 10->9; 12->10; 13->12; 14->13; 15->8; 16->15; 17->16; 18->17; 19->18; 20->19; 21->20; 22->21; 24->22; 25->24; 26->19; 27->26; 28->27; 30->28; 31->30; 32->31; 33->32; 34->33; 35->32; 37->35; 38->37; 39->38; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719], functions=[])

WholeStageCodegen (6)

AQEShuffleRead coalesced

Exchange hashpartitioning(name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719, 37), ENSURE_REQUIREMENTS, [plan_id=8022]

HashAggregate(keys=[name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719], functions=[])

Project [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]

BroadcastHashJoin [name#5717], [name#5711], Inner, BuildLeft, false

Project [explodedDemandChannels#5705.name AS name#5711]

Filter isnotnull(explodedDemandChannels#5705.name)

Generate explode(demandChannels#5693), false, [explodedDemandChannels#5705]

Project [demandChannels#5693]

BroadcastHashJoin [demandProfile#5692], [demandProfile#5541], Inner, BuildLeft, false

HashAggregate(keys=[demandProfile#5541], functions=[])

WholeStageCodegen (5)

BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]),false), [plan_id=7612]

Project [name#5717, demandStream#5719, forecastCode#5720, transform(distributionChains#5722, lambdafunction(named_struct(internalUUID, lambda chain#5738.internalUUID, salesOrganizationUUID, lambda chain#5738.salesOrganization.internalRefUUID, distributionChannelUUID, lambda chain#5738.distributionChannel.internalRefUUID), lambda chain#5738, false)) AS distributionChains#5737, transform(customerOrderTypes#5723, lambdafunction(named_struct(internalUUID, lambda documentType#5751.internalRefUUID), lambda documentType#5751, false)) AS customerOrderTypes#5750, transform(customerOrderItemTypeCodes#5724, lambdafunction(named_struct(customerOrderItemTypeCodeUUID, lambda itemType#5763.internalRefUUID), lambda itemType#5763, false)) AS customerOrderItemTypeCodes#5762, considerVariance#5726]

Filter isnotnull(name#5717)

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [name#5717,demandStream#5719,forecastCode#5720,distributionChains#5722,customerOrderTypes#5723,customerOrderItemTypeCodes#5724,considerVariance#5726] Batched: true, DataFilters: [isnotnull(name#5717)], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20...., PartitionFilters: [], PushedFilters: [IsNotNull(name)], ReadSchema: struct<name:string,demandStream:int,forecastCode:string,distributionChains:array<struct<internalU...

BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false]),false), [plan_id=7752]

AQEShuffleRead local

Exchange hashpartitioning(demandProfile#5692, 37), ENSURE_REQUIREMENTS, [plan_id=7623]

Filter (isnotnull(demandProfile#5692) AND ((size(demandChannels#5693, true) > 0) AND isnotnull(demandChannels#5693)))

Scan ExistingRDD[demandProfile#5692,demandChannels#5693]

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(demandProfile#5541, 37), ENSURE_REQUIREMENTS, [plan_id=7854]

HashAggregate(keys=[demandProfile#5541], functions=[])

Project [demandProfile#5541]

BroadcastHashJoin [coalesce(plant#5531, ), isnull(plant#5531), coalesce(product#5530, ), isnull(product#5530)], [coalesce(plant#5539, ), isnull(plant#5539), coalesce(product#5538, ), isnull(product#5538)], Inner, BuildRight, false

ColumnarToRow

WholeStageCodegen (4)

FileScan parquet [product#5530,plant#5531] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<product:string,plant:string>

BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=7636]

Filter isnotnull(demandProfile#5541)

ColumnarToRow

WholeStageCodegen (3)

FileScan parquet [product#5538,plant#5539,demandprofile#5541] Batched: true, DataFilters: [isnotnull(demandprofile#5541)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [IsNotNull(demandprofile)], ReadSchema: struct<product:string,plant:string,demandprofile:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (64)
+- == Final Plan ==
   * HashAggregate (39)
   +- AQEShuffleRead (38)
      +- ShuffleQueryStage (37), Statistics(sizeInBytes=240.0 B, rowCount=1)
         +- Exchange (36)
            +- * HashAggregate (35)
               +- * Project (34)
                  +- * BroadcastHashJoin Inner BuildLeft (33)
                     :- BroadcastQueryStage (6), Statistics(sizeInBytes=64.0 MiB, rowCount=10)
                     :  +- BroadcastExchange (5)
                     :     +- Project (4)
                     :        +- * Filter (3)
                     :           +- * ColumnarToRow (2)
                     :              +- Scan parquet  (1)
                     +- * Project (32)
                        +- * Filter (31)
                           +- * Generate (30)
                              +- * Project (29)
                                 +- * BroadcastHashJoin Inner BuildLeft (28)
                                    :- BroadcastQueryStage (13), Statistics(sizeInBytes=64.0 MiB, rowCount=10)
                                    :  +- BroadcastExchange (12)
                                    :     +- AQEShuffleRead (11)
                                    :        +- ShuffleQueryStage (10), Statistics(sizeInBytes=920.0 B, rowCount=10)
                                    :           +- Exchange (9)
                                    :              +- * Filter (8)
                                    :                 +- * Scan ExistingRDD (7)
                                    +- * HashAggregate (27)
                                       +- AQEShuffleRead (26)
                                          +- ShuffleQueryStage (25), Statistics(sizeInBytes=800.0 B, rowCount=25)
                                             +- Exchange (24)
                                                +- * HashAggregate (23)
                                                   +- * Project (22)
                                                      +- * BroadcastHashJoin Inner BuildRight (21)
                                                         :- * ColumnarToRow (15)
                                                         :  +- Scan parquet  (14)
                                                         +- BroadcastQueryStage (20), Statistics(sizeInBytes=64.1 MiB, rowCount=1.17E+3)
                                                            +- BroadcastExchange (19)
                                                               +- * Filter (18)
                                                                  +- * ColumnarToRow (17)
                                                                     +- Scan parquet  (16)
+- == Initial Plan ==
   HashAggregate (63)
   +- Exchange (62)
      +- HashAggregate (61)
         +- Project (60)
            +- BroadcastHashJoin Inner BuildLeft (59)
               :- BroadcastExchange (42)
               :  +- Project (41)
               :     +- Filter (40)
               :        +- Scan parquet  (1)
               +- Project (58)
                  +- Filter (57)
                     +- Generate (56)
                        +- Project (55)
                           +- SortMergeJoin Inner (54)
                              :- Sort (44)
                              :  +- Exchange (43)
                              :     +- Filter (8)
                              :        +- Scan ExistingRDD (7)
                              +- Sort (53)
                                 +- HashAggregate (52)
                                    +- Exchange (51)
                                       +- HashAggregate (50)
                                          +- Project (49)
                                             +- BroadcastHashJoin Inner BuildRight (48)
                                                :- Scan parquet  (45)
                                                +- BroadcastExchange (47)
                                                   +- Filter (46)
                                                      +- Scan parquet  (16)


(1) Scan parquet 
Output [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5722, customerOrderTypes#5723, customerOrderItemTypeCodes#5724, considerVariance#5726]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/demand-channel]
PushedFilters: [IsNotNull(name)]
ReadSchema: struct<name:string,demandStream:int,forecastCode:string,distributionChains:array<struct<internalUUID:string,salesOrganization:struct<internalRefUUID:string,id:string>,distributionChannel:struct<internalRefUUID:string,id:string>>>,customerOrderTypes:array<struct<id:string,internalRefUUID:string>>,customerOrderItemTypeCodes:array<struct<id:string,internalRefUUID:string>>,considerVariance:boolean>

(2) ColumnarToRow [codegen id : 1]
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5722, customerOrderTypes#5723, customerOrderItemTypeCodes#5724, considerVariance#5726]

(3) Filter [codegen id : 1]
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5722, customerOrderTypes#5723, customerOrderItemTypeCodes#5724, considerVariance#5726]
Condition : isnotnull(name#5717)

(4) Project
Output [7]: [name#5717, demandStream#5719, forecastCode#5720, transform(distributionChains#5722, lambdafunction(named_struct(internalUUID, lambda chain#5738.internalUUID, salesOrganizationUUID, lambda chain#5738.salesOrganization.internalRefUUID, distributionChannelUUID, lambda chain#5738.distributionChannel.internalRefUUID), lambda chain#5738, false)) AS distributionChains#5737, transform(customerOrderTypes#5723, lambdafunction(named_struct(internalUUID, lambda documentType#5751.internalRefUUID), lambda documentType#5751, false)) AS customerOrderTypes#5750, transform(customerOrderItemTypeCodes#5724, lambdafunction(named_struct(customerOrderItemTypeCodeUUID, lambda itemType#5763.internalRefUUID), lambda itemType#5763, false)) AS customerOrderItemTypeCodes#5762, considerVariance#5726]
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5722, customerOrderTypes#5723, customerOrderItemTypeCodes#5724, considerVariance#5726]

(5) BroadcastExchange
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true]),false), [plan_id=7612]

(6) BroadcastQueryStage
Output [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Arguments: 0

(7) Scan ExistingRDD [codegen id : 2]
Output [2]: [demandProfile#5692, demandChannels#5693]
Arguments: [demandProfile#5692, demandChannels#5693], MapPartitionsRDD[526] at createDataFrame at DemandProfileDataReaderImpl.java:60, ExistingRDD, UnknownPartitioning(0)

(8) Filter [codegen id : 2]
Input [2]: [demandProfile#5692, demandChannels#5693]
Condition : (isnotnull(demandProfile#5692) AND ((size(demandChannels#5693, true) > 0) AND isnotnull(demandChannels#5693)))

(9) Exchange
Input [2]: [demandProfile#5692, demandChannels#5693]
Arguments: hashpartitioning(demandProfile#5692, 37), ENSURE_REQUIREMENTS, [plan_id=7623]

(10) ShuffleQueryStage
Output [2]: [demandProfile#5692, demandChannels#5693]
Arguments: 1

(11) AQEShuffleRead
Input [2]: [demandProfile#5692, demandChannels#5693]
Arguments: local

(12) BroadcastExchange
Input [2]: [demandProfile#5692, demandChannels#5693]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false]),false), [plan_id=7752]

(13) BroadcastQueryStage
Output [2]: [demandProfile#5692, demandChannels#5693]
Arguments: 3

(14) Scan parquet 
Output [2]: [product#5530, plant#5531]
Batched: true
Location: InMemoryFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/product-plant-list/10000000092/shardId=0_1_10000000092]
ReadSchema: struct<product:string,plant:string>

(15) ColumnarToRow [codegen id : 4]
Input [2]: [product#5530, plant#5531]

(16) Scan parquet 
Output [3]: [product#5538, plant#5539, demandprofile#5541]
Batched: true
Location: InMemoryFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-param-determination-service/out/10000000092/0_1_10000000092/demandprofile]
PushedFilters: [IsNotNull(demandprofile)]
ReadSchema: struct<product:string,plant:string,demandprofile:string>

(17) ColumnarToRow [codegen id : 3]
Input [3]: [product#5538, plant#5539, demandprofile#5541]

(18) Filter [codegen id : 3]
Input [3]: [product#5538, plant#5539, demandprofile#5541]
Condition : isnotnull(demandProfile#5541)

(19) BroadcastExchange
Input [3]: [product#5538, plant#5539, demandprofile#5541]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=7636]

(20) BroadcastQueryStage
Output [3]: [product#5538, plant#5539, demandprofile#5541]
Arguments: 2

(21) BroadcastHashJoin [codegen id : 4]
Left keys [4]: [coalesce(plant#5531, ), isnull(plant#5531), coalesce(product#5530, ), isnull(product#5530)]
Right keys [4]: [coalesce(plant#5539, ), isnull(plant#5539), coalesce(product#5538, ), isnull(product#5538)]
Join type: Inner
Join condition: None

(22) Project [codegen id : 4]
Output [1]: [demandProfile#5541]
Input [5]: [product#5530, plant#5531, product#5538, plant#5539, demandprofile#5541]

(23) HashAggregate [codegen id : 4]
Input [1]: [demandProfile#5541]
Keys [1]: [demandProfile#5541]
Functions: []
Aggregate Attributes: []
Results [1]: [demandProfile#5541]

(24) Exchange
Input [1]: [demandProfile#5541]
Arguments: hashpartitioning(demandProfile#5541, 37), ENSURE_REQUIREMENTS, [plan_id=7854]

(25) ShuffleQueryStage
Output [1]: [demandProfile#5541]
Arguments: 4

(26) AQEShuffleRead
Input [1]: [demandProfile#5541]
Arguments: coalesced

(27) HashAggregate
Input [1]: [demandProfile#5541]
Keys [1]: [demandProfile#5541]
Functions: []
Aggregate Attributes: []
Results [1]: [demandProfile#5541]

(28) BroadcastHashJoin
Left keys [1]: [demandProfile#5692]
Right keys [1]: [demandProfile#5541]
Join type: Inner
Join condition: None

(29) Project
Output [1]: [demandChannels#5693]
Input [3]: [demandProfile#5692, demandChannels#5693, demandProfile#5541]

(30) Generate
Input [1]: [demandChannels#5693]
Arguments: explode(demandChannels#5693), false, [explodedDemandChannels#5705]

(31) Filter
Input [1]: [explodedDemandChannels#5705]
Condition : isnotnull(explodedDemandChannels#5705.name)

(32) Project
Output [1]: [explodedDemandChannels#5705.name AS name#5711]
Input [1]: [explodedDemandChannels#5705]

(33) BroadcastHashJoin [codegen id : 5]
Left keys [1]: [name#5717]
Right keys [1]: [name#5711]
Join type: Inner
Join condition: None

(34) Project [codegen id : 5]
Output [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Input [8]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726, name#5711]

(35) HashAggregate [codegen id : 5]
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Keys [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Functions: []
Aggregate Attributes: []
Results [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]

(36) Exchange
Input [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Arguments: hashpartitioning(name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719, 37), ENSURE_REQUIREMENTS, [plan_id=8022]

(37) ShuffleQueryStage
Output [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Arguments: 5

(38) AQEShuffleRead
Input [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Arguments: coalesced

(39) HashAggregate [codegen id : 6]
Input [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Keys [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Functions: []
Aggregate Attributes: []
Results: []

(40) Filter
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5722, customerOrderTypes#5723, customerOrderItemTypeCodes#5724, considerVariance#5726]
Condition : isnotnull(name#5717)

(41) Project
Output [7]: [name#5717, demandStream#5719, forecastCode#5720, transform(distributionChains#5722, lambdafunction(named_struct(internalUUID, lambda chain#5738.internalUUID, salesOrganizationUUID, lambda chain#5738.salesOrganization.internalRefUUID, distributionChannelUUID, lambda chain#5738.distributionChannel.internalRefUUID), lambda chain#5738, false)) AS distributionChains#5737, transform(customerOrderTypes#5723, lambdafunction(named_struct(internalUUID, lambda documentType#5751.internalRefUUID), lambda documentType#5751, false)) AS customerOrderTypes#5750, transform(customerOrderItemTypeCodes#5724, lambdafunction(named_struct(customerOrderItemTypeCodeUUID, lambda itemType#5763.internalRefUUID), lambda itemType#5763, false)) AS customerOrderItemTypeCodes#5762, considerVariance#5726]
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5722, customerOrderTypes#5723, customerOrderItemTypeCodes#5724, considerVariance#5726]

(42) BroadcastExchange
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true]),false), [plan_id=7548]

(43) Exchange
Input [2]: [demandProfile#5692, demandChannels#5693]
Arguments: hashpartitioning(demandProfile#5692, 37), ENSURE_REQUIREMENTS, [plan_id=7539]

(44) Sort
Input [2]: [demandProfile#5692, demandChannels#5693]
Arguments: [demandProfile#5692 ASC NULLS FIRST], false, 0

(45) Scan parquet 
Output [2]: [product#5530, plant#5531]
Batched: true
Location: InMemoryFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/product-plant-list/10000000092/shardId=0_1_10000000092]
ReadSchema: struct<product:string,plant:string>

(46) Filter
Input [3]: [product#5538, plant#5539, demandprofile#5541]
Condition : isnotnull(demandProfile#5541)

(47) BroadcastExchange
Input [3]: [product#5538, plant#5539, demandprofile#5541]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=7530]

(48) BroadcastHashJoin
Left keys [4]: [coalesce(plant#5531, ), isnull(plant#5531), coalesce(product#5530, ), isnull(product#5530)]
Right keys [4]: [coalesce(plant#5539, ), isnull(plant#5539), coalesce(product#5538, ), isnull(product#5538)]
Join type: Inner
Join condition: None

(49) Project
Output [1]: [demandProfile#5541]
Input [5]: [product#5530, plant#5531, product#5538, plant#5539, demandprofile#5541]

(50) HashAggregate
Input [1]: [demandProfile#5541]
Keys [1]: [demandProfile#5541]
Functions: []
Aggregate Attributes: []
Results [1]: [demandProfile#5541]

(51) Exchange
Input [1]: [demandProfile#5541]
Arguments: hashpartitioning(demandProfile#5541, 37), ENSURE_REQUIREMENTS, [plan_id=7535]

(52) HashAggregate
Input [1]: [demandProfile#5541]
Keys [1]: [demandProfile#5541]
Functions: []
Aggregate Attributes: []
Results [1]: [demandProfile#5541]

(53) Sort
Input [1]: [demandProfile#5541]
Arguments: [demandProfile#5541 ASC NULLS FIRST], false, 0

(54) SortMergeJoin
Left keys [1]: [demandProfile#5692]
Right keys [1]: [demandProfile#5541]
Join type: Inner
Join condition: None

(55) Project
Output [1]: [demandChannels#5693]
Input [3]: [demandProfile#5692, demandChannels#5693, demandProfile#5541]

(56) Generate
Input [1]: [demandChannels#5693]
Arguments: explode(demandChannels#5693), false, [explodedDemandChannels#5705]

(57) Filter
Input [1]: [explodedDemandChannels#5705]
Condition : isnotnull(explodedDemandChannels#5705.name)

(58) Project
Output [1]: [explodedDemandChannels#5705.name AS name#5711]
Input [1]: [explodedDemandChannels#5705]

(59) BroadcastHashJoin
Left keys [1]: [name#5717]
Right keys [1]: [name#5711]
Join type: Inner
Join condition: None

(60) Project
Output [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Input [8]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726, name#5711]

(61) HashAggregate
Input [7]: [name#5717, demandStream#5719, forecastCode#5720, distributionChains#5737, customerOrderTypes#5750, customerOrderItemTypeCodes#5762, considerVariance#5726]
Keys [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Functions: []
Aggregate Attributes: []
Results [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]

(62) Exchange
Input [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Arguments: hashpartitioning(name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719, 37), ENSURE_REQUIREMENTS, [plan_id=7553]

(63) HashAggregate
Input [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Keys [7]: [name#5717, considerVariance#5726, customerOrderItemTypeCodes#5762, customerOrderTypes#5750, distributionChains#5737, forecastCode#5720, demandStream#5719]
Functions: []
Aggregate Attributes: []
Results: []

(64) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true