In summary, NONE sort mode will offer the best write latency, but depending on your workload, could have impact on subsequent writes and on read latency as well. You could see an explosion of N*M files at max (where N is number of spark partitions and M is number of hudi partitions). And on top of all this, this will very likely result in lot of small files, since each spark partition could contain data from all hudi partitions. Also, for a partitioned dataset, with huge number of partitions, this could have memory pressure on each task, since it has to hold references to all hudi partitions (since each spark partition could receive data for all hudi partitions) and could result in Out of Memory(OOMs). Using NONE sort mode, records key ranges could be very chaotic and subsequent updates could touch all file groups in the table and thus could have higher write latencies. Lets say your record keys has some temporal affinity and your updates often touch latest data. This does not mean that this will be best sort mode as there could be some repercussions. Since there is no extra shuffle involved in the spark DAG, writes will be pretty fast. Incoming dataframe is delegated to N tasks as is, depending on the. This is the default sort mode and as you might have guessed, there is no sorting with this mode. NONE, GLOBAL_SORT, PARTITION_SORT, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT.
Hudi offers 5 different sort modes that you can leverage while ingesting data via “bulk_insert” operation. This blog is going to touch upon the different sort modes and some under the hood analysis of what happens in each of these cases. Bulk_insert has an optional step, where records could be sorted before being ingested to Hudi. We have already seen/read about bulk_insert operation with Apache Hudi on various occasions( link1, link2, link3).
0 kommentar(er)
