Блог

df.show выдает это:

 [scalatest]  ----------------------------- ---- ---- ------------------- 
[scalatest] |deserializedBlob             |a   |b   |c                  |
[scalatest]  ----------------------------- ---- ---- ------------------- 
[scalatest] |[2,2.1,2021-01-02 00:00:00.0]|2   |2.1 |2021-01-02 00:00:00|
[scalatest] |[1,1.1,2021-01-01 00:00:00.0]|1   |1.1 |2021-01-01 00:00:00|
[scalatest] |[null,null,null]             |null|null|null               |
[scalatest]  ----------------------------- ---- ---- ------------------- 
 

Я пытаюсь утверждать, что deserializedBlob столбец содержит эти строки, поэтому я попробовал это:

 df.filter(array_contains(col("deserializedBlob"), Timestamp.valueOf("2021-01-01 00:00:00.0"))).count shouldBe 1
 

но это приводит к ошибке

 [scalatest] - should test *** FAILED ***
[scalatest]   org.apache.spark.sql.AnalysisException: cannot resolve 'array_contains(`deserializedBlob`, TIMESTAMP('2021-01-01 00:00:00.0'))' due to data type mismatch: Arguments must be an array followed by a value of same type as the array members;;
[scalatest] 'Filter array_contains(deserializedBlob#38, 1609459200000000)
[scalatest]  - Project [deserializedBlob#38, a#45, b#46, c#47]
[scalatest]     - Project [deserializedBlob#38, fields_90688278_1d0e_47b8_a821_b555d78649ac#41, fields_90688278_1d0e_47b8_a821_b555d78649ac#41.a AS a#45, fields_90688278_1d0e_47b8_a821_b555d78649ac#41.b AS b#46, fields_90688278_1d0e_47b8_a821_b555d78649ac#41.c AS c#47]
[scalatest]        - Project [deserializedBlob#38, deserializedBlob#38 AS fields_90688278_1d0e_47b8_a821_b555d78649ac#41]
[scalatest]           - Project [method(StructField(a,IntegerType,true), StructField(b,DoubleType,true), StructField(c,TimestampType,true), binaryblob#34) AS deserializedBlob#38]
[scalatest]              - SubqueryAlias data
[scalatest]                 - Relation[binaryblob#34] customtype
[scalatest]   at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
[scalatest]   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$anonfun$checkAnalysis$1$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:93)
[scalatest]   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$anonfun$checkAnalysis$1$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:85)
[scalatest]   at org.apache.spark.sql.catalyst.trees.TreeNode$anonfun$transformUp$1.apply(TreeNode.scala:289)
[scalatest]   at org.apache.spark.sql.catalyst.trees.TreeNode$anonfun$transformUp$1.apply(TreeNode.scala:289)
[scalatest]   at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
[scalatest]   at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
[scalatest]   at org.apache.spark.sql.catalyst.plans.QueryPlan$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:268)
[scalatest]   at org.apache.spark.sql.catalyst.plans.QueryPlan$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:268)
[scalatest]   at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:279)
 

Даже если бы это сработало, для меня это не оптимально, поскольку я хочу убедиться, что столбец содержит весь массив строк, например [2,2.1,2021-01-02 00:00:00.0] .

Как мне это сделать?