Skip to content

.net for sqark and ML #5581

Closed as not planned
Closed as not planned
.net for sqark and ML#5581
Labels
SparkIssues relating to running ML.NET using .NET for SparkbugSomething isn't working

Description

@CJX-nice
CJX-nice
opened on Jan 9, 2021

System information

windows 10
.net core 3.1
spark 3.0.1
.net for spark 1.0.0

Issue

Learn < Sentiment analysis with .NET for Apache Spark and ML.NET> in https://docs.microsoft.com/zh-cn/dotnet/spark/tutorials/ml-sentiment-analysis

I create my project and copy the official code ;
but An error occurred
[Error] [TaskRunner] [0] ProcessStream() failed with exception: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.

Source code / logs

// This file was auto-generated by ML.NET Model Builder.
this is my Code :
using System;
using System.Collections.Generic;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.Spark.Sql;
using MySparkAppML.Model;

namespace MySparkAppML.ConsoleApp
{
public class Program
{
public static void Main(string[] args)
{
SparkSession spark = SparkSession
.Builder()
.AppName(".NET for Apache Spark Sentiment Analysis")
.GetOrCreate();
DataFrame df = spark .Read() .Option("header", true).Option("inferSchema", true) .Csv("yelptest.csv");
df.Show();
Console.WriteLine(predict("aaa"));
Console.WriteLine(predict("bbb"));
spark.Udf() .Register<string, float>("MLudf", predict);
df.CreateOrReplaceTempView("Reviews");
DataFrame sqlDf = spark.Sql("SELECT ReviewText, MLudf(ReviewText) FROM Reviews");
sqlDf.Show();
Console.ReadLine();
}
static float predict(string text)
{
MLContext mlContext = new MLContext();
ITransformer model = mlContext.Model.Load("MLModel.zip", out var schema);
var Engine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(model);
return Engine.Predict(new ModelInput() { ReviewText = text }).Score;
}
}
}

my log

C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\bin\Debug\netcoreapp3.1\publish>spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-3-0_2.12-1.0.0.jar dotnet MySparkAppML.ConsoleApp.dll
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/C:/bin/spark-3.0.1-bin-hadoop2.7/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/01/09 18:29:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/01/09 18:29:55 INFO DotnetRunner: Starting DotnetBackend with dotnet.
21/01/09 18:29:55 INFO DotnetBackend: The number of DotnetBackend threads is set to 10.
21/01/09 18:29:57 INFO DotnetRunner: Port number used by DotnetBackend is 53683
21/01/09 18:29:57 INFO DotnetRunner: Adding key=spark.jars and value=file:/C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/microsoft-spark-3-0_2.12-1.0.0.jar to environment
21/01/09 18:29:57 INFO DotnetRunner: Adding key=spark.app.name and value=org.apache.spark.deploy.dotnet.DotnetRunner to environment
21/01/09 18:29:57 INFO DotnetRunner: Adding key=spark.submit.pyFiles and value= to environment
21/01/09 18:29:57 INFO DotnetRunner: Adding key=spark.submit.deployMode and value=client to environment
21/01/09 18:29:57 INFO DotnetRunner: Adding key=spark.master and value=local to environment
[2021-01-09T10:29:57.7611666Z] [LAPTOP-8R49BD47] [Info] [ConfigurationService] Using port 53683 for connection.
[2021-01-09T10:29:57.7701007Z] [LAPTOP-8R49BD47] [Info] [JvmBridge] JvMBridge port is 53683
[2021-01-09T10:29:57.7750557Z] [LAPTOP-8R49BD47] [Info] [JvmBridge] The number of JVM backend thread is set to 10. The max number of concurrent sockets in JvmBridge is set to 7.
21/01/09 18:29:58 INFO SparkContext: Running Spark version 3.0.1
21/01/09 18:29:58 INFO ResourceUtils: ==============================================================
21/01/09 18:29:58 INFO ResourceUtils: Resources for spark.driver:

21/01/09 18:29:58 INFO ResourceUtils: ==============================================================
21/01/09 18:29:58 INFO SparkContext: Submitted application: .NET for Apache Spark Sentiment Analysis
21/01/09 18:29:58 INFO SecurityManager: Changing view acls to: YD
21/01/09 18:29:58 INFO SecurityManager: Changing modify acls to: YD
21/01/09 18:29:58 INFO SecurityManager: Changing view acls groups to:
21/01/09 18:29:58 INFO SecurityManager: Changing modify acls groups to:
21/01/09 18:29:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(YD); groups with view permissions: Set(); users with modify permissions: Set(YD); groups with modify permissions: Set()
21/01/09 18:29:58 INFO Utils: Successfully started service 'sparkDriver' on port 53691.
21/01/09 18:29:58 INFO SparkEnv: Registering MapOutputTracker
21/01/09 18:29:58 INFO SparkEnv: Registering BlockManagerMaster
21/01/09 18:29:58 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/01/09 18:29:58 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/01/09 18:29:58 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
21/01/09 18:29:58 INFO DiskBlockManager: Created local directory at C:\Users\YD\AppData\Local\Temp\blockmgr-f20af9bd-7dc2-4ff4-9595-1d93a63b47b4
21/01/09 18:29:58 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
21/01/09 18:29:58 INFO SparkEnv: Registering OutputCommitCoordinator
21/01/09 18:29:58 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
21/01/09 18:29:58 INFO Utils: Successfully started service 'SparkUI' on port 4041.
21/01/09 18:29:58 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://LAPTOP-8R49BD47:4041
21/01/09 18:29:59 INFO SparkContext: Added JAR file:/C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/microsoft-spark-3-0_2.12-1.0.0.jar at spark://LAPTOP-8R49BD47:53691/jars/microsoft-spark-3-0_2.12-1.0.0.jar with timestamp 1610188199046
21/01/09 18:29:59 INFO Executor: Starting executor ID driver on host LAPTOP-8R49BD47
21/01/09 18:29:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53706.
21/01/09 18:29:59 INFO NettyBlockTransferService: Server created on LAPTOP-8R49BD47:53706
21/01/09 18:29:59 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/01/09 18:29:59 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, LAPTOP-8R49BD47, 53706, None)
21/01/09 18:29:59 INFO BlockManagerMasterEndpoint: Registering block manager LAPTOP-8R49BD47:53706 with 434.4 MiB RAM, BlockManagerId(driver, LAPTOP-8R49BD47, 53706, None)
21/01/09 18:29:59 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, LAPTOP-8R49BD47, 53706, None)
21/01/09 18:29:59 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, LAPTOP-8R49BD47, 53706, None)
21/01/09 18:29:59 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/spark-warehouse').
21/01/09 18:30:00 INFO SharedState: Warehouse path is 'file:/C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/spark-warehouse'.
21/01/09 18:30:00 INFO InMemoryFileIndex: It took 53 ms to list leaf files for 1 paths.
21/01/09 18:30:01 INFO InMemoryFileIndex: It took 3 ms to list leaf files for 1 paths.
21/01/09 18:30:03 INFO FileSourceStrategy: Pruning directories with:
21/01/09 18:30:03 INFO FileSourceStrategy: Pushed Filters:
21/01/09 18:30:03 INFO FileSourceStrategy: Post-Scan Filters: (length(trim(value#0, None)) > 0)
21/01/09 18:30:03 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/01/09 18:30:04 INFO CodeGenerator: Code generated in 307.516 ms
21/01/09 18:30:04 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 171.3 KiB, free 434.2 MiB)
21/01/09 18:30:04 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.1 KiB, free 434.2 MiB)
21/01/09 18:30:04 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:04 INFO SparkContext: Created broadcast 0 from csv at NativeMethodAccessorImpl.java:0
21/01/09 18:30:04 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4225524 bytes, open cost is considered as scanning 4194304 bytes.
21/01/09 18:30:05 INFO SparkContext: Starting job: csv at NativeMethodAccessorImpl.java:0
21/01/09 18:30:05 INFO DAGScheduler: Got job 0 (csv at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/01/09 18:30:05 INFO DAGScheduler: Final stage: ResultStage 0 (csv at NativeMethodAccessorImpl.java:0)
21/01/09 18:30:05 INFO DAGScheduler: Parents of final stage: List()
21/01/09 18:30:05 INFO DAGScheduler: Missing parents: List()
21/01/09 18:30:05 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at csv at NativeMethodAccessorImpl.java:0), which has no missing parents
21/01/09 18:30:05 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 10.7 KiB, free 434.2 MiB)
21/01/09 18:30:05 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.3 KiB, free 434.2 MiB)
21/01/09 18:30:05 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 5.3 KiB, free: 434.4 MiB)
21/01/09 18:30:05 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1223
21/01/09 18:30:05 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at csv at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/01/09 18:30:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/01/09 18:30:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, LAPTOP-8R49BD47, executor driver, partition 0, PROCESS_LOCAL, 7798 bytes)
21/01/09 18:30:05 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/01/09 18:30:05 INFO Executor: Fetching spark://LAPTOP-8R49BD47:53691/jars/microsoft-spark-3-0_2.12-1.0.0.jar with timestamp 1610188199046
21/01/09 18:30:05 INFO TransportClientFactory: Successfully created connection to LAPTOP-8R49BD47/192.168.11.81:53691 after 24 ms (0 ms spent in bootstraps)
21/01/09 18:30:05 INFO Utils: Fetching spark://LAPTOP-8R49BD47:53691/jars/microsoft-spark-3-0_2.12-1.0.0.jar to C:\Users\YD\AppData\Local\Temp\spark-977ec410-19ef-4910-8e46-6e58ab7def14\userFiles-7db99229-141a-40b4-ad4c-108082a93a68\fetchFileTemp1545562991836894884.tmp
21/01/09 18:30:05 INFO Executor: Adding file:/C:/Users/YD/AppData/Local/Temp/spark-977ec410-19ef-4910-8e46-6e58ab7def14/userFiles-7db99229-141a-40b4-ad4c-108082a93a68/microsoft-spark-3-0_2.12-1.0.0.jar to class loader
21/01/09 18:30:05 INFO FileScanRDD: Reading File path: file:///C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/yelptest.csv, range: 0-31220, partition values: [empty row]
21/01/09 18:30:05 INFO CodeGenerator: Code generated in 16.4157 ms
21/01/09 18:30:05 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1583 bytes result sent to driver
21/01/09 18:30:05 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 549 ms on LAPTOP-8R49BD47 (executor driver) (1/1)
21/01/09 18:30:05 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
21/01/09 18:30:05 INFO DAGScheduler: ResultStage 0 (csv at NativeMethodAccessorImpl.java:0) finished in 0.730 s
21/01/09 18:30:05 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
21/01/09 18:30:05 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
21/01/09 18:30:05 INFO DAGScheduler: Job 0 finished: csv at NativeMethodAccessorImpl.java:0, took 0.786605 s
21/01/09 18:30:05 INFO CodeGenerator: Code generated in 15.1734 ms
21/01/09 18:30:06 INFO BlockManagerInfo: Removed broadcast_1_piece0 on LAPTOP-8R49BD47:53706 in memory (size: 5.3 KiB, free: 434.4 MiB)
21/01/09 18:30:06 INFO FileSourceStrategy: Pruning directories with:
21/01/09 18:30:06 INFO FileSourceStrategy: Pushed Filters:
21/01/09 18:30:06 INFO FileSourceStrategy: Post-Scan Filters:
21/01/09 18:30:06 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 171.3 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 24.1 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:06 INFO SparkContext: Created broadcast 2 from csv at NativeMethodAccessorImpl.java:0
21/01/09 18:30:06 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4225524 bytes, open cost is considered as scanning 4194304 bytes.
21/01/09 18:30:06 INFO SparkContext: Starting job: csv at NativeMethodAccessorImpl.java:0
21/01/09 18:30:06 INFO DAGScheduler: Got job 1 (csv at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/01/09 18:30:06 INFO DAGScheduler: Final stage: ResultStage 1 (csv at NativeMethodAccessorImpl.java:0)
21/01/09 18:30:06 INFO DAGScheduler: Parents of final stage: List()
21/01/09 18:30:06 INFO DAGScheduler: Missing parents: List()
21/01/09 18:30:06 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at csv at NativeMethodAccessorImpl.java:0), which has no missing parents
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 15.3 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 7.7 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 7.7 KiB, free: 434.3 MiB)
21/01/09 18:30:06 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1223
21/01/09 18:30:06 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at csv at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/01/09 18:30:06 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/01/09 18:30:06 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, LAPTOP-8R49BD47, executor driver, partition 0, PROCESS_LOCAL, 7798 bytes)
21/01/09 18:30:06 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/01/09 18:30:06 INFO BlockManagerInfo: Removed broadcast_0_piece0 on LAPTOP-8R49BD47:53706 in memory (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:06 INFO FileScanRDD: Reading File path: file:///C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/yelptest.csv, range: 0-31220, partition values: [empty row]
21/01/09 18:30:06 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1595 bytes result sent to driver
21/01/09 18:30:06 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 158 ms on LAPTOP-8R49BD47 (executor driver) (1/1)
21/01/09 18:30:06 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
21/01/09 18:30:06 INFO DAGScheduler: ResultStage 1 (csv at NativeMethodAccessorImpl.java:0) finished in 0.227 s
21/01/09 18:30:06 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
21/01/09 18:30:06 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
21/01/09 18:30:06 INFO DAGScheduler: Job 1 finished: csv at NativeMethodAccessorImpl.java:0, took 0.244445 s
21/01/09 18:30:06 INFO FileSourceStrategy: Pruning directories with:
21/01/09 18:30:06 INFO FileSourceStrategy: Pushed Filters:
21/01/09 18:30:06 INFO FileSourceStrategy: Post-Scan Filters:
21/01/09 18:30:06 INFO FileSourceStrategy: Output Data Schema: struct<ReviewText: string, Sentiment: string>
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 171.3 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO BlockManagerInfo: Removed broadcast_3_piece0 on LAPTOP-8R49BD47:53706 in memory (size: 7.7 KiB, free: 434.4 MiB)
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 24.1 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:06 INFO SparkContext: Created broadcast 4 from showString at NativeMethodAccessorImpl.java:0
21/01/09 18:30:06 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4225524 bytes, open cost is considered as scanning 4194304 bytes.
21/01/09 18:30:06 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
21/01/09 18:30:06 INFO DAGScheduler: Got job 2 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/01/09 18:30:06 INFO DAGScheduler: Final stage: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0)
21/01/09 18:30:06 INFO DAGScheduler: Parents of final stage: List()
21/01/09 18:30:06 INFO DAGScheduler: Missing parents: List()
21/01/09 18:30:06 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[12] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 8.8 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.9 KiB, free 434.0 MiB)
21/01/09 18:30:06 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 4.9 KiB, free: 434.3 MiB)
21/01/09 18:30:06 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1223
21/01/09 18:30:06 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[12] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/01/09 18:30:06 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
21/01/09 18:30:06 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, LAPTOP-8R49BD47, executor driver, partition 0, PROCESS_LOCAL, 7798 bytes)
21/01/09 18:30:06 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
21/01/09 18:30:06 INFO FileScanRDD: Reading File path: file:///C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/yelptest.csv, range: 0-31220, partition values: [empty row]
21/01/09 18:30:06 INFO CodeGenerator: Code generated in 22.741099 ms
21/01/09 18:30:06 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 2563 bytes result sent to driver
21/01/09 18:30:06 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 107 ms on LAPTOP-8R49BD47 (executor driver) (1/1)
21/01/09 18:30:06 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
21/01/09 18:30:06 INFO DAGScheduler: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0) finished in 0.142 s
21/01/09 18:30:06 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
21/01/09 18:30:06 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
21/01/09 18:30:06 INFO DAGScheduler: Job 2 finished: showString at NativeMethodAccessorImpl.java:0, took 0.162733 s
21/01/09 18:30:06 INFO CodeGenerator: Code generated in 26.014001 ms
+--------------------+---------+
| ReviewText|Sentiment|
+--------------------+---------+
|Waitress was swee...| 1|
|I also had to tas...| 1|
|I'd rather eat ai...| 0|
|Cant say enough g...| 1|
|The ambiance was ...| 1|
|The waitress and ...| 1|
|I would not recom...| 0|
|Overall I wasn't ...| 0|
|My gyro was basic...| 0|
| Terrible service!| 0|
|Thoroughly disapp...| 0|
|I don't each much...| 1|
|Give it a try, yo...| 1|
|By far the BEST c...| 1|
|Reasonably priced...| 1|
|Everything was pe...| 1|
|The food is very ...| 1|
|it was a drive to...| 0|
|At first glance i...| 1|
|Anyway, I do not ...| 0|
+--------------------+---------+
only showing top 20 rows

0.61130136
0.61130136
[2021-01-09T10:30:08.1248483Z] [LAPTOP-8R49BD47] [Debug] [ConfigurationService] Using the environment variable to construct .NET worker path: C:\bin\Microsoft.Spark.Worker-1.0.0\Microsoft.Spark.Worker.exe.
21/01/09 18:30:08 INFO BlockManagerInfo: Removed broadcast_5_piece0 on LAPTOP-8R49BD47:53706 in memory (size: 4.9 KiB, free: 434.4 MiB)
21/01/09 18:30:08 INFO BlockManagerInfo: Removed broadcast_2_piece0 on LAPTOP-8R49BD47:53706 in memory (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:08 INFO FileSourceStrategy: Pruning directories with:
21/01/09 18:30:08 INFO FileSourceStrategy: Pushed Filters:
21/01/09 18:30:08 INFO FileSourceStrategy: Post-Scan Filters:
21/01/09 18:30:08 INFO FileSourceStrategy: Output Data Schema: struct<ReviewText: string>
21/01/09 18:30:08 INFO CodeGenerator: Code generated in 31.628 ms
21/01/09 18:30:08 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 171.3 KiB, free 434.0 MiB)
21/01/09 18:30:08 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 24.1 KiB, free 434.0 MiB)
21/01/09 18:30:08 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:08 INFO SparkContext: Created broadcast 6 from showString at NativeMethodAccessorImpl.java:0
21/01/09 18:30:08 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4225524 bytes, open cost is considered as scanning 4194304 bytes.
21/01/09 18:30:08 INFO BlockManagerInfo: Removed broadcast_4_piece0 on LAPTOP-8R49BD47:53706 in memory (size: 24.1 KiB, free: 434.4 MiB)
21/01/09 18:30:08 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
21/01/09 18:30:09 INFO DAGScheduler: Got job 3 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/01/09 18:30:09 INFO DAGScheduler: Final stage: ResultStage 3 (showString at NativeMethodAccessorImpl.java:0)
21/01/09 18:30:09 INFO DAGScheduler: Parents of final stage: List()
21/01/09 18:30:09 INFO DAGScheduler: Missing parents: List()
21/01/09 18:30:09 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[18] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
21/01/09 18:30:09 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 17.1 KiB, free 434.2 MiB)
21/01/09 18:30:09 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 8.6 KiB, free 434.2 MiB)
21/01/09 18:30:09 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on LAPTOP-8R49BD47:53706 (size: 8.6 KiB, free: 434.4 MiB)
21/01/09 18:30:09 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1223
21/01/09 18:30:09 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[18] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/01/09 18:30:09 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
21/01/09 18:30:09 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, LAPTOP-8R49BD47, executor driver, partition 0, PROCESS_LOCAL, 7798 bytes)
21/01/09 18:30:09 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
21/01/09 18:30:09 INFO CodeGenerator: Code generated in 23.0987 ms
DotnetWorker PID:[22312] Args:[-m pyspark.worker] SparkVersion:[3.0.1]
[2021-01-09T10:30:09.6927376Z] [LAPTOP-8R49BD47] [Info] [SimpleWorker] RunSimpleWorker() is starting with port = 53719.
[2021-01-09T10:30:09.7797201Z] [LAPTOP-8R49BD47] [Info] [TaskRunner] [0] Starting with ReuseSocket[False].
21/01/09 18:30:09 INFO FileScanRDD: Reading File path: file:///C:/Users/YD/source/repos/MySparkAppML.ConsoleApp/bin/Debug/netcoreapp3.1/publish/yelptest.csv, range: 0-31220, partition values: [empty row]
[2021-01-09T10:30:09.7980951Z] [LAPTOP-8R49BD47] [Info] [ConfigurationService] 'DOTNETBACKEND_PORT' environment variable is not set.
[2021-01-09T10:30:09.7981653Z] [LAPTOP-8R49BD47] [Info] [ConfigurationService] Using port 5567 for connection.
[2021-01-09T10:30:09.8053556Z] [LAPTOP-8R49BD47] [Info] [JvmBridge] JvMBridge port is 5567
21/01/09 18:30:09 INFO CodeGenerator: Code generated in 19.9682 ms
[2021-01-09T10:30:09.8144165Z] [LAPTOP-8R49BD47] [Info] [JvmBridge] The number of JVM backend thread is set to 10. The max number of concurrent sockets in JvmBridge is set to 7.
[2021-01-09T10:30:10.6518123Z] [LAPTOP-8R49BD47] [Error] [TaskRunner] [0] ProcessStream() failed with exception: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51 at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252 at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152 at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76 at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65 at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154 [2021-01-09T10:30:10.6527954Z] [LAPTOP-8R49BD47] [Error] [TaskRunner] [0] Exiting with exception: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes) at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction) at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction) at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition) at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35 at Microsoft.Spark.Sql.PicklingUdfWrapper2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in //src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs21/01/09 18:30:10 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51 at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252 at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152 at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76 at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65 at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154 at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503) at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81) at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:835) :line 154 at Microsoft.Spark.Worker.TaskRunner.Run() in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 66 [2021-01-09T10:30:10.6537382Z] [LAPTOP-8R49BD47] [Info] [TaskRunner] [0] Finished running 0 task(s). [2021-01-09T10:30:10.6537724Z] [LAPTOP-8R49BD47] [Info] [SimpleWorker] RunSimpleWorker() finished successfully 21/01/09 18:30:10 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 3, LAPTOP-8R49BD47, executor driver): org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes) at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction) at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction) at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition) at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35 at Microsoft.Spark.Sql.PicklingUdfWrapper2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in //src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)

21/01/09 18:30:10 ERROR TaskSetManager: Task 0 in stage 3.0 failed 1 times; aborting job
21/01/09 18:30:10 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
21/01/09 18:30:10 INFO TaskSchedulerImpl: Cancelling stage 3
21/01/09 18:30:10 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage cancelled
21/01/09 18:30:10 INFO DAGScheduler: ResultStage 3 (showString at NativeMethodAccessorImpl.java:0) failed in 1.700 s due to Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, LAPTOP-8R49BD47, executor driver): org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper`2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)

Driver stacktrace:
21/01/09 18:30:10 INFO DAGScheduler: Job 3 failed: showString at NativeMethodAccessorImpl.java:0, took 1.716029 s
21/01/09 18:30:10 ERROR DotnetBackendHandler: Failed to execute 'showString' on 'org.apache.spark.sql.Dataset' with args=([Type=java.lang.Integer, Value: 20], [Type=java.lang.Integer, Value: 20], [Type=java.lang.Boolean, Value: false])
[2021-01-09T10:30:10.7275550Z] [LAPTOP-8R49BD47] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'showString' failed for class '18' when called with 3 arguments ([Index=1, Type=Int32, Value=20], [Index=2, Type=Int32, Value=20], [Index=3, Type=Boolean, Value=False], )
[2021-01-09T10:30:10.7276810Z] [LAPTOP-8R49BD47] [Error] [JvmBridge] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, LAPTOP-8R49BD47, executor driver): org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper`2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2139)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:467)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:420)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:47)
at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3627)
at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2697)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2697)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2904)
at org.apache.spark.sql.Dataset.getRows(Dataset.scala:300)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:337)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:159)
at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:99)
at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper`2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
... 3 more

[2021-01-09T10:30:10.7801524Z] [LAPTOP-8R49BD47] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'showString' failed for class '18' when called with 3 arguments ([Index=1, Type=Int32, Value=20], [Index=2, Type=Int32, Value=20], [Index=3, Type=Boolean, Value=False], )
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'showString' failed for class '18' when called with 3 arguments ([Index=1, Type=Int32, Value=20], [Index=2, Type=Int32, Value=20], [Index=3, Type=Boolean, Value=False], )
---> Microsoft.Spark.JvmException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, LAPTOP-8R49BD47, executor driver): org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper`2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2139)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:467)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:420)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:47)
at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3627)
at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2697)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2697)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2904)
at org.apache.spark.sql.Dataset.getRows(Dataset.scala:300)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:337)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:159)
at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:99)
at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: org.apache.spark.api.python.PythonException: System.TypeLoadException: Could not load type 'Microsoft.ML.Data.DataViewTypeAttribute' from assembly 'Microsoft.ML.DataView, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
at Microsoft.ML.Data.SchemaDefinition.GetNameAndCustomAttributes(MemberInfo memberInfo, Type userType, HashSet1 colNames, String& name, IEnumerable1& customAttributes)
at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.InternalSchemaDefinition.Create(Type userType, Direction direction)
at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
at Microsoft.ML.PredictionEngineBase2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition) at Microsoft.ML.PredictionEngine2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
at MySparkAppML.ConsoleApp.Program.predict(String text) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 35
at Microsoft.Spark.Sql.PicklingUdfWrapper`2.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 51
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
... 3 more

--- End of inner exception stack trace ---
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object[] args)
at Microsoft.Spark.Sql.DataFrame.Show(Int32 numRows, Int32 truncate, Boolean vertical)
at MySparkAppML.ConsoleApp.Program.Main(String[] args) in C:\Users\YD\source\repos\MySparkAppML.ConsoleApp\Program.cs:line 28

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    SparkIssues relating to running ML.NET using .NET for SparkbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions