[FEATURE REQUEST]: Implement ML Features

[GoEddie]

** These should all be implemented with #1031 **

=========================================

This is to track implementation of the ML-Features: https://spark.apache.org/docs/latest/ml-features

Bucketizer has been implemented in #378 but there are more features that should be implemented.

• Feature Extractors
  • TF-IDF
  • Word2Vec (Implement ML Features: Word2Vec #491)
  • CountVectorizer (Implement ML/CountVectorizer and ML/CountVectorizerModel #608)
  • FeatureHasher (FeatureHasher #652)
• Feature Transformers
  • Tokenizer (base class for Feature as lots of methods are shared between the objects (more methods to be added in later pr's) #574)
  • StopWordsRemover (Add stop words removers #726 thanks @SARAVANA1501 )
  • n-gram (in-progress Add NGram #734)
  • Binarizer (in-progress Add Binarizer #744)
  • [] PCA (in-progress)
  • PolynormalExpansion
  • Dicrete Cosine Transform (DCT)
  • StringIndexer (in-progress)
  • IndexToString
  • OneHotEncoderEstimator
  • VectorIndexer
  • Normalizer
  • StandardScaler
  • MinMaxScaler
  • MaxAbsScaler
  • Bucketizer
  • ElementwiseProduct
  • SQLTransformer (Implement ML Features #381. SQLTransformer class and testcase #781 @ramanathanv)
  • VectorAssembler
  • VectorSizeHint
  • QuantileDiscretizer
  • Imputer
• Feature Selectors
  • VectorSlicer
  • RFormula
  • ChiSqSelector
• Locality Sensitive Hashing
  • LSH Operations
    • Feature Transformation
    • Approximate Similarity Join
    • Approximate Nearest Neighbour Search
  • LSH Algorithms
    • Bucketed Random Projection for Euclidean Distance
    • MinHash for Jaccard Distance

If anyone else is going to implement probably best to put a comment here and I'll keep the list up to date.

[imback82]

Thanks @GoEddie for compiling the list.

If anyone else is going to implement probably best to put a comment here or something??

This sounds good to me. We will review #378 in the next few days (some of us are out on vacation).

[GoEddie]

Great thanks, @imback82 I didn’t expect anyone to reply until the new year :)

[Niharikadutta]

@GoEddie Thanks so much for your work on this! Would you be willing to write a blog post on these ML-features currently supported on .NET for Apache Spark? Please let us know if that is something that could interest you, and if you might require any help from us. Thanks! :)

[GoEddie]

Hi @Niharikadutta sure no problem I would be happy to do this.

Where is best to publish it? My site or is there a ms one I could guest write it for?

[Niharikadutta]

Hi @GoEddie Awesome, thanks so much! I will get back to you with the details about where to publish and how to get you guest access for it. Could you provide me with a preferred email account you would wish to have this associated with, and I will work on getting that access. Thanks again! :)

[GoEddie]

Great, ed.elliott AT outlook.com is good :)

[Niharikadutta]

Perfect, thank you! Will get back to you soon with the details. Also, is this email good to also have the monthly office hour Teams meeting on, as discussed in a separate thread?

[GoEddie]

Yes great (sorry forgot about that issue!)

[Niharikadutta]

Awesome, thanks for confirming! Will set that one up starting sometime next month if that works for you, or please let me know when you would like to have it start, along with your day of the week/time preference.

[GoEddie]

Just whatever suits you really, I didn’t want to make work for anyone

[Niharikadutta]

oh no we are happy to do this :) I just want to make sure I have your timezone/work hour constraints in mind before setting a meeting up.

[GoEddie]

Great, I’m in the UK so morning US time works better for me