You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tokenization of the word “serakah” in IndoBERT #49
In my undergraduate sentiment analysis research, I found that indobenchmark/indobert-base-p1 tokenizes the standard Indonesian word “serakah” as: ['ser', '##akah'] This means the word is not available as a complete token in the vocabulary. Although this is normal WordPiece behavior, I suggest considering frequently used Indonesian sentiment words when evaluating the vocabulary for future model development.
Activity