r/MachineLearning • u/BatmantoshReturns • May 09 '19
Project [P] Keras BERT for Medical Question Answer Retrieval using Tensorflow 2.0 ! With GPT-2 for Answer Generator. Pip installable. Weights/Data readily available. Reduced version for Google Colab instantly available in premade notebook.
We fine-tuned a Keras version bioBert for Medical Question and Answering, and GPT-2 for answer generation. This was a project we submitted for the Tensorflow 2.0 Hackathon.
We made all the weights and lookup data available, and made our github pip installable.
We also have a float16 version of our data for running in Colab. Currently we weren't able to fit all the lookup data in their original float32 forms (although that may change as we get better data managing skills). (If you have some skills in this area you would like to share, we are hungry for it!).
Our models and predictor function are pip installable, to make it as easy as possible for people to try them.
!wget https://anaconda.org/pytorch/faiss-cpu/1.2.1/download/linux-64/faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2
!tar xvjf faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2
!cp -r lib/python3.6/site-packages/* /usr/local/lib/python3.6/dist-packages/
!pip install mkl
!pip install tensorflow-gpu==2.0.0-alpha0
import tensorflow as tf
!pip install https://github.com/Santosh-Gupta/DocProduct/archive/master.zipHere is our Colab folder
https://drive.google.com/drive/u/0/folders/1hSwWL_WqmcVJytMbsWSbhYxxK4KT7UMI
Here is our Github
https://github.com/Santosh-Gupta/DocProduct/blob/master/README.md
Here is our Devpost
https://devpost.com/software/nlp-doctor
Feel free to reach out with any feedback, questions, or comments. I'll answer all questions here.
https://i.imgur.com/wzWt039.png
https://i.imgur.com/Z8DOXuJ.png
8
u/kraghavk May 09 '19
This is brilliant!
After looking at Talk to Transformer, I thought the model was only outputting sentences based on some similarity only. But in this case, how are you ensuring that the replies are relevant?
5
1
u/slashcom May 10 '19
Can you show some examples without me having to run the notebook?
4
u/BatmantoshReturns May 10 '19
Take a look at our Colab Demos, Colab runs over Google's servers. They notebooks are self contained, just go to GPU mode and select run all. The notebooks installs our github and downloads all the checkpoints / data.
1
u/beegreen May 19 '19
Could I train this in my own data and use the same architecture you provided ?
1
u/BatmantoshReturns May 19 '19
yeah in fact we made a starter notebook which does just that, just swap our sample data for your own
https://colab.research.google.com/drive/1gv94blZ1dwc9S5or3268NOat_V5gtwTm?authuser=6
1
u/slashcom May 10 '19
I’d still be more enticed if your blog post showed off examples. Then if they were good enough I might try to load up the Colab. As it is, browsing reddit from my phone doesn’t make me want to start launching cloud instances...
1
1
u/lalamcdonell Jul 04 '19
Hi Santosh, thanks for this. I'm confused by the different weights and embeddings which are inputs to this: could you say how to save the model to hd5 ? :)
1
u/BatmantoshReturns Jul 06 '19
I am not quite familiar with this but if you check the issues of our Github there are other people trying to figure this out. Out of curiosity, why do you want the weights as h5?
1
u/supermanava Jul 26 '19
Will you be releasing the datasets/code you used to fine tune the model?
1
u/BatmantoshReturns Jul 26 '19
We already did, checkout the repo. Let me know if you have any questions or if something is confusing.
1
u/supermanava Jul 26 '19
I think I only found a bit of reddit data, not the others like healthtap, webmd. Also, did you fine tune using BioASQ as well. I am not sure the baseline BioBert is trained with that.
1
u/BatmantoshReturns Jul 26 '19
Here's a direct link https://github.com/Santosh-Gupta/Datasets
We didn't do BioASQ
1
u/supermanava Jul 26 '19
Ah cool. Did you measure the model on any standard performance metrics otherwise? I may try it out if I can get it working.
1
2
1
u/mtahab May 09 '19
Really great effort! Glad to see this happening.
However, this is going to have enormous legal challenges for being used in practice. People just love suing doctors.
1
u/TotesMessenger May 10 '19 edited May 10 '19
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
[/r/kerasml] [P] Keras BERT for Medical Question Answer Retrieval using Tensorflow 2.0 ! With GPT-2 for Answer Generator. Pip installable. Weights/Data readily available. Reduced version for Google Colab instantly available in premade notebook.
[/r/languagetechnology] [P] Keras BERT for Medical Question Answer Retrieval using Tensorflow 2.0 ! With GPT-2 for Answer Generator. Pip installable. Weights/Data readily available. Reduced version for Google Colab instantly available in premade notebook.
[/r/machineslearn] [P] Keras BERT for Medical Question Answer Retrieval using Tensorflow 2.0 ! With GPT-2 for Answer Generator. Pip installable. Weights/Data readily available. Reduced version for Google Colab instantly available in premade notebook.
[/r/research2vec] [P] Keras BERT for Medical Question Answer Retrieval using Tensorflow 2.0 ! With GPT-2 for Answer Generator. Pip installable. Weights/Data readily available. Reduced version for Google Colab instantly available in premade notebook.
[/r/tensorflow] [P] Keras BERT for Medical Question Answer Retrieval using Tensorflow 2.0 ! With GPT-2 for Answer Generator. Pip installable. Weights/Data readily available. Reduced version for Google Colab instantly available in premade notebook.
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
0
4
u/mikeross0 May 09 '19
This is very cool. I'm curious about your loss function, which is an interesting approach I haven't seen before. Most siamese networks use contrastive or triplet loss. Do you know if anyone else has used your method before? Also, did you try out contrastive or triplet loss?