Understanding and Implementing Medprompt | by Anand Subramanian | Jul, 2024

Understanding and Implementing Medprompt | by Anand Subramanian | Jul, 2024
Understanding and Implementing Medprompt | by Anand Subramanian | Jul, 2024

We now carry out selection shuffling ensembling by shuffling the order of reply selections for every check query, creating a number of variants of the identical query. The LLM is then prompted with these variants, together with the corresponding few-shot exemplars, to generate reasoning steps and a solution for every variant. Lastly, we carry out a majority vote over the predictions from all variants and choose the ultimate prediction.

The code associated to this implementation could be discovered at this github repo link.

We use the MedQA [6] dataset for implementing and evaluating Medprompt. We first outline helper features for parsing the jsonl information.

def write_jsonl_file(file_path, dict_list):
Write an inventory of dictionaries to a JSON Strains file.

- file_path (str): The trail to the file the place the information will likely be written.
- dict_list (listing): A listing of dictionaries to jot down to the file.
with open(file_path, 'w') as file:
for dictionary in dict_list:
json_line = json.dumps(dictionary)
file.write(json_line + 'n')

def read_jsonl_file(file_path):
Parses a JSONL (JSON Strains) file and returns an inventory of dictionaries.

file_path (str): The trail to the JSONL file to be learn.

listing of dict: A listing the place every component is a dictionary representing
a JSON object from the file.
jsonl_lines = []
with open(file_path, 'r', encoding="utf-8") as file:
for line in file:
json_object = json.masses(line)

return jsonl_lines

Implementing Self-Generated CoT

For our implementation, we make the most of the coaching set from MedQA. We implement a zero-shot CoT immediate and course of all of the coaching questions. We use GPT-4o in our implementation. For every query, we generate the CoT and the corresponding reply. We outline a immediate which relies on the template supplied within the Medprompt paper.

system_prompt = """You might be an skilled medical skilled. You might be supplied with a medical query with a number of reply selections.
Your purpose is to assume by means of the query fastidiously and clarify your reasoning step-by-step earlier than deciding on the ultimate reply.
Reply solely with the reasoning steps and reply as specified under.
Under is the format for every query and reply:

## Query: {{query}}

## Reply
(mannequin generated chain of thought clarification)
Subsequently, the reply is [final model answer (e.g. A,B,C,D)]"""

def build_few_shot_prompt(system_prompt, query, examples, include_cot=True):
Builds the zero-shot immediate.

system_prompt (str): Job Instruction for the LLM
content material (dict): The content material for which to create a question, formatted as
required by `create_query`.

listing of dict: A listing of messages, together with a system message defining
the duty and a person message with the enter query.
messages = [{"role": "system", "content": system_prompt}]

for elem in examples:
messages.append({"function": "person", "content material": create_query(elem)})
if include_cot:
messages.append({"function": "assistant", "content material": format_answer(elem["cot"], elem["answer_idx"])})
answer_string = f"""## AnswernTherefore, the reply is {elem["answer_idx"]}"""
messages.append({"function": "assistant", "content material": answer_string})

messages.append({"function": "person", "content material": create_query(query)})
return messages

def get_response(messages, model_name, temperature = 0.0, max_tokens = 10):
Obtains the responses/solutions of the mannequin by means of the chat-completions API.

messages (listing of dict): The constructed messages supplied to the API.
model_name (str): Identify of the mannequin to entry by means of the API
temperature (float): A price between 0 and 1 that controls the randomness of the output.
A temperature worth of 0 ideally makes the mannequin decide the probably token, making the outputs deterministic.
max_tokens (int): Most variety of tokens that the mannequin ought to generate

str: The response message content material from the mannequin.
response = consumer.chat.completions.create(
return response.selections[0].message.content material

We additionally outline helper features for parsing the reasoning and the ultimate reply choice from the LLM response.

def matches_ans_option(s):
Checks if the string begins with the particular sample 'Subsequently, the reply is [A-Z]'.

s (str): The string to be checked.

bool: True if the string matches the sample, False in any other case.
return bool(re.match(r'^Subsequently, the reply is [A-Z]', s))

def extract_ans_option(s):
Extracts the reply choice (a single capital letter) from the beginning of the string.

s (str): The string containing the reply sample.

str or None: The captured reply choice if the sample is discovered, in any other case None.
match = re.search(r'^Subsequently, the reply is ([A-Z])', s)
if match:
return match.group(1) # Returns the captured alphabet
return None

def matches_answer_start(s):
Checks if the string begins with the markdown header '## Reply'.

s (str): The string to be checked.

bool: True if the string begins with '## Reply', False in any other case.
return s.startswith("## Reply")

def validate_response(s):
Validates a multi-line string response that it begins with '## Reply' and ends with the reply sample.

s (str): The multi-line string response to be validated.

bool: True if the response is legitimate, False in any other case.
file_content = s.break up("n")

return matches_ans_option(file_content[-1]) and matches_answer_start(s)

def parse_answer(response):
Parses a response that begins with '## Reply', extracting the reasoning and the reply selection.

response (str): The multi-line string response containing the reply and reasoning.

tuple: A tuple containing the extracted CoT reasoning and the reply selection.
split_response = response.break up("n")
assert split_response[0] == "## Reply"
cot_reasoning = "n".be a part of(split_response[1:-1]).strip()
ans_choice = extract_ans_option(split_response[-1])
return cot_reasoning, ans_choice

We now course of the questions within the coaching set of MedQA. We receive CoT responses and solutions for all questions and retailer them to a folder.

train_data = read_jsonl_file("information/phrases_no_exclude_train.jsonl")

cot_responses = []
# os.mkdir("cot_responses")
existing_files = os.listdir("cot_responses/")

for idx, merchandise in enumerate(tqdm(train_data)):
if str(idx) + ".txt" in existing_files:

immediate = build_zero_shot_prompt(system_prompt, merchandise)
response = get_response(immediate, model_name="gpt-4o", max_tokens=500)
with open(os.path.be a part of("cot_responses", str(idx) + ".txt"), "w", encoding="utf-8") as f:
besides Exception as e :

We now iterate throughout all of the generated responses to test if they’re legitimate and cling to the prediction format outlined within the immediate. We discard responses that don’t conform to the required format. After that, we test the anticipated solutions in opposition to the bottom reality for every query and solely retain questions for which the anticipated solutions match the bottom reality.

questions_dict = []
ctr = 0
for idx, query in enumerate(tqdm(train_data)):
file = open(os.path.be a part of("cot_responses/", str(idx) + ".txt"), encoding="utf-8").learn()
if not validate_response(file):

cot, pred_ans = parse_answer(file)

dict_elem = {}
dict_elem["idx"] = idx
dict_elem["question"] = query["question"]
dict_elem["answer"] = query["answer"]
dict_elem["options"] = query["options"]
dict_elem["cot"] = cot
dict_elem["pred_ans"] = pred_ans

filtered_questions_dict = []
for merchandise in tqdm(questions_dict):
pred_ans = merchandise["options"][item["pred_ans"]]
if pred_ans == merchandise["answer"]:

Implementing the KNN mannequin

Having processed the coaching set and obtained the CoT response for all these questions, we now embed all questions utilizing the text-embedding-ada-002 from OpenAI.

def get_embedding(textual content, mannequin="text-embedding-ada-002"):
return consumer.embeddings.create(enter = [text], mannequin=mannequin).information[0].embedding

for merchandise in tqdm(filtered_questions_dict):
merchandise["embedding"] = get_embedding(merchandise["question"])
inv_options_map = {v:okay for okay,v in merchandise["options"].gadgets()}
merchandise["answer_idx"] = inv_options_map[item["answer"]]

We now practice a KNN mannequin utilizing these query embeddings. This acts as a retriever at inference time, because it helps us to retrieve related datapoints from the coaching set which might be most just like the query from the check set.

import numpy as np
from sklearn.neighbors import NearestNeighbors

embeddings = np.array([d["embedding"] for d in filtered_questions_dict])
indices = listing(vary(len(filtered_questions_dict)))

knn = NearestNeighbors(n_neighbors=5, algorithm='auto', metric='cosine').match(embeddings)

Implementing the Dynamic Few-Shot and Selection Shuffling Ensemble Logic

We will now run inference. We subsample 500 questions from the MedQA check set for our analysis. For every query, we retrieve the 5 most related questions from the practice set utilizing the KNN module, together with their respective CoT reasoning steps and predicted solutions. We assemble a few-shot immediate utilizing these examples.

For every query, we additionally shuffle the order of the choices 5 instances to create completely different variants. We then make the most of the constructed few-shot immediate to get the anticipated reply for every of the variants with shuffled choices.

def shuffle_option_labels(answer_options):
Shuffles the choices of the query.

answer_options (dict): A dictionary with the choices.

dict: A brand new dictionary with the shuffled choices.
choices = listing(answer_options.values())
labels = [chr(i) for i in range(ord('A'), ord('A') + len(options))]
shuffled_options_dict = {label: choice for label, choice in zip(labels, choices)}

return shuffled_options_dict

test_samples = read_jsonl_file("final_processed_test_set_responses_medprompt.jsonl")

for query in tqdm(test_samples, color ="inexperienced"):
question_variants = []
prompt_variants = []
cot_responses = []
question_embedding = get_embedding(query["question"])
distances, top_k_indices = knn.kneighbors([question_embedding], n_neighbors=5)
top_k_dicts = [filtered_questions_dict[i] for i in top_k_indices[0]]
query["outputs"] = []

for idx in vary(5):
question_copy = query.copy()
shuffled_options = shuffle_option_labels(query["options"])
inv_map = {v:okay for okay,v in shuffled_options.gadgets()}

question_copy["options"] = shuffled_options
question_copy["answer_idx"] = inv_map[question_copy["answer"]]
immediate = build_few_shot_prompt(system_prompt, question_copy, top_k_dicts)

for immediate in tqdm(prompt_variants):
response = get_response(immediate, model_name="gpt-4o", max_tokens=500)

for question_sample, reply in zip(question_variants, cot_responses):
if validate_response(reply):
cot, pred_ans = parse_answer(reply)

cot = ""
pred_ans = ""

query["outputs"].append({"query": question_sample["question"], "choices": question_sample["options"], "cot": cot, "pred_ans": question_sample["options"].get(pred_ans, "")})

We now consider the outcomes of Medprompt over the check set. For every query, we now have 5 predictions generated by means of the ensemble logic. We take the mode, or most steadily occurring prediction, for every query as the ultimate prediction and consider the efficiency. Two edge circumstances are potential right here:

  1. Two completely different reply choices are predicted two instances every, with no clear winner.
  2. There’s an error with the response generated, which means that we don’t have a predicted reply choice.

For each of those edge circumstances, we think about the query to be wrongly answered by the LLM.

def find_mode_string_list(string_list):
Finds probably the most steadily occurring strings.

string_list (listing of str): A listing of strings.
listing of str or None: A listing containing probably the most frequent string(s) from the enter listing.
Returns None if the enter listing is empty.
if not string_list:
return None

string_counts = Counter(string_list)
max_freq = max(string_counts.values())
mode_strings = [string for string, count in string_counts.items() if count == max_freq]
return mode_strings

ctr = 0
for merchandise in test_samples:
pred_ans = [x["pred_ans"] for x in merchandise["outputs"]]
freq_ans = find_mode_string_list(pred_ans)

if len(freq_ans) > 1:
final_prediction = ""
final_prediction = freq_ans[0]

if final_prediction == merchandise["answer"]:
ctr +=1

print(ctr / len(test_samples))

We consider the efficiency of Medprompt with GPT-4o when it comes to accuracy on the MedQA check subset. Moreover, we benchmark the efficiency of Zero-shot prompting, Random Few-Shot prompting, and Random Few-Shot with CoT prompting.

Outcomes of our analysis (Picture by Writer)

We observe that Medprompt and Random Few-Shot CoT prompting outperform the Zero and Few-Shot prompting baselines. Nevertheless, surprisingly, we discover that Random Few-Shot CoT outperforms our Medprompt efficiency. This could possibly be because of a few causes:

  1. The unique Medprompt paper benchmarked the efficiency of GPT-4. We observe that GPT-4o outperforms GPT-4T and GPT-4 on numerous textual content benchmarks considerably (https://openai.com/index/hello-gpt-4o/), indicating that Medprompt might have a lesser impact on a stronger mannequin like GPT-4o.
  2. We prohibit our analysis to 500 questions subsampled from MedQA. The Medprompt paper evaluates different Medical MCQA datasets and the complete model of MedQA. Evaluating GPT-4o on the whole variations of the datasets might give a greater image of the general efficiency.

Medprompt is an fascinating framework for creating subtle prompting pipelines, notably for adapting a generalist LLM to a particular area with out the necessity for fine-tuning. It additionally highlights the issues concerned in deciding between prompting and fine-tuning for numerous use circumstances. Exploring how far prompting could be pushed to boost LLM efficiency is necessary, because it gives a useful resource and cost-efficient various to fine-tuning.

[1] Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., … & Horvitz, E. (2023). Can generalist basis fashions outcompete special-purpose tuning? case research in medication. arXiv preprint arXiv:2311.16452. (https://arxiv.org/abs/2311.16452)

[2] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., … & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in massive language fashions. Advances in Neural Data Processing Programs, 35, 24824–24837. (https://openreview.net/pdf?id=_VjQlMeSB_J)

[3] Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., & Herzig, J. (2024). Does Tremendous-Tuning LLMs on New Data Encourage Hallucinations?. arXiv preprint arXiv:2405.05904. (https://arxiv.org/abs/2405.05904)

[4] Singhal, Ok., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., … & Natarajan, V. (2023). Giant language fashions encode medical data. Nature, 620(7972), 172–180. (https://www.nature.com/articles/s41586-023-06291-2)

[5] Singhal, Ok., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., … & Natarajan, V. (2023). In direction of expert-level medical query answering with massive language fashions. arXiv preprint arXiv:2305.09617. (https://arxiv.org/abs/2305.09617)

[6] Jin, D., Pan, E., Oufattole, N., Weng, W. H., Fang, H., & Szolovits, P. (2021). What illness does this affected person have? a large-scale open area query answering dataset from medical exams. Utilized Sciences, 11(14), 6421. (https://arxiv.org/abs/2009.13081) (Unique dataset is launched underneath a MIT License)

Leave a Reply

Your email address will not be published. Required fields are marked *