Customer Feedback Analysis for Financial Services LLM

Project Overview:

Objective

The goal was to build a comprehensive dataset of customer feedback, annotated for sentiment and topic, to improve the accuracy of LLMs in analyzing sentiment and categorizing customer support issues in financial services.

Scope

The dataset includes both structure and unstructure customer feedback from financial service providers, including banks and insurance companies. The feedback was annotate for sentiment and categorize by topic to ensure precise analysis by LLMs.

Sources

Customer Feedback Collection: The data was sourced from 30,000 feedback entries provided by customers of various financial services, including both banks and insurance firms.

Data Collection Metrics

Total Feedback Entries: 30,000 feedback entries were collected.
Sentiment Tags: Each feedback entry was annotated with 3 sentiment tags (positive, negative, or neutral).

Annotation Process

Stages

Sentiment Classification: Annotators classified the feedback into three categories: positive, negative, or neutral.
Topic Categorization: Feedback was also categorized by topics relevant to financial services, such as account management, loan services, and customer support.

Annotation Metrics

Total Sentiment Annotations: 30,000 sentiment tags were applied.
Team Involvement: 35 annotators worked on the project over a duration of 1 month.

Quality Assurance

Stages

Annotation Accuracy: Continuous checks were performed to ensure that sentiment and topic annotations were accurate and aligned with the feedback content.
Consistency Checks: Regular reviews were conducted to maintain consistent tagging across all entries.

QA Metrics

Sentiment Accuracy: The project achieved a high accuracy rate in correctly identifying customer sentiment across feedback entries.
Topic Classification Accuracy: The feedback was accurately categorized by topic, enhancing the relevance of responses generated by LLMs.

Conclusion

The creation of this dataset marked a significant improvement in the ability of LLMs to analyze customer sentiment and support issues within the financial services industry. By accurately interpreting customer feedback, the dataset has contributed to better automated customer service responses and increased customer satisfaction.